Enabling fully disaggregated shuffle on Spark

I would like to start a conversation about extending the Spark shuffle manager surface to support fully disaggregated shuffle implementations. This is closely related to the work in SPARK-25299 , which is focused on refactoring the shuffle manager

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

Cc Yuming, Steve, and Dongjoon On Fri, Nov 15, 2019 at 10:37 AM Cheng Lian wrote: > Similar to Xiao, my major concern about making Hadoop 3.2 the default > Hadoop version is quality control. The current hadoop-3.2 profile covers > too many major component upgrades, i.e.: > >- Hadoop 3.2 >

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

Similar to Xiao, my major concern about making Hadoop 3.2 the default Hadoop version is quality control. The current hadoop-3.2 profile covers too many major component upgrades, i.e.: - Hadoop 3.2 - Hive 2.3 - JDK 11 We have already found and fixed some feature and performance

Re: Adding JIRA ID as the prefix for the test case name

Junit5: Display names. Goes all the way to the XML. https://junit.org/junit5/docs/current/user-guide/#writing-tests-display-names On Thu, Nov 14, 2019 at 6:13 PM Shixiong(Ryan) Zhu wrote: > Should we also add a guideline for non Scala tests? Other languages (Java, > Python, R) don't support

Re: Ask for ARM CI for spark

Hi @Sean Owen , Thanks for your idea. We may use the bad words to describe our request. That's true that we cannot just say "Spark support ARM from release 3.0.0", and we also cannot say the past releases cannot run on ARM. But the reality is the past releases didn't get a fully test on ARM

Re: Ask for ARM CI for spark

@Sean Owen, Thanks for attention this. I agree with you, it's probably not very appropriate to say 'support arm from 3.0 release'. How about change to the word "Spark community supports fully tests on arm from 3.0 release"? Let's try to think about it from the user's point of view than

Re: Ask for ARM CI for spark

I'm not against it, but the JIRAs will already show that the small ARM-related difference like floating-point in log() were resolved. Those aren't major enough to highlight as key changes in the 2000+ resolved. it didn't really not-work before either, as I understand; Spark isn't specific to an