Great to hear this! It's encouraging that Celeborn helps MR3. Celeborn is a general purpose remote shuffle service that stores and serves shuffle data (and other intermediate data in the future) to help compute engines better use disaggregated architecture, as well as become more efficient and stable for huge shuffle sized jobs.
Currently Celeborn supports Hive on MR, and I think integrating with MR3 provides a good example to support Hive on Tez. Thanks, Keyong Zhou On 2023/10/24 12:08:54 Sungwoo Park wrote: > Hi Hive users, > > Before the impending release of MR3 1.8, we would like to announce the > release of Hive-MR3 with Celeborn (Hive 3.1.3 on MR3 1.8 with Celeborn > 0.3.1). > > Apache Celeborn [1] is remote shuffle service, similar to Magnet [2] and > Apache Uniffle [3] (which was discussed in this Hive mailing list a while > ago). Celeborn officially supports Spark and Flink, and we have implemented > an MR3-extension for Celeborn. > > In addition to all the benefits of using remote shuffle service, > Hive-MR3-Celeborn supports direct processing of mapper output on the > reducer side, which means that reducers do not store mapper output on local > disks (for unordered edges). In this way, Hive-MR3-Celeborn can eliminate > over 95% of local disk writes when tested on the 10TB TPC-DS benchmark. > This can be particularly useful when running Hive-MR3 on public clouds > where fast local disk storage is expensive or not available. > > We have documented the usage of Hive-MR3-Celeborn in [4]. You can download > Hive-MR3-Celeborn in [5]. > > FYI, MR3 is an execution engine providing native support for Hadoop, > Kubernetes, and standalone mode [6]. Hive-MR3, its main application, > provides the performance of LLAP yet is very easy to install and operate. > If you are using Hive-Tez for running ETL jobs, switching to Hive-MR3 will > give you a much higher throughput thanks to its advanced resource sharing > model. > > We have recently opened a Slack channel. If interested, please join the > Slack channel and ask any question on MR3: > > https://join.slack.com/t/mr3-help/shared_invite/zt-1wpqztk35-AN8JRDznTkvxFIjtvhmiNg > > Thank you, > > --- Sungwoo > > [1] https://celeborn.apache.org/ > [2] https://www.vldb.org/pvldb/vol13/p3382-shen.pdf > [3] https://uniffle.apache.org/ > [4] https://mr3docs.datamonad.com/docs/mr3/features/celeborn/ > [5] https://github.com/mr3project/mr3-release/releases/tag/v1.8 > [6] https://mr3docs.datamonad.com/ >
