Hi Rory, Let me express my gratitude and positive impression of Uniffle. Actually, we also feel the necessity of a shuffle service for our Hive deployment, and I've been watching the project. I will check the implementation for Tez and send feedback or PRs if I find something.
Regards, Okumin On Tue, Jul 11, 2023 at 9:48 PM roryqi <ror...@apache.org> wrote: > Dear Apache Hive community, > > > We are delighted to announce the support of Tez on Uniffle. Uniffle havs > supported Apache Spark, Apache,Hadoop MapReduce and Apache Tez. > > Uniffle is a remote shuffle service. In several situations, Uniffle will > provide great help. > > 1. If you use AWS spot instances or mix resources, tasks may be > preempted. It will be great if we store shuffle data in the Uniffle and > we > can deploy Uniffle on some stable resource. It will improve the > stability > of tasks. If tasks are preempted, we won’t recompute tasks if we store > shuffle in the Uniffle. > 2. For large shuffle jobs, Uniffle can reduce random IO for the jobs. > Uniffle can improve the performance of jobs. For 1TB MapReduce > Terasort, 1w > map tasks, 1w reduce tasks, job performance will increase 30%. > > We also welcome pull requests and are eager to see how you might use > Uniffle to make Hive more user-friendly. More information, you can access > https://github.com/apache/incubator-uniffle > > > Best > > Rory >
