GitHub user FMX added a comment to the discussion: Not Seeing Speedup with Celeborn
@lioryankovitz Hi, this usually happens when your clusters are not busy enough because Celeborn will optimize the shuffle IO pattern and gain time benefits. Your workload isn't harsh enough to create IO bottlenecks. There is no need to add extra hardware, but you'll need to increase the IO pressure of your tests. Perhaps you can try using terasort with a large number of partitions (30k+ or more), and then you should see a noticeable time difference. You are welcome to join our Slack channel if convenient. IM communications will be more efficient. GitHub link: https://github.com/apache/celeborn/discussions/3400#discussioncomment-13932112 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
