[GitHub] [incubator-uniffle] zuston commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

zuston commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1343970165 > I feel that it's necessary. Because it will be too slow that the server only use single thread to flush a large partition although we have multiple servers. Make

[GitHub] [incubator-uniffle] jerqi commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

jerqi commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1343904586 > > I mean that we can write multiple files at the same time. > > We can tryLock the file lock, we have multiple locks, if we succeed to tryLock, we can write the file. If

[GitHub] [incubator-uniffle] jerqi commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

jerqi commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1343770114 > > I mean that we can write multiple files at the same time. > > We can tryLock the file lock, we have multiple locks, if we succeed to tryLock, we can write the file. If

[GitHub] [incubator-uniffle] zuston commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

zuston commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1343768515 > I mean that we can write multiple files at the same time. We can tryLock the file lock, we have multiple locks, if we succeed to tryLock, we can write the file. If we

[GitHub] [incubator-uniffle] jerqi commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

jerqi commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1343762784 > > Maybe we could introduce multi-thread writing HDFS. If the file is too big, we could split them to multiple files. > > Yes. The key of problem is the low speed of

[GitHub] [incubator-uniffle] zuston commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

zuston commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1343746781 > Maybe we could introduce multi-thread writing HDFS. If the file is too big, we could split them to multiple files. Yes. The key of problem is the low speed of

[GitHub] [incubator-uniffle] jerqi commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

jerqi commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1342670082 Maybe we could introduce multi-thread writing HDFS. If the file is too big, we could split them to multiple files. ByteDance CSS have similar concept. If file exceed the

Re: [DISCUSS] Do we need a online meeting to discuss future plan about Uniffle

Our online meeting link is https://meeting.tencent.com/dm/Vu30COk8v1ky zhang junfan 于2022年12月5日周一 22:50写道： > > Please send me the invitation. > > Thanks > > > 获取 Outlook for iOS > > 发件人: roryqi > 发送时间: Monday, December 5, 2022 10:21:41 PM

Re: [DISCUSS] Do we need a online meeting to discuss future plan about Uniffle

Our online meeting link is https://meeting.tencent.com/dm/Vu30COk8v1ky 雷献铭 Jerry Lei 于2022年12月6日周二 01:45写道： > Please send me the invitation. > > Thanks > > On 2022/12/05 14:21:41 roryqi wrote: > > Hi, Uniffle Community! > > > > It's almost the end of the year. We're preparing the plans of

[GitHub] [incubator-uniffle] zuston commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

zuston commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1342637715 > In our production environment the disk is enough in most of the time. And it is difficult to set the threshold of partition size, and this will result in reduced disk

[GitHub] [incubator-uniffle] zuston commented on issue #380: [Bug] Overlook fallback when data-flush event enters into pending queue

zuston commented on issue #380: URL: https://github.com/apache/incubator-uniffle/issues/380#issuecomment-1342588509 Fixed by #383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [incubator-uniffle] zuston closed issue #380: [Bug] Overlook fallback when data-flush event enters into pending queue

zuston closed issue #380: [Bug] Overlook fallback when data-flush event enters into pending queue URL: https://github.com/apache/incubator-uniffle/issues/380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-uniffle] zuston closed issue #392: [Bug] Cleanup the data of localStorageChecker's file to cause shuffle-server unhealthy

zuston closed issue #392: [Bug] Cleanup the data of localStorageChecker's file to cause shuffle-server unhealthy URL: https://github.com/apache/incubator-uniffle/issues/392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [incubator-uniffle] zuston commented on issue #392: [Bug] Cleanup the data of localStorageChecker's file to cause shuffle-server unhealthy

zuston commented on issue #392: URL: https://github.com/apache/incubator-uniffle/issues/392#issuecomment-1342582820 Fixed by #393 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific