[GitHub] [incubator-uniffle] kaijchen opened a new issue, #507: Flaky Test: ShuffleBufferManagerTest#cacheShuffleDataTest

2023-01-20 Thread GitBox
kaijchen opened a new issue, #507: URL: https://github.com/apache/incubator-uniffle/issues/507 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searc

[GitHub] [incubator-uniffle] zuston closed issue #95: [Performance Optimization] Multiple channels when getting shuffle data in client side

2023-01-19 Thread GitBox
zuston closed issue #95: [Performance Optimization] Multiple channels when getting shuffle data in client side URL: https://github.com/apache/incubator-uniffle/issues/95 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [incubator-uniffle] zuston closed issue #198: [Problem] Inconsistent blocks when reading shuffle data

2023-01-19 Thread GitBox
zuston closed issue #198: [Problem] Inconsistent blocks when reading shuffle data URL: https://github.com/apache/incubator-uniffle/issues/198 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [incubator-uniffle] advancedxy commented on issue #503: [Bug] Frequent and long lasting full GCs when there are too many partitions on a shuffle server

2023-01-18 Thread GitBox
advancedxy commented on issue #503: URL: https://github.com/apache/incubator-uniffle/issues/503#issuecomment-1396538481 > from the metric dashboard, during peek hours, there are 20k partitions in a shuffle-server, but disk used capacity ratio is 0.1-0.2 If possible, could you share m

[GitHub] [incubator-uniffle] advancedxy commented on issue #503: [Bug] Frequent and long lasting full GCs when there are too many partitions on a shuffle server

2023-01-18 Thread GitBox
advancedxy commented on issue #503: URL: https://github.com/apache/incubator-uniffle/issues/503#issuecomment-1396537726 > 1. Garbage collector changes to CMS > 2. Expand the uniffle cluster by adding more shuffle-servers > 3. If one shuffle-server has partition number exceeding the thr

[GitHub] [incubator-uniffle] zuston commented on issue #503: [Bug] Frequent and long lasting full GCs when there are too many partitions on a shuffle server

2023-01-18 Thread GitBox
zuston commented on issue #503: URL: https://github.com/apache/incubator-uniffle/issues/503#issuecomment-1396503732 @leixm @xianjingfeng Do you encounter similar problems? cc @advancedxy @jerqi -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [incubator-uniffle] zuston opened a new issue, #503: [Bug] Frequent and long lasting full GCs when there are too many partitions on a shuffle server

2023-01-18 Thread GitBox
zuston opened a new issue, #503: URL: https://github.com/apache/incubator-uniffle/issues/503 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] kaijchen commented on issue #502: Flaky Test: QuorumTest#case5

2023-01-18 Thread GitBox
kaijchen commented on issue #502: URL: https://github.com/apache/incubator-uniffle/issues/502#issuecomment-1396396718 @zuston thanks for reporting this issue. I'm not sure what's happened, have you seen this failure before #492? https://user-images.githubusercontent.com/5821159/213349218

[GitHub] [incubator-uniffle] zuston commented on issue #502: Flaky Test: QuorumTest#case5

2023-01-18 Thread GitBox
zuston commented on issue #502: URL: https://github.com/apache/incubator-uniffle/issues/502#issuecomment-1396388475 This pr's test failed. https://github.com/apache/incubator-uniffle/actions/runs/3954858017 cc @kaijchen -- This is an automated message from the Apache Git Service. To re

[GitHub] [incubator-uniffle] zuston opened a new issue, #502: Flaky Test: QuorumTest#case5

2023-01-18 Thread GitBox
zuston opened a new issue, #502: URL: https://github.com/apache/incubator-uniffle/issues/502 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] zuston commented on issue #497: [Improvement] Rethink the config of 'rss.server.write.slow.threshold'

2023-01-18 Thread GitBox
zuston commented on issue #497: URL: https://github.com/apache/incubator-uniffle/issues/497#issuecomment-1396385846 > However we should approach this with a compatible way. something like: Yes. Currently Uniffle don't introduce the deprecated annotation shown in the doc yet. Let's o

[GitHub] [incubator-uniffle] advancedxy commented on issue #497: [Improvement] Rethink the config of 'rss.server.write.slow.threshold'

2023-01-18 Thread GitBox
advancedxy commented on issue #497: URL: https://github.com/apache/incubator-uniffle/issues/497#issuecomment-1396370660 > writing speed +1. However we should approach this with a compatible way. something like: 1. introduce new configuration item: `slowSpeed. threshold` and deprec

[GitHub] [incubator-uniffle] zuston commented on issue #498: [Improvement] Extend event size threshold metric configuration to any number of levels

2023-01-18 Thread GitBox
zuston commented on issue #498: URL: https://github.com/apache/incubator-uniffle/issues/498#issuecomment-1396369309 > So, you want to generate metric name dynamically, something like rss.server.event.size.threshold.{20k,1g}? Not accurate. Metrics will be exposed like: event_size_rang

[GitHub] [incubator-uniffle] advancedxy commented on issue #498: [Improvement] Extend event size threshold metric configuration to any number of levels

2023-01-18 Thread GitBox
advancedxy commented on issue #498: URL: https://github.com/apache/incubator-uniffle/issues/498#issuecomment-1396367926 > rss.server.event.size.threshold.level.ranges=20k,1m,10m,100m,1g,10g So, you want to generate metric name dynamically, something like `rss.server.event.size.thresh

[GitHub] [incubator-uniffle] advancedxy commented on issue #496: [Subtask][operator] set init-container's resource request

2023-01-18 Thread GitBox
advancedxy commented on issue #496: URL: https://github.com/apache/incubator-uniffle/issues/496#issuecomment-1386801023 > Do you mean uniffle should set the default resource request? like that > > ![image](https://user-images.githubusercontent.com/8609142/213128323-dd4a05d3-24de-4b23

[GitHub] [incubator-uniffle] zuston opened a new issue, #498: [Improvement] Extend event size threshold metric configuration to any number of levels

2023-01-18 Thread GitBox
zuston opened a new issue, #498: URL: https://github.com/apache/incubator-uniffle/issues/498 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] zuston opened a new issue, #497: [Improvement] Rethink the metric of 'rss.server.write.slow.threshold'

2023-01-18 Thread GitBox
zuston opened a new issue, #497: URL: https://github.com/apache/incubator-uniffle/issues/497 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] zuston commented on issue #496: [Subtask][operator] set init-container's resource request

2023-01-18 Thread GitBox
zuston commented on issue #496: URL: https://github.com/apache/incubator-uniffle/issues/496#issuecomment-1386707714 Do you mean uniffle should set the default resource request? like that ![image](https://user-images.githubusercontent.com/8609142/213128323-dd4a05d3-24de-4b23-bec3-d996

[GitHub] [incubator-uniffle] advancedxy opened a new issue, #496: [Subtask][operator] set init-container's resource request

2023-01-17 Thread GitBox
advancedxy opened a new issue, #496: URL: https://github.com/apache/incubator-uniffle/issues/496 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have sea

[GitHub] [incubator-uniffle] advancedxy commented on issue #483: Flaky Test: ShuffleServerFaultToleranceTest#testReadFaultTolerance

2023-01-17 Thread GitBox
advancedxy commented on issue #483: URL: https://github.com/apache/incubator-uniffle/issues/483#issuecomment-1385083012 https://github.com/apache/incubator-uniffle/actions/runs/3937625492/jobs/6735376117 there's another instance about this case... I think maybe we need to priority t

[GitHub] [incubator-uniffle] advancedxy closed issue #418: [Improvement] Add until.CreateTempDir

2023-01-17 Thread GitBox
advancedxy closed issue #418: [Improvement] Add until.CreateTempDir URL: https://github.com/apache/incubator-uniffle/issues/418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [incubator-uniffle] zuston opened a new issue, #493: [Improvement] Replace putIfAbsent with computeIfAbsent in some critical paths

2023-01-16 Thread GitBox
zuston opened a new issue, #493: URL: https://github.com/apache/incubator-uniffle/issues/493 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] zuston closed issue #484: [Bug] HdfsStorageManager#appIdToStorages data lost when removing resources for partial shuffleIds

2023-01-16 Thread GitBox
zuston closed issue #484: [Bug] HdfsStorageManager#appIdToStorages data lost when removing resources for partial shuffleIds URL: https://github.com/apache/incubator-uniffle/issues/484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [incubator-uniffle] jerqi closed issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-16 Thread GitBox
jerqi closed issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter URL: https://github.com/apache/incubator-uniffle/issues/475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [incubator-uniffle] kaijchen closed issue #489: [Improvement] We can simplify and standardize some of the code

2023-01-16 Thread GitBox
kaijchen closed issue #489: [Improvement] We can simplify and standardize some of the code URL: https://github.com/apache/incubator-uniffle/issues/489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [incubator-uniffle] zuston commented on issue #483: Flaky Test: ShuffleServerFaultToleranceTest#testReadFaultTolerance

2023-01-16 Thread GitBox
zuston commented on issue #483: URL: https://github.com/apache/incubator-uniffle/issues/483#issuecomment-1383728781 I suspect the multiple test cases sequence will cause different test result rather than running multiple times for one test -- This is an automated message from the Apache

[GitHub] [incubator-uniffle] kaijchen commented on issue #483: Flaky Test: ShuffleServerFaultToleranceTest#testReadFaultTolerance

2023-01-16 Thread GitBox
kaijchen commented on issue #483: URL: https://github.com/apache/incubator-uniffle/issues/483#issuecomment-1383712879 It's weird that this test fails so often in CI. https://github.com/apache/incubator-uniffle/actions/runs/3928645075 But It's hard to reproduce it locally. https:

[GitHub] [incubator-uniffle] kaijchen closed issue #487: [Improvement] Java 8 syntax helps with code readability improvements

2023-01-16 Thread GitBox
kaijchen closed issue #487: [Improvement] Java 8 syntax helps with code readability improvements URL: https://github.com/apache/incubator-uniffle/issues/487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-uniffle] iwangjie opened a new issue, #489: [Improvement] 我们可以简化和规范一些代码

2023-01-15 Thread GitBox
iwangjie opened a new issue, #489: URL: https://github.com/apache/incubator-uniffle/issues/489 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searc

[GitHub] [incubator-uniffle] iwangjie opened a new issue, #487: [Improvement] Java 8 syntax helps with code readability improvements

2023-01-15 Thread GitBox
iwangjie opened a new issue, #487: URL: https://github.com/apache/incubator-uniffle/issues/487 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searc

[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.

2023-01-15 Thread GitBox
zuston commented on issue #391: URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1383558806 Any update on this? @yuyang733 cc @jerqi @advancedxy If object store is supported, I will use this to store huge partition to reduce HDFS pressure for iQiyi. And this

[GitHub] [incubator-uniffle] zuston opened a new issue, #484: [Bug] HdfsStorageManager#appIdToStorages data lost when removing resources for partial shuffleIds

2023-01-13 Thread GitBox
zuston opened a new issue, #484: URL: https://github.com/apache/incubator-uniffle/issues/484 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] zuston opened a new issue, #483: Flaky Test: ShuffleServerFaultToleranceTest#testReadFaultTolerance

2023-01-12 Thread GitBox
zuston opened a new issue, #483: URL: https://github.com/apache/incubator-uniffle/issues/483 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] jerqi commented on issue #479: [Improvement] [Operator] better build system support

2023-01-12 Thread GitBox
jerqi commented on issue #479: URL: https://github.com/apache/incubator-uniffle/issues/479#issuecomment-1381245599 > > 2 We would better not do this because binary image release also need a vote process to release the binary image. > > 4 It seems that we don't use alpine. > > For

[GitHub] [incubator-uniffle] advancedxy commented on issue #479: [Improvement] [Operator] better build system support

2023-01-12 Thread GitBox
advancedxy commented on issue #479: URL: https://github.com/apache/incubator-uniffle/issues/479#issuecomment-1381241762 > 2 We would better not do this because binary image release also need a vote process to release the binary image. > 4 It seems that we don't use alpine. For 2,

[GitHub] [incubator-uniffle] zjf2012 commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
zjf2012 commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1381212198 > The groupedPartitions is a local variable, it will never be used across threads? correct. let me fix it along with RssShuffleWriter. -- This is an automated messa

[GitHub] [incubator-uniffle] advancedxy commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
advancedxy commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1381210039 The groupedPartitions is a local variable, it will never be used across threads? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [incubator-uniffle] zjf2012 commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
zjf2012 commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1381200442 > I'm in favor of this. The following code path doesn't need a ConcurrentHashMap as well. > > https://github.com/apache/incubator-uniffle/blob/ddf83843a9fbce8155d1598

[GitHub] [incubator-uniffle] advancedxy opened a new issue, #479: [Improvement] [Operator] better build system support

2023-01-12 Thread GitBox
advancedxy opened a new issue, #479: URL: https://github.com/apache/incubator-uniffle/issues/479 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have sea

[GitHub] [incubator-uniffle] advancedxy opened a new issue, #477: [FEATURE] Support stage recompute for Spark clients

2023-01-12 Thread GitBox
advancedxy opened a new issue, #477: URL: https://github.com/apache/incubator-uniffle/issues/477 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have sea

[GitHub] [incubator-uniffle] advancedxy commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
advancedxy commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1380210362 I'm in favor of this. The following code path doesn't need a ConcurrentHashMap as well. https://github.com/apache/incubator-uniffle/blob/ddf83843a9fbce8155d159815c351

[GitHub] [incubator-uniffle] zuston commented on issue #476: [FEATURE] Respect "spark.shuffle.compress" configuration in Uniffle

2023-01-12 Thread GitBox
zuston commented on issue #476: URL: https://github.com/apache/incubator-uniffle/issues/476#issuecomment-1380200912 > I believe Uniffle uses org.apache.uniffle.common.compression.Codec.NOOP to disable compress. But it's never used in production and may have other hidden issues. cc @zuston.

[GitHub] [incubator-uniffle] advancedxy commented on issue #476: [FEATURE] Respect "spark.shuffle.compress" configuration in Uniffle

2023-01-12 Thread GitBox
advancedxy commented on issue #476: URL: https://github.com/apache/incubator-uniffle/issues/476#issuecomment-1380199120 I believe Uniffle uses `org.apache.uniffle.common.compression.Codec.NOOP` to disable compress. But it's never used in production and may have other hidden issues. cc @zus

[GitHub] [incubator-uniffle] jerqi commented on issue #476: [FEATURE] Respect "spark.shuffle.compress" configuration in Uniffle

2023-01-12 Thread GitBox
jerqi commented on issue #476: URL: https://github.com/apache/incubator-uniffle/issues/476#issuecomment-1380147177 > > Some compress algorithms which Uniffle supports may be different from Spark's. > > It's not related to Spark's compress alg. since rss shuffle manager are separated

[GitHub] [incubator-uniffle] zjf2012 commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
zjf2012 commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1380081188 > If there is no thread conflicts, the clock operation only cost ns. maybe. There is other potential side-effect, like flushing CPU buffer to memory due to lock involv

[GitHub] [incubator-uniffle] zjf2012 commented on issue #476: [FEATURE] Respect "spark.shuffle.compress" configuration in Uniffle

2023-01-12 Thread GitBox
zjf2012 commented on issue #476: URL: https://github.com/apache/incubator-uniffle/issues/476#issuecomment-1380051963 > Some compress algorithms which Uniffle supports may be different from Spark's. It's not related to Spark's compress alg. since rss shuffle manager are separated fro

[GitHub] [incubator-uniffle] jerqi commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
jerqi commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1380051443 If there is no thread conflicts, the clock operation only cost ns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [incubator-uniffle] zjf2012 commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
zjf2012 commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1380040161 > Will it bring too much cost? I cannot tell how much cost it is. But you can imagine we need to get lock first before each map operation in each shuffle write partiti

[GitHub] [incubator-uniffle] jerqi commented on issue #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-12 Thread GitBox
jerqi commented on issue #475: URL: https://github.com/apache/incubator-uniffle/issues/475#issuecomment-1380032446 Will it bring too much cost? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [incubator-uniffle] jerqi commented on issue #476: [FEATURE] Respect "spark.shuffle.compress" configuration in Uniffle

2023-01-12 Thread GitBox
jerqi commented on issue #476: URL: https://github.com/apache/incubator-uniffle/issues/476#issuecomment-1380031918 Some compress algorithms which Uniffle supports may be different from Spark's. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [incubator-uniffle] zjf2012 opened a new issue, #476: [FEATURE] Respect "spark.shuffle.compress" configuration in Uniffle

2023-01-12 Thread GitBox
zjf2012 opened a new issue, #476: URL: https://github.com/apache/incubator-uniffle/issues/476 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have search

[GitHub] [incubator-uniffle] kaijchen closed issue #472: Flaky Test: LocalFileServerReadHandlerTest#testDataInconsistent

2023-01-11 Thread GitBox
kaijchen closed issue #472: Flaky Test: LocalFileServerReadHandlerTest#testDataInconsistent URL: https://github.com/apache/incubator-uniffle/issues/472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [incubator-uniffle] kaijchen commented on issue #472: Flaky Test: LocalFileServerReadHandlerTest#testDataInconsistent

2023-01-11 Thread GitBox
kaijchen commented on issue #472: URL: https://github.com/apache/incubator-uniffle/issues/472#issuecomment-1379925945 This is due to `LocalFileHandlerTest` and `LocalFileServerReadHandlerTest` sharing a common state in `LocalFileHandlerTestBase#ATOMIC_LONG` -- This is an automated messag

[GitHub] [incubator-uniffle] zjf2012 opened a new issue, #475: [Improvement] It's unnecessary to use ConcurrentHashMap for "partitionToBlockIds" in RssShuffleWriter

2023-01-11 Thread GitBox
zjf2012 opened a new issue, #475: URL: https://github.com/apache/incubator-uniffle/issues/475 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have search

[GitHub] [incubator-uniffle] xianjingfeng closed issue #108: [Improvement] Read shuffle data fail because read index file fail

2023-01-11 Thread GitBox
xianjingfeng closed issue #108: [Improvement] Read shuffle data fail because read index file fail URL: https://github.com/apache/incubator-uniffle/issues/108 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [incubator-uniffle] xianjingfeng commented on issue #108: [Improvement] Read shuffle data fail because read index file fail

2023-01-11 Thread GitBox
xianjingfeng commented on issue #108: URL: https://github.com/apache/incubator-uniffle/issues/108#issuecomment-1379900278 Fix by #213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [incubator-uniffle] zuston opened a new issue, #472: Flaky Test: LocalFileServerReadHandlerTest#testDataInconsistent

2023-01-11 Thread GitBox
zuston opened a new issue, #472: URL: https://github.com/apache/incubator-uniffle/issues/472 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searche

[GitHub] [incubator-uniffle] xianjingfeng commented on issue #468: [Improvement] Put unavailable shuffle servers to the end of the server list when sending shuffle data

2023-01-10 Thread GitBox
xianjingfeng commented on issue #468: URL: https://github.com/apache/incubator-uniffle/issues/468#issuecomment-1378297602 > > If we use multiple replicas and the first shuffle server becomes unavailable > > `shuffle server becomes unavailable` means the shuffle server goes down or t

[GitHub] [incubator-uniffle] wangao1236 opened a new issue, #469: [FEATURE] Operator supports adding labels to rss pods.

2023-01-10 Thread GitBox
wangao1236 opened a new issue, #469: URL: https://github.com/apache/incubator-uniffle/issues/469 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have sea

[GitHub] [incubator-uniffle] advancedxy commented on issue #468: [Improvement] Put unavailable shuffle servers to the end of the server list when sending shuffle data

2023-01-10 Thread GitBox
advancedxy commented on issue #468: URL: https://github.com/apache/incubator-uniffle/issues/468#issuecomment-1378206507 > If we use multiple replicas and the first shuffle server becomes unavailable `shuffle server becomes unavailable` means the shuffle server goes down or the memo

[GitHub] [incubator-uniffle] xianjingfeng opened a new issue, #468: [Improvement] Put unavailable shuffle servers to the end of the server list when sending shuffle data

2023-01-10 Thread GitBox
xianjingfeng opened a new issue, #468: URL: https://github.com/apache/incubator-uniffle/issues/468 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have s

[GitHub] [incubator-uniffle] jerqi closed issue #448: [Subtask] Shuffle server shall report its local storage capability

2023-01-10 Thread GitBox
jerqi closed issue #448: [Subtask] Shuffle server shall report its local storage capability URL: https://github.com/apache/incubator-uniffle/issues/448 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [incubator-uniffle] kaijchen opened a new issue, #465: [FEATURE] Dynamic allocation support for Spark 3.3

2023-01-10 Thread GitBox
kaijchen opened a new issue, #465: URL: https://github.com/apache/incubator-uniffle/issues/465 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searc

[GitHub] [incubator-uniffle] kaijchen closed issue #461: [FEATURE] Spark 3.3 support

2023-01-10 Thread GitBox
kaijchen closed issue #461: [FEATURE] Spark 3.3 support URL: https://github.com/apache/incubator-uniffle/issues/461 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

[GitHub] [incubator-uniffle] kaijchen commented on issue #461: [FEATURE] Spark 3.3 support

2023-01-10 Thread GitBox
kaijchen commented on issue #461: URL: https://github.com/apache/incubator-uniffle/issues/461#issuecomment-1377084921 Closed by #463. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [incubator-uniffle] zuston commented on issue #456: [Improvement] Avoid removeResources for multiple times

2023-01-09 Thread GitBox
zuston commented on issue #456: URL: https://github.com/apache/incubator-uniffle/issues/456#issuecomment-1376646155 Fixed by #459 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [incubator-uniffle] zuston closed issue #456: [Improvement] Avoid removeResources for multiple times

2023-01-09 Thread GitBox
zuston closed issue #456: [Improvement] Avoid removeResources for multiple times URL: https://github.com/apache/incubator-uniffle/issues/456 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [incubator-uniffle] advancedxy commented on issue #378: [Improvement] Optimize data flushing and memory usage for huge partitions to improve stability

2023-01-09 Thread GitBox
advancedxy commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1376624355 > This issue will track all the optimizations of huge partition, and all sub-tasks will be connected with this. The solution of handling huge partitions is to make it flu

[GitHub] [incubator-uniffle] advancedxy commented on issue #462: [Umbrella] Better K8S operator support

2023-01-09 Thread GitBox
advancedxy commented on issue #462: URL: https://github.com/apache/incubator-uniffle/issues/462#issuecomment-1375724144 cc @wangao1236 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [incubator-uniffle] advancedxy opened a new issue, #462: [Umbrella] Better K8S operator support

2023-01-09 Thread GitBox
advancedxy opened a new issue, #462: URL: https://github.com/apache/incubator-uniffle/issues/462 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have sea

[GitHub] [incubator-uniffle] kaijchen commented on issue #461: [FEATURE] Spark 3.3 support

2023-01-09 Thread GitBox
kaijchen commented on issue #461: URL: https://github.com/apache/incubator-uniffle/issues/461#issuecomment-1375485900 I'm not sure if I added the profile correctly, seems spark 3.3 requires some test code change: https://github.com/kaijchen/incubator-uniffle/actions/runs/3873117861/jobs

[GitHub] [incubator-uniffle] kaijchen commented on issue #461: [FEATURE] Spark 3.3 support

2023-01-09 Thread GitBox
kaijchen commented on issue #461: URL: https://github.com/apache/incubator-uniffle/issues/461#issuecomment-1375466211 Plus: we may need to create a patch for spark 3.3 in https://github.com/apache/incubator-uniffle/tree/master/spark-patches -- This is an automated message from the Apache

[GitHub] [incubator-uniffle] zuston commented on issue #460: [Improvement] Exit on OutOfMemoryError

2023-01-09 Thread GitBox
zuston commented on issue #460: URL: https://github.com/apache/incubator-uniffle/issues/460#issuecomment-1375423608 > The bin/start-shuffle-server.sh may handle that as a cli argument or env variable. Good idea. This should be supported. -- This is an automated message from the Ap

[GitHub] [incubator-uniffle] zuston commented on issue #461: [FEATURE] Spark 3.3 support

2023-01-09 Thread GitBox
zuston commented on issue #461: URL: https://github.com/apache/incubator-uniffle/issues/461#issuecomment-1375422440 Make sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [incubator-uniffle] zuston commented on issue #80: [Feature Request] Support shuffle server decommissioned

2023-01-09 Thread GitBox
zuston commented on issue #80: URL: https://github.com/apache/incubator-uniffle/issues/80#issuecomment-1375420047 > Design doc: https://docs.google.com/document/d/1p1PksBN2LJ-OtGEHvdyEuH9b1Mv1aD_exMPl4TNaTs0/edit?usp=sharing PTAL @jerqi @zuston Commented. @xianjingfeng PTAL -- Th

[GitHub] [incubator-uniffle] zuston commented on issue #80: [Feature Request] Support shuffle server decommissioned

2023-01-09 Thread GitBox
zuston commented on issue #80: URL: https://github.com/apache/incubator-uniffle/issues/80#issuecomment-1375397876 Thanks a lot for proposing this, I will take a look ASSP -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [incubator-uniffle] xianjingfeng commented on issue #80: [Feature Request] Support shuffle server decommissioned

2023-01-09 Thread GitBox
xianjingfeng commented on issue #80: URL: https://github.com/apache/incubator-uniffle/issues/80#issuecomment-1375395560 Design doc: https://docs.google.com/document/d/1p1PksBN2LJ-OtGEHvdyEuH9b1Mv1aD_exMPl4TNaTs0/edit?usp=sharing PTAL @jerqi @zuston -- This is an automated message fro

[GitHub] [incubator-uniffle] kaijchen commented on issue #461: [FEATURE] Spark 3.3 support

2023-01-09 Thread GitBox
kaijchen commented on issue #461: URL: https://github.com/apache/incubator-uniffle/issues/461#issuecomment-1375305375 > Would you like to submit a PR here? Or do you encounter some issues? I'm not sure. > I was expected we can simply use spark 3.2 support to run spark 3.3 apps

[GitHub] [incubator-uniffle] advancedxy commented on issue #461: [FEATURE] Spark 3.3 support

2023-01-09 Thread GitBox
advancedxy commented on issue #461: URL: https://github.com/apache/incubator-uniffle/issues/461#issuecomment-1375301393 Would you like to submit a PR here? Or do you encounter some issues? I was expected we can simply use spark 3.2 support to run spark 3.3 apps as Spark 3.3 doesn't c

[GitHub] [incubator-uniffle] advancedxy commented on issue #460: [Improvement] Exit on OutOfMemoryError

2023-01-08 Thread GitBox
advancedxy commented on issue #460: URL: https://github.com/apache/incubator-uniffle/issues/460#issuecomment-1375149205 > > > PTAL @jerqi @zuston @advancedxy > > > > > > Do we need dump the memory when OOM occurs? > > The dump file maybe too big and disk space may be not en

[GitHub] [incubator-uniffle] xianjingfeng commented on issue #460: [Improvement] Exit on OutOfMemoryError

2023-01-08 Thread GitBox
xianjingfeng commented on issue #460: URL: https://github.com/apache/incubator-uniffle/issues/460#issuecomment-1375139463 > > PTAL @jerqi @zuston @advancedxy > > Do we need dump the memory when OOM occurs? The dump file maybe too big and disk space may be not enough to store it

[GitHub] [incubator-uniffle] kaijchen opened a new issue, #461: [FEATURE] Spark 3.3 support

2023-01-08 Thread GitBox
kaijchen opened a new issue, #461: URL: https://github.com/apache/incubator-uniffle/issues/461 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searc

[GitHub] [incubator-uniffle] jerqi commented on issue #460: [Improvement] Exit on OutOfMemoryError

2023-01-08 Thread GitBox
jerqi commented on issue #460: URL: https://github.com/apache/incubator-uniffle/issues/460#issuecomment-1375067843 > PTAL @jerqi @zuston @advancedxy Do we need dump the memory when OOM occurs。 -- This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [incubator-uniffle] xianjingfeng commented on issue #460: [Improvement] Exit on OutOfMemoryError

2023-01-08 Thread GitBox
xianjingfeng commented on issue #460: URL: https://github.com/apache/incubator-uniffle/issues/460#issuecomment-1375065814 PTAL @jerqi @zuston @advancedxy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-uniffle] xianjingfeng opened a new issue, #460: [Improvement] Exit on OutOfMemoryError

2023-01-08 Thread GitBox
xianjingfeng opened a new issue, #460: URL: https://github.com/apache/incubator-uniffle/issues/460 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have s

[GitHub] [incubator-uniffle] jerqi commented on issue #378: [Improvement] Optimize data flushing and memory usage for huge partitions to improve stability

2023-01-06 Thread GitBox
jerqi commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1373483327 > This issue will track all the optimizations of huge partition, and all sub-tasks will be connected with this. The solution of handling huge partitions is to make it flush to

[GitHub] [incubator-uniffle] jerqi closed issue #455: [Improvement] Lazily create `uncompressedData`

2023-01-06 Thread GitBox
jerqi closed issue #455: [Improvement] Lazily create `uncompressedData` URL: https://github.com/apache/incubator-uniffle/issues/455 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[GitHub] [incubator-uniffle] jerqi commented on issue #455: [Improvement] Lazily create `uncompressedData`

2023-01-06 Thread GitBox
jerqi commented on issue #455: URL: https://github.com/apache/incubator-uniffle/issues/455#issuecomment-1373458127 closed by #457 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [incubator-uniffle] zuston commented on issue #378: [Improvement] Introduce partition size based strategy to flush single huge partition data to HDFS

2023-01-06 Thread GitBox
zuston commented on issue #378: URL: https://github.com/apache/incubator-uniffle/issues/378#issuecomment-1373447729 This issue will track all the optimizations of huge partition, and all sub-tasks will be connected with this. The solution of handling huge partitions is to make it flush to

[GitHub] [incubator-uniffle] zuston commented on issue #456: [Improvement] Avoid removeResources for multiple times

2023-01-06 Thread GitBox
zuston commented on issue #456: URL: https://github.com/apache/incubator-uniffle/issues/456#issuecomment-1373421872 Nice catch, could you help fix this? If not, you could give some points to fix and mark it as `good first issue`. -- This is an automated message from the Apache Git Servi

[GitHub] [incubator-uniffle] zuston commented on issue #313: [Improvement] Support ZSTD in the spark2.x version

2023-01-05 Thread GitBox
zuston commented on issue #313: URL: https://github.com/apache/incubator-uniffle/issues/313#issuecomment-1372010859 This problem will happen when enable ZSTD compression of uniffle in Spark 2.x, which is caused by the incompatible interface of different version ZSTD. In uniffle, ZSTD versi

[GitHub] [incubator-uniffle] a-li commented on issue #313: [Improvement] Support ZSTD in the spark2.x version

2023-01-04 Thread GitBox
a-li commented on issue #313: URL: https://github.com/apache/incubator-uniffle/issues/313#issuecomment-1371582785 I want to give this a shot. @zuston Could you provide me with some tips/pointers to get started? In addition, how do I go about manually testing this? -- This is an automate

[GitHub] [incubator-uniffle] zuston commented on issue #455: [Improvement] Lazily create `uncompressedData`

2023-01-04 Thread GitBox
zuston commented on issue #455: URL: https://github.com/apache/incubator-uniffle/issues/455#issuecomment-1370745842 Nice catch, could you help fix this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [incubator-uniffle] xianjingfeng opened a new issue, #456: [Improvement] Avoid removeResources for multiple times

2023-01-04 Thread GitBox
xianjingfeng opened a new issue, #456: URL: https://github.com/apache/incubator-uniffle/issues/456 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have s

[GitHub] [incubator-uniffle] xianjingfeng commented on issue #455: [Improvement] Lazily create `uncompressedData`

2023-01-04 Thread GitBox
xianjingfeng commented on issue #455: URL: https://github.com/apache/incubator-uniffle/issues/455#issuecomment-1370632006 PTAL @zuston -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [incubator-uniffle] xianjingfeng opened a new issue, #455: [Improvement] Lazily create `uncompressedData`

2023-01-04 Thread GitBox
xianjingfeng opened a new issue, #455: URL: https://github.com/apache/incubator-uniffle/issues/455 ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have s

[GitHub] [incubator-uniffle] advancedxy commented on issue #448: [Subtask] Shuffle server shall report its local storage capability

2023-01-03 Thread GitBox
advancedxy commented on issue #448: URL: https://github.com/apache/incubator-uniffle/issues/448#issuecomment-1370542550 Related design doc: https://docs.google.com/document/d/1JcXJxwmJWCtNMWb8bS9LdcKz2x7kqSyxDjKt3BcPZmk/edit?usp=sharing -- This is an automated message from the Apache Git

[GitHub] [incubator-uniffle] zuston closed issue #451: [Improvement] Read HDFS data files with random sequence to distribute pressure

2023-01-03 Thread GitBox
zuston closed issue #451: [Improvement] Read HDFS data files with random sequence to distribute pressure URL: https://github.com/apache/incubator-uniffle/issues/451 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [incubator-uniffle] jerqi closed issue #426: [Bug] preAllocatedSize could be negative in corner cases

2022-12-30 Thread GitBox
jerqi closed issue #426: [Bug] preAllocatedSize could be negative in corner cases URL: https://github.com/apache/incubator-uniffle/issues/426 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [incubator-uniffle] jerqi commented on issue #426: [Bug] preAllocatedSize could be negative in corner cases

2022-12-30 Thread GitBox
jerqi commented on issue #426: URL: https://github.com/apache/incubator-uniffle/issues/426#issuecomment-1367811787 closed by #428 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

  1   2   3   4   5   6   7   8   9   10   >