[
https://issues.apache.org/jira/browse/SPARK-43100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711148#comment-17711148
]
Ye Zhou commented on SPARK-43100:
-
Created PR to fix the issue
Ye Zhou created SPARK-43100:
---
Summary: Mismatch of field name in log event writer and parser for
push shuffle metrics
Key: SPARK-43100
URL: https://issues.apache.org/jira/browse/SPARK-43100
Project: Spark
Ye Zhou created SPARK-38987:
---
Summary: Handle fallback when merged shuffle blocks are corrupted
and spark.shuffle.detectCorrupt is set to true
Key: SPARK-38987
URL: https://issues.apache.org/jira/browse/SPARK-38987
[ https://issues.apache.org/jira/browse/SPARK-33236 ]
Ye Zhou deleted comment on SPARK-33236:
-
was (Author: zhouyejoe):
WIP PR posted [https://github.com/apache/spark/pull/35906.]
> Enable Push-based shuffle service to store state in NM level DB
[
https://issues.apache.org/jira/browse/SPARK-33236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17508498#comment-17508498
]
Ye Zhou commented on SPARK-33236:
-
WIP PR posted [https://github.com/apache/spark/pull/35906.]
>
Ye Zhou created SPARK-37023:
---
Summary: Avoid fetching merge status when shuffleMergeEnabled is
false for a shuffleDependency during retry
Key: SPARK-37023
URL: https://issues.apache.org/jira/browse/SPARK-37023
[
https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422558#comment-17422558
]
Ye Zhou commented on SPARK-36892:
-
Raised PR [https://github.com/apache/spark/pull/34156.] UT to be
[
https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422425#comment-17422425
]
Ye Zhou commented on SPARK-36892:
-
I am working on this issue. We have a job which can reproduce this
[
https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415879#comment-17415879
]
Ye Zhou commented on SPARK-36772:
-
I will work on this one and post PR ASAP.
> FinalizeShuffleMerge
[
https://issues.apache.org/jira/browse/SPARK-36744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-36744:
Description:
As a follow up to SPARK-36705, push-based shuffle is not compatible with IO
encryption. We
[
https://issues.apache.org/jira/browse/SPARK-33573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-33573:
Description: Shuffle Server side metrics for push based shuffle. (was:
Need to add metrics on both
[
https://issues.apache.org/jira/browse/SPARK-33573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-33573:
Summary: Server side metrics related to push-based shuffle (was: Server
and client side metrics related
[
https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390184#comment-17390184
]
Ye Zhou commented on SPARK-30602:
-
[~Gengliang.Wang] This is the last PR(Subtask 9) to be merged
[
https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou resolved SPARK-35546.
-
Fix Version/s: 3.2.0
Target Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull
[
https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-35546:
Parent: SPARK-30602
Issue Type: Sub-task (was: Bug)
> Enable push-based shuffle when multiple
[
https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-35546:
Parent: (was: SPARK-33235)
Issue Type: Bug (was: Sub-task)
> Enable push-based shuffle when
[
https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-35546:
Summary: Properly handle race conditions in RemoteBlockPushResolver to
support push based shuffle with
[
https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-35546:
Summary: Properly handle race conditions in RemoteBlockPushResolver for
access to the internal
[
https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-35546:
Summary: Properly handle race conditions in RemoteBlockPushResolver for
access to the internal
Ye Zhou created SPARK-35548:
---
Summary: Handling new attempt has started error message in
BlockPushErrorHandler in client
Key: SPARK-35548
URL: https://issues.apache.org/jira/browse/SPARK-35548
Project:
Ye Zhou created SPARK-35546:
---
Summary: Handling race condition and memory leak in
RemoteBlockPushResolver
Key: SPARK-35546
URL: https://issues.apache.org/jira/browse/SPARK-35546
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-25634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637436#comment-16637436
]
Ye Zhou commented on SPARK-25634:
-
[~felixcheung] [~vanzin] [~tgraves] [~irashid] [~zsxwing] More
Ye Zhou created SPARK-25634:
---
Summary: New Metrics in External Shuffle Service to help identify
abusing application
Key: SPARK-25634
URL: https://issues.apache.org/jira/browse/SPARK-25634
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou resolved SPARK-21961.
-
Resolution: Won't Fix
> Filter out BlockStatuses Accumulators during replaying history logs in Spark
>
[
https://issues.apache.org/jira/browse/SPARK-23607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396193#comment-16396193
]
Ye Zhou commented on SPARK-23607:
-
[~vanzin] Cool. I will post a PR soon. Thanks.
> Use HDFS extended
[
https://issues.apache.org/jira/browse/SPARK-23608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387175#comment-16387175
]
Ye Zhou commented on SPARK-23608:
-
Pull Request: https://github.com/apache/spark/pull/20744
> SHS needs
[
https://issues.apache.org/jira/browse/SPARK-23608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387162#comment-16387162
]
Ye Zhou commented on SPARK-23608:
-
I will post a pull request for this minor change.
[~vanzin]
Ye Zhou created SPARK-23608:
---
Summary: SHS needs synchronization between attachSparkUI and
detachSparkUI functions
Key: SPARK-23608
URL: https://issues.apache.org/jira/browse/SPARK-23608
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-23607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387126#comment-16387126
]
Ye Zhou edited comment on SPARK-23607 at 3/6/18 1:42 AM:
-
[~vanzin] [~zsxwing]
[
https://issues.apache.org/jira/browse/SPARK-23607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-23607:
Shepherd: (was: Marcelo Vanzin)
> Use HDFS extended attributes to store application summary to improve
[
https://issues.apache.org/jira/browse/SPARK-23607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387126#comment-16387126
]
Ye Zhou commented on SPARK-23607:
-
[~vanzin] Any comments? Thanks.
> Use HDFS extended attributes to
Ye Zhou created SPARK-23607:
---
Summary: Use HDFS extended attributes to store application summary
to improve the Spark History Server performance
Key: SPARK-23607
URL: https://issues.apache.org/jira/browse/SPARK-23607
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338378#comment-16338378
]
Ye Zhou commented on SPARK-23206:
-
[~zsxwing] Hi, Can you help find some one who can help review this
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172162#comment-16172162
]
Ye Zhou commented on SPARK-21961:
-
[~zsxwing] Can you help to take a look? Thanks.
> Filter out
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159514#comment-16159514
]
Ye Zhou commented on SPARK-21961:
-
Pull Request Added: https://github.com/apache/spark/pull/19170
>
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21961:
Description:
As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of
memory in
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21961:
Description:
As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of
memory in
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21961:
Description:
As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of
memory in
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21961:
Description:
As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of
memory in
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21961:
Description:
As described in SPARK-20923, TaskMetrics._updatedBlockStatuses uses a lot of
memory in
[
https://issues.apache.org/jira/browse/SPARK-21961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21961:
Attachment: Objects_Count_in_Heap.png
One_Thread_Took_24GB.png
> Filter out BlockStatuses
Ye Zhou created SPARK-21961:
---
Summary: Filter out BlockStatuses Accumulators during replaying
history logs in Spark History Server
Key: SPARK-21961
URL: https://issues.apache.org/jira/browse/SPARK-21961
[
https://issues.apache.org/jira/browse/SPARK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126398#comment-16126398
]
Ye Zhou commented on SPARK-21715:
-
Pull Request: https://github.com/apache/spark/pull/18941
> History
[
https://issues.apache.org/jira/browse/SPARK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21715:
Summary: History Server should not respond history page html content
multiple times for only one http
[
https://issues.apache.org/jira/browse/SPARK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21715:
Attachment: ResponseContent.png
> History Server respondes history page html content multiple times for
[
https://issues.apache.org/jira/browse/SPARK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21715:
Description:
UI looks fine for the home page. But we check the performance for each
individual
[
https://issues.apache.org/jira/browse/SPARK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21715:
Description:
UI looks fine for the home page. But we check the performance for each
individual
[
https://issues.apache.org/jira/browse/SPARK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21715:
Description:
UI looks fine for the home page. But we check the performance for each
individual
[
https://issues.apache.org/jira/browse/SPARK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Zhou updated SPARK-21715:
Attachment: Performance.png
> History Server respondes history page html content multiple times for only
Ye Zhou created SPARK-21715:
---
Summary: History Server respondes history page html content
multiple times for only one http request
Key: SPARK-21715
URL: https://issues.apache.org/jira/browse/SPARK-21715
[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084451#comment-16084451
]
Ye Zhou commented on SPARK-18085:
-
I want to add my own testing experience with the codes from the HEAD
51 matches
Mail list logo