Zhongwei Zhu created SPARK-46352:
Summary: Support spark conf to configure log level of specific
package or class
Key: SPARK-46352
URL: https://issues.apache.org/jira/browse/SPARK-46352
Project:
Zhongwei Zhu created SPARK-45622:
Summary: java -target should use java.version instead of 17
Key: SPARK-45622
URL: https://issues.apache.org/jira/browse/SPARK-45622
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-41954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-41954:
-
Description: When one executor is dead, we want to know whether this dead
executor is caused by
[
https://issues.apache.org/jira/browse/SPARK-45372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-45372:
-
Summary: Handle ClassNotFoundException when load extension (was: Handle
ClassNotFound when
Zhongwei Zhu created SPARK-45372:
Summary: Handle ClassNotFound when load extension
Key: SPARK-45372
URL: https://issues.apache.org/jira/browse/SPARK-45372
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-45217:
Summary: Support change log level of specific package or class
Key: SPARK-45217
URL: https://issues.apache.org/jira/browse/SPARK-45217
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-45057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-45057:
-
Description:
When 2 tasks try to compute same rdd with replication level of 2 and running on
[
https://issues.apache.org/jira/browse/SPARK-45057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-45057:
-
Description:
When 2 tasks try to compute same rdd with replication level of 2 and running on
Zhongwei Zhu created SPARK-45057:
Summary: Deadlock caused by rdd replication level of 2
Key: SPARK-45057
URL: https://issues.apache.org/jira/browse/SPARK-45057
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-44345:
Summary: Only log unknown shuffle map output as error when shuffle
migration disabled
Key: SPARK-44345
URL: https://issues.apache.org/jira/browse/SPARK-44345
[
https://issues.apache.org/jira/browse/SPARK-44126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-44126:
-
Description:
When shuffle migration to decommissioned executor, the below exception is
thrown:
Zhongwei Zhu created SPARK-44126:
Summary: Migration shuffle to decommissioned executor should not
count as block failure
Key: SPARK-44126
URL: https://issues.apache.org/jira/browse/SPARK-44126
Zhongwei Zhu created SPARK-44084:
Summary: Dynamic allocation pending tasks should not include
finished ones
Key: SPARK-44084
URL: https://issues.apache.org/jira/browse/SPARK-44084
Project: Spark
Zhongwei Zhu created SPARK-43828:
Summary: Add config to control whether close idle connection
Key: SPARK-43828
URL: https://issues.apache.org/jira/browse/SPARK-43828
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-43398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-43398:
-
Affects Version/s: (was: 3.4.0)
> Executor timeout should be max of idleTimeout rddTimeout
[
https://issues.apache.org/jira/browse/SPARK-43398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-43398:
-
Affects Version/s: 3.0.0
> Executor timeout should be max of idleTimeout rddTimeout
Zhongwei Zhu created SPARK-43399:
Summary: Add config to control threshold of unregister map output
when fetch failed
Key: SPARK-43399
URL: https://issues.apache.org/jira/browse/SPARK-43399
Project:
Zhongwei Zhu created SPARK-43398:
Summary: Executor timeout should be max of idleTimeout rddTimeout
shuffleTimeout
Key: SPARK-43398
URL: https://issues.apache.org/jira/browse/SPARK-43398
Project:
[
https://issues.apache.org/jira/browse/SPARK-43397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-43397:
-
Description: Log executor decommission duration.
> Log executor decommission duration
>
Zhongwei Zhu created SPARK-43397:
Summary: Log executor decommission duration
Key: SPARK-43397
URL: https://issues.apache.org/jira/browse/SPARK-43397
Project: Spark
Issue Type: Improvement
Zhongwei Zhu created SPARK-43396:
Summary: Add config to control max ratio of decommissioning
executors
Key: SPARK-43396
URL: https://issues.apache.org/jira/browse/SPARK-43396
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-43391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-43391:
-
Description: Spark will close idle connection when there're outstanding
requests but no traffic
Zhongwei Zhu created SPARK-43391:
Summary: Idle connection should not be closed when
closeIdleConnection is disabled
Key: SPARK-43391
URL: https://issues.apache.org/jira/browse/SPARK-43391
Project:
[
https://issues.apache.org/jira/browse/SPARK-43224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-43224:
-
Description: Currently, executor info in standalone master will be removed
once decommissioned.
Zhongwei Zhu created SPARK-43238:
Summary: Support only decommission idle workers in standalone
Key: SPARK-43238
URL: https://issues.apache.org/jira/browse/SPARK-43238
Project: Spark
Issue
Zhongwei Zhu created SPARK-43237:
Summary: Handle null exception message in event log
Key: SPARK-43237
URL: https://issues.apache.org/jira/browse/SPARK-43237
Project: Spark
Issue Type: Bug
Zhongwei Zhu created SPARK-43224:
Summary: Executor should not be removed when decommissioned in
standalone
Key: SPARK-43224
URL: https://issues.apache.org/jira/browse/SPARK-43224
Project: Spark
Zhongwei Zhu created SPARK-43086:
Summary: Support bin pack task scheduling on executors
Key: SPARK-43086
URL: https://issues.apache.org/jira/browse/SPARK-43086
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-43052:
Summary: Handle stacktrace with null file name in event log
Key: SPARK-43052
URL: https://issues.apache.org/jira/browse/SPARK-43052
Project: Spark
Issue
Zhongwei Zhu created SPARK-43037:
Summary: Support fetch migrated shuffle with multiple reducers
Key: SPARK-43037
URL: https://issues.apache.org/jira/browse/SPARK-43037
Project: Spark
Issue
Zhongwei Zhu created SPARK-42925:
Summary: Check executor alive from driver before fetch blocks
Key: SPARK-42925
URL: https://issues.apache.org/jira/browse/SPARK-42925
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-41956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-41956:
-
Summary: Refetch shuffle blocks when executor is decommissioned (was:
Shuffle output location
[
https://issues.apache.org/jira/browse/SPARK-41955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-41955:
-
Summary: Support fetch latest map output from executor (was: Support
fetch latest map output
Zhongwei Zhu created SPARK-42104:
Summary: Throw ExecutorDeadException in fetchBlocks when executor
dead
Key: SPARK-42104
URL: https://issues.apache.org/jira/browse/SPARK-42104
Project: Spark
Zhongwei Zhu created SPARK-41956:
Summary: Shuffle output location refetch in
ShuffleBlockFetcherIterator
Key: SPARK-41956
URL: https://issues.apache.org/jira/browse/SPARK-41956
Project: Spark
Zhongwei Zhu created SPARK-41955:
Summary: Support fetch latest map output from worker
Key: SPARK-41955
URL: https://issues.apache.org/jira/browse/SPARK-41955
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-41954:
Summary: Add isDecommissioned in ExecutorDeadException
Key: SPARK-41954
URL: https://issues.apache.org/jira/browse/SPARK-41954
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-41953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17656365#comment-17656365
]
Zhongwei Zhu commented on SPARK-41953:
--
[~dongjoon] [~mridulm80] [~Ngone51] Any comments for this?
[
https://issues.apache.org/jira/browse/SPARK-41953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-41953:
-
Description:
When shuffle migration enabled during spark decommissionm, shuffle data will be
Zhongwei Zhu created SPARK-41953:
Summary: Shuffle output location refetch during shuffle migration
in decommission
Key: SPARK-41953
URL: https://issues.apache.org/jira/browse/SPARK-41953
Project:
[
https://issues.apache.org/jira/browse/SPARK-41766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-41766:
-
Summary: Handle decommission request sent before executor registration
(was: Handle
Zhongwei Zhu created SPARK-41766:
Summary: Handle decommission request for unregistered executor
Key: SPARK-41766
URL: https://issues.apache.org/jira/browse/SPARK-41766
Project: Spark
Issue
Zhongwei Zhu created SPARK-41341:
Summary: Wait shuffle fetch to finish when decommission executor
Key: SPARK-41341
URL: https://issues.apache.org/jira/browse/SPARK-41341
Project: Spark
Zhongwei Zhu created SPARK-41153:
Summary: Log migrated shuffle data size and migration time
Key: SPARK-41153
URL: https://issues.apache.org/jira/browse/SPARK-41153
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-40979:
-
Description:
Removed executor due to decommission should be kept in a separate set. To avoid
Zhongwei Zhu created SPARK-40781:
Summary: Explain exit code 137 as killed due to OOM
Key: SPARK-40781
URL: https://issues.apache.org/jira/browse/SPARK-40781
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-40778:
Summary: Make HeartbeatReceiver as an IsolatedRpcEndpoint
Key: SPARK-40778
URL: https://issues.apache.org/jira/browse/SPARK-40778
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-40636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-40636:
-
Description:
BlockManagerDecommissioner should log correct remained shuffles.
{code:java}
4 of
[
https://issues.apache.org/jira/browse/SPARK-40636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-40636:
-
Description:
BlockManagerDecommissioner should log correct remained shuffles
```
4 of 24
[
https://issues.apache.org/jira/browse/SPARK-40636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-40636:
-
Description:
BlockManagerDecommissioner should log correct remained shuffles
{code:java}
4
Zhongwei Zhu created SPARK-40636:
Summary: Fix wrong remained shuffles log in
BlockManagerDecommissioner
Key: SPARK-40636
URL: https://issues.apache.org/jira/browse/SPARK-40636
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-40636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-40636:
-
Description:
```
4 of 24 local shuffles are added. In total, 24 shuffles are remained.
Zhongwei Zhu created SPARK-40481:
Summary: Ignore stage fetch failure caused by decommissioned
executor
Key: SPARK-40481
URL: https://issues.apache.org/jira/browse/SPARK-40481
Project: Spark
Zhongwei Zhu created SPARK-40381:
Summary: Support standalone worker recommission
Key: SPARK-40381
URL: https://issues.apache.org/jira/browse/SPARK-40381
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-40269:
Summary: Randomize the orders of peer in BlockManagerDecommissioner
Key: SPARK-40269
URL: https://issues.apache.org/jira/browse/SPARK-40269
Project: Spark
Zhongwei Zhu created SPARK-40267:
Summary: Add description for ExecutorAllocationManager metrics
Key: SPARK-40267
URL: https://issues.apache.org/jira/browse/SPARK-40267
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-40168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-40168:
-
Description:
When shuffle files not found, decommissioner will handles IOException, but the
[
https://issues.apache.org/jira/browse/SPARK-40168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-40168:
-
Description:
{code:java}
// Some comments here
public String getFoo()
{
return foo;
}
Zhongwei Zhu created SPARK-40168:
Summary: Handle FileNotFoundException when shuffle file deleted in
decommissioner
Key: SPARK-40168
URL: https://issues.apache.org/jira/browse/SPARK-40168
Project:
Zhongwei Zhu created SPARK-40060:
Summary: Add numberDecommissioningExecutors metric
Key: SPARK-40060
URL: https://issues.apache.org/jira/browse/SPARK-40060
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-36893:
Summary: upgrade mesos into 1.4.3
Key: SPARK-36893
URL: https://issues.apache.org/jira/browse/SPARK-36893
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-36864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-36864:
-
Summary: guava version mismatch with hadoop-aws (was: guava version
mismatch between
Zhongwei Zhu created SPARK-36864:
Summary: guava version mismatch between hadoop-aws and spark
Key: SPARK-36864
URL: https://issues.apache.org/jira/browse/SPARK-36864
Project: Spark
Issue
Zhongwei Zhu created SPARK-36793:
Summary: [K8S] Support write container stdout/stderr to file
Key: SPARK-36793
URL: https://issues.apache.org/jira/browse/SPARK-36793
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-32288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-32288:
-
Summary: [UI] Add failure summary table in stage page (was: [UI] Add
exception summary table
[
https://issues.apache.org/jira/browse/SPARK-34777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-34777:
-
Summary: [UI] StagePage input size/records not show when records greater
than zero (was: [UI]
[
https://issues.apache.org/jira/browse/SPARK-34777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-34777:
-
Description:
!No input size records.png|width=547,height=212!
The `Input Size / Records`
[
https://issues.apache.org/jira/browse/SPARK-34777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-34777:
-
Description:
!No input size records.png|width=547,height=212!
The `Input Size / Records`
[
https://issues.apache.org/jira/browse/SPARK-34777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-34777:
-
Attachment: No input size records.png
> [UI] StagePage input size records not show when records
Zhongwei Zhu created SPARK-34777:
Summary: [UI] StagePage input size records not show when records
greater than zero
Key: SPARK-34777
URL: https://issues.apache.org/jira/browse/SPARK-34777
Project:
Zhongwei Zhu created SPARK-34232:
Summary: [CORE] redact credentials not working when log slow event
enabled
Key: SPARK-34232
URL: https://issues.apache.org/jira/browse/SPARK-34232
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257091#comment-17257091
]
Zhongwei Zhu edited comment on SPARK-19450 at 12/31/20, 7:58 PM:
-
For
[
https://issues.apache.org/jira/browse/SPARK-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257091#comment-17257091
]
Zhongwei Zhu commented on SPARK-19450:
--
For old askWithRetry method, it can use provided config
Zhongwei Zhu created SPARK-33446:
Summary: [CORE] Add config spark.executor.coresOverhead
Key: SPARK-33446
URL: https://issues.apache.org/jira/browse/SPARK-33446
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-32314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-32314:
-
Description:
Currently, EventLoggingListeneer write both "Stack Trace" and "Full Stack
Trace"
[
https://issues.apache.org/jira/browse/SPARK-32314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-32314:
-
Summary: [SHS] Remove old format of stacktrace in event log (was: [SHS]
Add config to control
[
https://issues.apache.org/jira/browse/SPARK-33375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-33375:
-
Summary: [CORE] Add config spark.yarn.pyspark.archives (was: [CORE] config
Zhongwei Zhu created SPARK-33375:
Summary: [CORE] config spark.yarn.pyspark.archives
Key: SPARK-33375
URL: https://issues.apache.org/jira/browse/SPARK-33375
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-33374:
Summary: [CORE] Remove unnecessary python path from spark home
Key: SPARK-33374
URL: https://issues.apache.org/jira/browse/SPARK-33374
Project: Spark
Issue
Zhongwei Zhu created SPARK-33274:
Summary: [SS] Fix job hang in cp mode when total cores less than
total kafka partition
Key: SPARK-33274
URL: https://issues.apache.org/jira/browse/SPARK-33274
[
https://issues.apache.org/jira/browse/SPARK-32863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219872#comment-17219872
]
Zhongwei Zhu commented on SPARK-32863:
--
[~chengsu] Have you already worked on this? I want to help
Zhongwei Zhu created SPARK-32446:
Summary: Add new executor metrics summary REST APIs and parameters
Key: SPARK-32446
URL: https://issues.apache.org/jira/browse/SPARK-32446
Project: Spark
Zhongwei Zhu created SPARK-32349:
Summary: [UI] Reduce unnecessary allexecutors call when render
stage page executor summary
Key: SPARK-32349
URL: https://issues.apache.org/jira/browse/SPARK-32349
Zhongwei Zhu created SPARK-32314:
Summary: [SHS] Add config to control whether log old format of
stacktrace
Key: SPARK-32314
URL: https://issues.apache.org/jira/browse/SPARK-32314
Project: Spark
Zhongwei Zhu created SPARK-32288:
Summary: [UI] Add exception summary table in stage page
Key: SPARK-32288
URL: https://issues.apache.org/jira/browse/SPARK-32288
Project: Spark
Issue Type:
Zhongwei Zhu created SPARK-32125:
Summary: [UI] Support get taskList by status in Web UI and SHS
Rest API
Key: SPARK-32125
URL: https://issues.apache.org/jira/browse/SPARK-32125
Project: Spark
Zhongwei Zhu created SPARK-32124:
Summary: [SHS] Failed to parse FetchFailed TaskEndReason from
event log produce by Spark 2.4
Key: SPARK-32124
URL: https://issues.apache.org/jira/browse/SPARK-32124
[
https://issues.apache.org/jira/browse/SPARK-32044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-32044:
-
Summary: [SS] 2.4 Kafka continuous processing print mislead initial offsets
log (was: [SS]
[
https://issues.apache.org/jira/browse/SPARK-32044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-32044:
-
Summary: [SS] 2.4 Kakfa continuous processing print mislead initial offsets
log (was: [SS]
[
https://issues.apache.org/jira/browse/SPARK-32044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-32044:
-
Description:
When using structured streaming in continuous processing mode, after restart
[
https://issues.apache.org/jira/browse/SPARK-32044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhongwei Zhu updated SPARK-32044:
-
Summary: [SS] Kakfa continuous processing print mislead initial offsets log
(was: Kakfa
Zhongwei Zhu created SPARK-32044:
Summary: Kakfa continuous processing print mislead initial offsets
log
Key: SPARK-32044
URL: https://issues.apache.org/jira/browse/SPARK-32044
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008493#comment-17008493
]
Zhongwei Zhu commented on SPARK-23432:
--
I'll work on this.
> Expose executor memory metrics in
93 matches
Mail list logo
