[
https://issues.apache.org/jira/browse/SPARK-37473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-37473:
--
Description:
!image-2021-11-27-15-07-41-516.png|width=715,height=484!
We think it has no data when
[
https://issues.apache.org/jira/browse/SPARK-37473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-37473:
--
Attachment: image-2021-11-27-15-07-41-516.png
> BypassMergeSortShuffleWriter may loss data when disk
[
https://issues.apache.org/jira/browse/SPARK-37473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-37473:
--
Description:
!image-2021-11-27-15-07-41-516.png|width=715,height=484!
We think it has no data when
haiyangyu created SPARK-37473:
-
Summary: BypassMergeSortShuffleWriter may loss data when disk is
missing however catagory is present
Key: SPARK-37473
URL: https://issues.apache.org/jira/browse/SPARK-37473
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Summary: zstd-jni lead to read less shuffle data (was: zstd-jni lead read
less shuffle data)
>
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Description:
h2. BackGround
I find a rare case which lead some partitions read less data when use
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Description:
h2. BackGround
I find a rare case which lead some partitions read less data when use
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Description:
h2. BackGround
I find a rare case which lead some partitions read less data when use
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Description:
h2. BackGround
I find a rare case which lead some partitions read less data when use
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Description:
h2. BackGround
I find a rare case which lead some partitions read less data when use
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Attachment: image-2021-02-25-17-51-49-998.png
> zstd-jni lead read less shuffle data
>
[
https://issues.apache.org/jira/browse/SPARK-34536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34536:
--
Attachment: image-2021-02-25-17-50-49-427.png
> zstd-jni lead read less shuffle data
>
haiyangyu created SPARK-34536:
-
Summary: zstd-jni lead read less shuffle data
Key: SPARK-34536
URL: https://issues.apache.org/jira/browse/SPARK-34536
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Description:
We will build a new rpc message `FetchShuffleBlocks` when
`OneForOneBlockFetcher` init
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Description:
We will build a new rpc message `FetchShuffleBlocks` when
`OneForOneBlockFetcher` init
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Attachment: image-2021-02-25-11-31-59-110.png
> New protocol FetchShuffleBlocks in
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Attachment: image-2021-02-25-11-30-03-834.png
> New protocol FetchShuffleBlocks in
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Attachment: image-2021-02-25-11-28-31-255.png
> New protocol FetchShuffleBlocks in
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Attachment: image-2021-02-25-11-27-34-429.png
> New protocol FetchShuffleBlocks in
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Description:
We will build a new rpc message `FetchShuffleBlocks` when
`OneForOneBlockFetcher` init
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Attachment: image-2021-02-25-11-17-12-714.png
> New protocol FetchShuffleBlocks in
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Description:
We will build a new rpc message
{code:java}
FetchShuffleBlocks{code}
when
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34534:
--
Summary: New protocol FetchShuffleBlocks in OneForOneBlockFetcher lead to
data loss or correctness
haiyangyu created SPARK-34534:
-
Summary: FetchShuffleBlocks in OneForOneBlockFetcher lead to data
loss or correctness
Key: SPARK-34534
URL: https://issues.apache.org/jira/browse/SPARK-34534
Project:
[
https://issues.apache.org/jira/browse/SPARK-34242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34242:
--
Description:
In HiveShim, `getPartitionByFilters` will lead parttion scan in special , ec:
[
https://issues.apache.org/jira/browse/SPARK-34242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-34242:
--
Description:
In HiveShim,
{code:java}
getPartitionByFilters{code}
will lead parttion scan in special
haiyangyu created SPARK-34242:
-
Summary: Use getPartitionByNames to filter partition to avoid
partition scan
Key: SPARK-34242
URL: https://issues.apache.org/jira/browse/SPARK-34242
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Summary: markPartitionCompleted cause task status inconsistent (was: Stage
retry and executor
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17001661#comment-17001661
]
haiyangyu commented on SPARK-30325:
---
[https://github.com/apache/spark/pull/26975]
> Stage retry and
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Description:
h3. Corner case
The bugs occurs in the coren case as follows:
# The stage occurs for
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Attachment: image-2019-12-21-17-17-42-244.png
> Stage retry and executor crashed cause app hung up
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Attachment: image-2019-12-21-17-16-40-998.png
> Stage retry and executor crashed cause app hung up
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Attachment: image-2019-12-21-17-15-51-512.png
> Stage retry and executor crashed cause app hung up
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Attachment: image-2019-12-21-17-11-38-565.png
> Stage retry and executor crashed cause app hung up
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Description:
h3. Corner case
The bugs occurs in the coren case as follows:
# The stage occurs for
haiyangyu created SPARK-30325:
-
Summary: Stage retry and executor crashed cause app hung up forever
Key: SPARK-30325
URL: https://issues.apache.org/jira/browse/SPARK-30325
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30325:
--
Description:
Kill tasks which succeeded in origin stage when new retry stage has started the
same
[
https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999237#comment-16999237
]
haiyangyu edited comment on SPARK-30297 at 12/18/19 2:53 PM:
-
[
https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999237#comment-16999237
]
haiyangyu commented on SPARK-30297:
---
[~r...@databricks.com] [~AMateenM]
[~dongjoon]
please look this
[
https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999235#comment-16999235
]
haiyangyu commented on SPARK-30297:
---
pr here [https://github.com/apache/spark/pull/26938]
> Executor
[
https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30297:
--
Description:
h3. *Backgroud*
The driver can't sense this executor was lost through the network
[
https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30297:
--
Description:
h3. *Backgroud*
The driver can't sense this executor was lost through the network
[
https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haiyangyu updated SPARK-30297:
--
Description:
h3. *Backgroud*
The driver can't sense this executor was lost through the network
haiyangyu created SPARK-30297:
-
Summary: Executor heartbeat expired cause app hung up forever
Key: SPARK-30297
URL: https://issues.apache.org/jira/browse/SPARK-30297
Project: Spark
Issue Type:
44 matches
Mail list logo
