[jira] [Updated] (SPARK-27221) Improve the assert error message in TreeNode

[ https://issues.apache.org/jira/browse/SPARK-27221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27221: - Description: When TreeNode.parseToJson may throw an assert error without any error message

[jira] [Created] (SPARK-27221) Improve the assert error message in TreeNode

Shixiong Zhu created SPARK-27221: Summary: Improve the assert error message in TreeNode Key: SPARK-27221 URL: https://issues.apache.org/jira/browse/SPARK-27221 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25449) Don't send zero accumulators in heartbeats

[ https://issues.apache.org/jira/browse/SPARK-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791960#comment-16791960 ] Shixiong Zhu commented on SPARK-25449: -- I think this patch actually fixed a bug introduced by

[jira] [Updated] (SPARK-27111) A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally

[ https://issues.apache.org/jira/browse/SPARK-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27111: - Fix Version/s: 2.3.4 > A continuous query may fail with InterruptedException when kafka

[jira] [Resolved] (SPARK-27111) A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally

[ https://issues.apache.org/jira/browse/SPARK-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-27111. -- Resolution: Fixed > A continuous query may fail with InterruptedException when kafka consumer

[jira] [Updated] (SPARK-27111) A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally

[ https://issues.apache.org/jira/browse/SPARK-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27111: - Fix Version/s: 2.4.2 > A continuous query may fail with InterruptedException when kafka

[jira] [Updated] (SPARK-27111) A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally

[ https://issues.apache.org/jira/browse/SPARK-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27111: - Fix Version/s: 3.0.0 > A continuous query may fail with InterruptedException when kafka

[jira] [Updated] (SPARK-27111) A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally

[ https://issues.apache.org/jira/browse/SPARK-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27111: - Affects Version/s: 2.4.1 2.4.0 > A continuous query may fail with

[jira] [Created] (SPARK-27111) A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally

Shixiong Zhu created SPARK-27111: Summary: A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally Key: SPARK-27111 URL:

[jira] [Updated] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26824: - Docs Text: Earlier version of Spark incorrectly escaped paths when writing out checkpoints and

[jira] [Resolved] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26824. -- Resolution: Fixed Fix Version/s: 3.0.0 > Streaming queries may store checkpoint data

[jira] [Reopened] (SPARK-20977) NPE in CollectionAccumulator

[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-20977: -- Reopening this issue as I believe I understand the cause. An accumulator is escaped before it's

[jira] [Updated] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26824: - Affects Version/s: 2.0.0 2.1.0 2.2.0

[jira] [Commented] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760071#comment-16760071 ] Shixiong Zhu commented on SPARK-26824: -- This will need a release note. After the fix, the paths to

[jira] [Updated] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26824: - Description: When a user specifies a checkpoint location containing special chars that need to

[jira] [Created] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

Shixiong Zhu created SPARK-26824: Summary: Streaming queries may store checkpoint data in a wrong directory Key: SPARK-26824 URL: https://issues.apache.org/jira/browse/SPARK-26824 Project: Spark

[jira] [Updated] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26824: - Labels: release-notes (was: ) > Streaming queries may store checkpoint data in a wrong

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Affects Version/s: 2.3.3 > EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly >

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Fix Version/s: (was: 2.3.3) 2.3.4 > EventTimeStats.merge doesn't handle

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Affects Version/s: 2.2.2 2.2.3 > EventTimeStats.merge doesn't handle

[jira] [Resolved] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26806. -- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1

[jira] [Commented] (SPARK-26783) Kafka parameter documentation doesn't match with the reality (upper/lowercase)

[ https://issues.apache.org/jira/browse/SPARK-26783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757824#comment-16757824 ] Shixiong Zhu commented on SPARK-26783: -- [~gsomogyi] This seems just an API document issue. Right?

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Description: Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will make

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Reporter: liancheng (was: Shixiong Zhu) > EventTimeStats.merge doesn't handle

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Affects Version/s: 2.2.1 2.3.0 2.3.1

[jira] [Created] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

Shixiong Zhu created SPARK-26806: Summary: EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly Key: SPARK-26806 URL: https://issues.apache.org/jira/browse/SPARK-26806 Project: Spark

[jira] [Commented] (SPARK-26682) Task attempt ID collision causes lost data

[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751651#comment-16751651 ] Shixiong Zhu commented on SPARK-26682: -- For future reference, data loss could happen when one task

[jira] [Commented] (SPARK-26682) Task attempt ID collision causes lost data

[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750486#comment-16750486 ] Shixiong Zhu commented on SPARK-26682: -- IIUC, this issue will cause a file deletion (delete the

[jira] [Updated] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26665: - Fix Version/s: 2.3.4 > BlockTransferService.fetchBlockSync may hang forever >

[jira] [Updated] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26665: - Affects Version/s: 2.3.0 2.3.1 > BlockTransferService.fetchBlockSync may

[jira] [Updated] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26665: - Affects Version/s: 2.3.2 > BlockTransferService.fetchBlockSync may hang forever >

[jira] [Resolved] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26665. -- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 >

[jira] [Created] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

Shixiong Zhu created SPARK-26665: Summary: BlockTransferService.fetchBlockSync may hang forever Key: SPARK-26665 URL: https://issues.apache.org/jira/browse/SPARK-26665 Project: Spark Issue

[jira] [Updated] (SPARK-26629) Error with multiple file stream in a query + restart on a batch that has no data for one file stream

[ https://issues.apache.org/jira/browse/SPARK-26629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26629: - Fix Version/s: (was: 2.3.4) > Error with multiple file stream in a query + restart on a

[jira] [Updated] (SPARK-26629) Error with multiple file stream in a query + restart on a batch that has no data for one file stream

[ https://issues.apache.org/jira/browse/SPARK-26629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26629: - Fix Version/s: 3.0.0 2.4.1 2.3.4 > Error with multiple

[jira] [Resolved] (SPARK-26629) Error with multiple file stream in a query + restart on a batch that has no data for one file stream

[ https://issues.apache.org/jira/browse/SPARK-26629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26629. -- Resolution: Fixed > Error with multiple file stream in a query + restart on a batch that has

[jira] [Updated] (SPARK-26629) Error with multiple file stream in a query + restart on a batch that has no data for one file stream

[ https://issues.apache.org/jira/browse/SPARK-26629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26629: - Affects Version/s: 2.3.3 > Error with multiple file stream in a query + restart on a batch that

[jira] [Resolved] (SPARK-26350) Allow the user to override the group id of the Kafka's consumer

[ https://issues.apache.org/jira/browse/SPARK-26350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26350. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 3.0.0 > Allow the

[jira] [Updated] (SPARK-26586) Streaming queries should have isolated SparkSessions and confs

[ https://issues.apache.org/jira/browse/SPARK-26586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26586: - Affects Version/s: 2.2.1 2.2.2 2.3.1

[jira] [Updated] (SPARK-26586) Streaming queries should have isolated SparkSessions and confs

[ https://issues.apache.org/jira/browse/SPARK-26586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26586: - Affects Version/s: 2.2.0 > Streaming queries should have isolated SparkSessions and confs >

[jira] [Updated] (SPARK-26586) Streaming queries should have isolated SparkSessions and confs

[ https://issues.apache.org/jira/browse/SPARK-26586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26586: - Target Version/s: 3.0.0 (was: 2.5.0, 3.0.0) > Streaming queries should have isolated

[jira] [Resolved] (SPARK-26586) Streaming queries should have isolated SparkSessions and confs

[ https://issues.apache.org/jira/browse/SPARK-26586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26586. -- Resolution: Fixed Assignee: Mukul Murthy Fix Version/s: 3.0.0

[jira] [Created] (SPARK-26594) DataSourceOptions.asMap should return CaseInsensitiveMap

Shixiong Zhu created SPARK-26594: Summary: DataSourceOptions.asMap should return CaseInsensitiveMap Key: SPARK-26594 URL: https://issues.apache.org/jira/browse/SPARK-26594 Project: Spark

[jira] [Resolved] (SPARK-26267) Kafka source may reprocess data

[ https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26267. -- Resolution: Fixed Fix Version/s: 2.4.1 > Kafka source may reprocess data >

[jira] [Updated] (SPARK-26267) Kafka source may reprocess data

[ https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26267: - Fix Version/s: 3.0.0 > Kafka source may reprocess data > --- > >

[jira] [Assigned] (SPARK-26267) Kafka source may reprocess data

[ https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-26267: Assignee: Shixiong Zhu > Kafka source may reprocess data >

[jira] [Created] (SPARK-26350) Allow the user to override the group id of the Kafka's consumer

Shixiong Zhu created SPARK-26350: Summary: Allow the user to override the group id of the Kafka's consumer Key: SPARK-26350 URL: https://issues.apache.org/jira/browse/SPARK-26350 Project: Spark

[jira] [Updated] (SPARK-26267) Kafka source may reprocess data

[ https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26267: - Priority: Blocker (was: Major) > Kafka source may reprocess data >

[jira] [Updated] (SPARK-26267) Kafka source may reprocess data

[ https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26267: - Labels: correctness (was: ) > Kafka source may reprocess data >

[jira] [Commented] (SPARK-26267) Kafka source may reprocess data

[ https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709169#comment-16709169 ] Shixiong Zhu commented on SPARK-26267: -- KAFKA-7703 only exists in Kafka 1.1.0 and above, so a

[jira] [Created] (SPARK-26267) Kafka source may reprocess data

Shixiong Zhu created SPARK-26267: Summary: Kafka source may reprocess data Key: SPARK-26267 URL: https://issues.apache.org/jira/browse/SPARK-26267 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-26120) Fix a streaming query leak in Structured Streaming R tests

Shixiong Zhu created SPARK-26120: Summary: Fix a streaming query leak in Structured Streaming R tests Key: SPARK-26120 URL: https://issues.apache.org/jira/browse/SPARK-26120 Project: Spark

[jira] [Updated] (SPARK-26120) Fix a streaming query leak in Structured Streaming R tests

[ https://issues.apache.org/jira/browse/SPARK-26120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26120: - Priority: Minor (was: Major) > Fix a streaming query leak in Structured Streaming R tests >

[jira] [Updated] (SPARK-26120) Fix a streaming query leak in Structured Streaming R tests

[ https://issues.apache.org/jira/browse/SPARK-26120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26120: - Component/s: Structured Streaming SparkR > Fix a streaming query leak in

[jira] [Resolved] (SPARK-26092) Use CheckpointFileManager to write the streaming metadata file

[ https://issues.apache.org/jira/browse/SPARK-26092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26092. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 3.0.0

[jira] [Updated] (SPARK-26092) Use CheckpointFileManager to write the streaming metadata file

[ https://issues.apache.org/jira/browse/SPARK-26092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26092: - Issue Type: Bug (was: Test) > Use CheckpointFileManager to write the streaming metadata file >

[jira] [Created] (SPARK-26092) Use CheckpointFileManager to write the streaming metadata file

Shixiong Zhu created SPARK-26092: Summary: Use CheckpointFileManager to write the streaming metadata file Key: SPARK-26092 URL: https://issues.apache.org/jira/browse/SPARK-26092 Project: Spark

[jira] [Resolved] (SPARK-26069) Flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures

[ https://issues.apache.org/jira/browse/SPARK-26069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26069. -- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 > Flaky test:

[jira] [Created] (SPARK-26069) Flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures

Shixiong Zhu created SPARK-26069: Summary: Flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures Key: SPARK-26069 URL: https://issues.apache.org/jira/browse/SPARK-26069 Project: Spark

[jira] [Resolved] (SPARK-26042) KafkaContinuousSourceTopicDeletionSuite may hang forever

[ https://issues.apache.org/jira/browse/SPARK-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26042. -- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 >

[jira] [Created] (SPARK-26042) KafkaContinuousSourceTopicDeletionSuite may hang forever

Shixiong Zhu created SPARK-26042: Summary: KafkaContinuousSourceTopicDeletionSuite may hang forever Key: SPARK-26042 URL: https://issues.apache.org/jira/browse/SPARK-26042 Project: Spark

[jira] [Commented] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671900#comment-16671900 ] Shixiong Zhu commented on SPARK-25692: -- [~sanket991] You can download the unit test logs from

[jira] [Commented] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671895#comment-16671895 ] Shixiong Zhu commented on SPARK-25692: -- It's still flaky on Jenkins:

[jira] [Updated] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25692: - Attachment: Screen Shot 2018-11-01 at 10.17.16 AM.png > Flaky test: ChunkFetchIntegrationSuite

[jira] [Commented] (SPARK-20568) Delete files after processing in structured streaming

[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670897#comment-16670897 ] Shixiong Zhu commented on SPARK-20568: -- [~kabhwan] I think this is pretty useful. Do you have time

[jira] [Updated] (SPARK-25899) Flaky test: CoarseGrainedSchedulerBackendSuite.compute max number of concurrent tasks can be launched

[ https://issues.apache.org/jira/browse/SPARK-25899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25899: - Issue Type: Test (was: Documentation) > Flaky test: CoarseGrainedSchedulerBackendSuite.compute

[jira] [Created] (SPARK-25899) Flaky test: CoarseGrainedSchedulerBackendSuite.compute max number of concurrent tasks can be launched

Shixiong Zhu created SPARK-25899: Summary: Flaky test: CoarseGrainedSchedulerBackendSuite.compute max number of concurrent tasks can be launched Key: SPARK-25899 URL:

[jira] [Resolved] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

[ https://issues.apache.org/jira/browse/SPARK-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25773. -- Resolution: Fixed Fix Version/s: 3.0.0 > Cancel zombie tasks in a result stage when

[jira] [Created] (SPARK-25849) Improve document for task cancellation.

Shixiong Zhu created SPARK-25849: Summary: Improve document for task cancellation. Key: SPARK-25849 URL: https://issues.apache.org/jira/browse/SPARK-25849 Project: Spark Issue Type:

[jira] [Created] (SPARK-25822) Fix a race condition when releasing a Python worker

Shixiong Zhu created SPARK-25822: Summary: Fix a race condition when releasing a Python worker Key: SPARK-25822 URL: https://issues.apache.org/jira/browse/SPARK-25822 Project: Spark Issue

[jira] [Resolved] (SPARK-25771) Fix improper synchronization in PythonWorkerFactory

[ https://issues.apache.org/jira/browse/SPARK-25771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25771. -- Resolution: Fixed Fix Version/s: 3.0.0 > Fix improper synchronization in

[jira] [Updated] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

[ https://issues.apache.org/jira/browse/SPARK-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25773: - Description: When a job finishes, there may be some zombie tasks still running due to stage

[jira] [Created] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

Shixiong Zhu created SPARK-25773: Summary: Cancel zombie tasks in a result stage when the job finishes Key: SPARK-25773 URL: https://issues.apache.org/jira/browse/SPARK-25773 Project: Spark

[jira] [Created] (SPARK-25771) Fix improper synchronization in PythonWorkerFactory

Shixiong Zhu created SPARK-25771: Summary: Fix improper synchronization in PythonWorkerFactory Key: SPARK-25771 URL: https://issues.apache.org/jira/browse/SPARK-25771 Project: Spark Issue

[jira] [Commented] (SPARK-25738) LOAD DATA INPATH doesn't work if hdfs conf includes port

[ https://issues.apache.org/jira/browse/SPARK-25738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650796#comment-16650796 ] Shixiong Zhu commented on SPARK-25738: -- Marked as a blocker since this is a regression > LOAD DATA

[jira] [Updated] (SPARK-25738) LOAD DATA INPATH doesn't work if hdfs conf includes port

[ https://issues.apache.org/jira/browse/SPARK-25738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25738: - Priority: Blocker (was: Critical) > LOAD DATA INPATH doesn't work if hdfs conf includes port >

[jira] [Commented] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645399#comment-16645399 ] Shixiong Zhu commented on SPARK-23390: -- [~dongjoon] when spark cancels a task, the task thread will

[jira] [Commented] (SPARK-10816) EventTime based sessionization

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645370#comment-16645370 ] Shixiong Zhu commented on SPARK-10816: -- Thanks a lot for the design docs and prototypes. I had a

[jira] [Commented] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644162#comment-16644162 ] Shixiong Zhu commented on SPARK-25692: -- It may be caused by

[jira] [Updated] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25692: - Description: Looks like the whole test suite is pretty flaky. See:

[jira] [Created] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

Shixiong Zhu created SPARK-25692: Summary: Flaky test: ChunkFetchIntegrationSuite Key: SPARK-25692 URL: https://issues.apache.org/jira/browse/SPARK-25692 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644036#comment-16644036 ] Shixiong Zhu commented on SPARK-23390: -- I didn't look at parquet. It may have a similar issue. >

[jira] [Commented] (SPARK-23390) Flaky test: FileBasedDataSourceSuite

[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644034#comment-16644034 ] Shixiong Zhu commented on SPARK-23390: -- I think the issue is probably in orc. Any exception

[jira] [Resolved] (SPARK-25644) Fix java foreachBatch API

[ https://issues.apache.org/jira/browse/SPARK-25644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25644. -- Resolution: Fixed Fix Version/s: 2.4.0 > Fix java foreachBatch API >

[jira] [Updated] (SPARK-25644) Fix java foreachBatch API

[ https://issues.apache.org/jira/browse/SPARK-25644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25644: - Target Version/s: 2.4.0 > Fix java foreachBatch API > - > >

[jira] [Created] (SPARK-25644) Fix java foreachBatch API

Shixiong Zhu created SPARK-25644: Summary: Fix java foreachBatch API Key: SPARK-25644 URL: https://issues.apache.org/jira/browse/SPARK-25644 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25644) Fix java foreachBatch API

[ https://issues.apache.org/jira/browse/SPARK-25644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25644: - Description: The java foreachBatch API in DataStreamWriter should accept java.lang.Long rather

[jira] [Commented] (SPARK-25005) Structured streaming doesn't support kafka transaction (creating empty offset with abort & markers)

[ https://issues.apache.org/jira/browse/SPARK-25005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637564#comment-16637564 ] Shixiong Zhu commented on SPARK-25005: -- [~qambard] Not sure about your question. If Kafka consumers

[jira] [Commented] (SPARK-25005) Structured streaming doesn't support kafka transaction (creating empty offset with abort & markers)

[ https://issues.apache.org/jira/browse/SPARK-25005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637541#comment-16637541 ] Shixiong Zhu commented on SPARK-25005: -- [~qambard] If `poll` returns and offset gets changed, it

[jira] [Resolved] (SPARK-25315) setting "auto.offset.reset" to "earliest" has no effect in Structured Streaming with Spark 2.3.1 and Kafka 1.0

[ https://issues.apache.org/jira/browse/SPARK-25315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25315. -- Resolution: Not A Bug > setting "auto.offset.reset" to "earliest" has no effect in Structured

[jira] [Commented] (SPARK-25315) setting "auto.offset.reset" to "earliest" has no effect in Structured Streaming with Spark 2.3.1 and Kafka 1.0

[ https://issues.apache.org/jira/browse/SPARK-25315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634423#comment-16634423 ] Shixiong Zhu commented on SPARK-25315: -- Kafka’s own configurations should be set with "kafka."

[jira] [Resolved] (SPARK-25449) Don't send zero accumulators in heartbeats

[ https://issues.apache.org/jira/browse/SPARK-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25449. -- Resolution: Fixed Assignee: Mukul Murthy Fix Version/s: 2.5.0 > Don't send

[jira] [Created] (SPARK-25569) Failing a Spark job when an accumulator cannot be updated

Shixiong Zhu created SPARK-25569: Summary: Failing a Spark job when an accumulator cannot be updated Key: SPARK-25569 URL: https://issues.apache.org/jira/browse/SPARK-25569 Project: Spark

[jira] [Created] (SPARK-25568) Continue to update the remaining accumulators when failing to update one accumulator

Shixiong Zhu created SPARK-25568: Summary: Continue to update the remaining accumulators when failing to update one accumulator Key: SPARK-25568 URL: https://issues.apache.org/jira/browse/SPARK-25568

[jira] [Updated] (SPARK-25568) Continue to update the remaining accumulators when failing to update one accumulator

[ https://issues.apache.org/jira/browse/SPARK-25568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25568: - Description: Currently when failing to update an accumulator, DAGScheduler.updateAccumulators

[jira] [Resolved] (SPARK-25495) FetchedData.reset doesn't reset _nextOffsetInFetchedData and _offsetAfterPoll

[ https://issues.apache.org/jira/browse/SPARK-25495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25495. -- Resolution: Fixed Fix Version/s: 2.4.0 > FetchedData.reset doesn't reset

[jira] [Created] (SPARK-25495) FetchedData.reset doesn't reset _nextOffsetInFetchedData and _offsetAfterPoll

Shixiong Zhu created SPARK-25495: Summary: FetchedData.reset doesn't reset _nextOffsetInFetchedData and _offsetAfterPoll Key: SPARK-25495 URL: https://issues.apache.org/jira/browse/SPARK-25495

[jira] [Updated] (SPARK-25449) Don't send zero accumulators in heartbeats

[ https://issues.apache.org/jira/browse/SPARK-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25449: - Issue Type: Improvement (was: Task) > Don't send zero accumulators in heartbeats >

[jira] [Updated] (SPARK-25449) Don't send zero accumulators in heartbeats

[ https://issues.apache.org/jira/browse/SPARK-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25449: - Target Version/s: (was: 2.5.0) > Don't send zero accumulators in heartbeats >

[jira] [Updated] (SPARK-19903) Watermark metadata is lost when using resolved attributes

[ https://issues.apache.org/jira/browse/SPARK-19903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19903: - Target Version/s: (was: 2.4.0) > Watermark metadata is lost when using resolved attributes >

<    1   2   3   4   5   6   7   8   9   10   >