[jira] [Updated] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

2019-12-10 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21869: - Fix Version/s: (was: 3.0.0) > A cached Kafka producer should not be closed if any task is

[jira] [Commented] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

2019-12-10 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992975#comment-16992975 ] Shixiong Zhu commented on SPARK-21869: -- Reopened this. https://github.com/apache/spark/pull/25853

[jira] [Reopened] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

2019-12-10 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-21869: -- > A cached Kafka producer should not be closed if any task is using it. >

[jira] [Created] (SPARK-30208) A race condition when reading from Kafka in PySpark

2019-12-10 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-30208: Summary: A race condition when reading from Kafka in PySpark Key: SPARK-30208 URL: https://issues.apache.org/jira/browse/SPARK-30208 Project: Spark Issue

[jira] [Resolved] (SPARK-29953) File stream source cleanup options may break a file sink output

2019-12-05 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-29953. -- Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed > File stream

[jira] [Updated] (SPARK-29953) File stream source cleanup options may break a file sink output

2019-11-18 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-29953: - Description: SPARK-20568 added options to file streaming source to clean up processed files.

[jira] [Commented] (SPARK-29953) File stream source cleanup options may break a file sink output

2019-11-18 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976971#comment-16976971 ] Shixiong Zhu commented on SPARK-29953: -- cc [~kabhwan] > File stream source cleanup options may

[jira] [Created] (SPARK-29953) File stream source cleanup options may break a file sink output

2019-11-18 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-29953: Summary: File stream source cleanup options may break a file sink output Key: SPARK-29953 URL: https://issues.apache.org/jira/browse/SPARK-29953 Project: Spark

[jira] [Commented] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-10-28 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961412#comment-16961412 ] Shixiong Zhu commented on SPARK-28841: -- [~srowen] Yep. ":" is not a valid char in a HDFS path. But

[jira] [Commented] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-10-28 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961402#comment-16961402 ] Shixiong Zhu commented on SPARK-28841: -- [~srowen] "?" is to trigger glob pattern codes. It matches

[jira] [Resolved] (SPARK-27254) Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted

2019-09-27 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-27254. -- Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed > Cleanup

[jira] [Resolved] (SPARK-29099) org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set

2019-09-20 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-29099. -- Resolution: Duplicate > org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is

[jira] [Created] (SPARK-29099) org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set

2019-09-16 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-29099: Summary: org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set Key: SPARK-29099 URL: https://issues.apache.org/jira/browse/SPARK-29099

[jira] [Resolved] (SPARK-28976) Use KeyLock to simplify MapOutputTracker.getStatuses

2019-09-05 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-28976. -- Fix Version/s: 3.0.0 Resolution: Fixed > Use KeyLock to simplify

[jira] [Assigned] (SPARK-28976) Use KeyLock to simplify MapOutputTracker.getStatuses

2019-09-04 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-28976: Assignee: Shixiong Zhu > Use KeyLock to simplify MapOutputTracker.getStatuses >

[jira] [Created] (SPARK-28976) Use KeyLock to simplify MapOutputTracker.getStatuses

2019-09-04 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-28976: Summary: Use KeyLock to simplify MapOutputTracker.getStatuses Key: SPARK-28976 URL: https://issues.apache.org/jira/browse/SPARK-28976 Project: Spark Issue

[jira] [Resolved] (SPARK-3137) Use finer grained locking in TorrentBroadcast.readObject

2019-09-03 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-3137. - Fix Version/s: 3.0.0 Resolution: Fixed > Use finer grained locking in

[jira] [Commented] (SPARK-28883) Fix a flaky test: ThriftServerQueryTestSuite

2019-09-03 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921413#comment-16921413 ] Shixiong Zhu commented on SPARK-28883: -- Marked this is a 3.0.0 blocker. We should either fix the

[jira] [Updated] (SPARK-28883) Fix a flaky test: ThriftServerQueryTestSuite

2019-09-03 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28883: - Priority: Blocker (was: Major) > Fix a flaky test: ThriftServerQueryTestSuite >

[jira] [Updated] (SPARK-28883) Fix a flaky test: ThriftServerQueryTestSuite

2019-09-03 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28883: - Target Version/s: 3.0.0 > Fix a flaky test: ThriftServerQueryTestSuite >

[jira] [Reopened] (SPARK-3137) Use finer grained locking in TorrentBroadcast.readObject

2019-08-28 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-3137: - Assignee: Shixiong Zhu > Use finer grained locking in TorrentBroadcast.readObject >

[jira] [Assigned] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-08-23 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-28025: Assignee: Jungtaek Lim > HDFSBackedStateStoreProvider should not leak .crc files >

[jira] [Resolved] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-08-23 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-28025. -- Fix Version/s: 3.0.0 Resolution: Fixed > HDFSBackedStateStoreProvider should not leak

[jira] [Comment Edited] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-08-21 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912719#comment-16912719 ] Shixiong Zhu edited comment on SPARK-28841 at 8/21/19 10:35 PM: Okey, at

[jira] [Comment Edited] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-08-21 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912719#comment-16912719 ] Shixiong Zhu edited comment on SPARK-28841 at 8/21/19 10:00 PM: Okey, at

[jira] [Commented] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-08-21 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912719#comment-16912719 ] Shixiong Zhu commented on SPARK-28841: -- Okey, at least the following codes are legit but failing.

[jira] [Created] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-08-21 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-28841: Summary: Spark cannot read a relative path containing ":" Key: SPARK-28841 URL: https://issues.apache.org/jira/browse/SPARK-28841 Project: Spark Issue Type:

[jira] [Commented] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-08-21 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912572#comment-16912572 ] Shixiong Zhu commented on SPARK-28841: -- In a second thought, this is probably a limitation of

[jira] [Comment Edited] (SPARK-28841) Spark cannot read a relative path containing ":"

2019-08-21 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912572#comment-16912572 ] Shixiong Zhu edited comment on SPARK-28841 at 8/21/19 6:04 PM: --- In a

[jira] [Commented] (SPARK-28605) Performance regression in SS's foreach

2019-08-20 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911635#comment-16911635 ] Shixiong Zhu commented on SPARK-28605: -- [~kabhwan] Thanks for pointing it out. Yes, we can close

[jira] [Resolved] (SPARK-28605) Performance regression in SS's foreach

2019-08-20 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-28605. -- Resolution: Invalid > Performance regression in SS's foreach >

[jira] [Resolved] (SPARK-28650) Fix the guarantee of ForeachWriter

2019-08-20 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-28650. -- Fix Version/s: 2.4.5 Assignee: Jungtaek Lim Resolution: Fixed > Fix the

[jira] [Commented] (SPARK-28650) Fix the guarantee of ForeachWriter

2019-08-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904085#comment-16904085 ] Shixiong Zhu commented on SPARK-28650: -- Go ahead. I'm not working on this. For the signature of

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Docs Text: All fields of the Structured Streaming's file source schema will be forced to be

[jira] [Assigned] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-28651: Assignee: Shixiong Zhu > Streaming file source doesn't change the schema to nullable

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Labels: release-notes (was: ) > Streaming file source doesn't change the schema to nullable

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Description: Right now, batch DataFrame always changes the schema to nullable automatically

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Description: Right now, batch DataFrame always changes the schema to nullable automatically

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Reporter: Tomasz (was: Shixiong Zhu) > Streaming file source doesn't change the schema to

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Description: Right now, batch DataFrame always changes the schema to nullable automatically

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Description: Right now, batch DataFrame always changes the schema to nullable automatically

[jira] [Updated] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28651: - Description: Right now, batch DataFrame always changes the schema to nullable automatically

[jira] [Created] (SPARK-28651) Streaming file source doesn't change the schema to nullable automatically

2019-08-07 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28651: Summary: Streaming file source doesn't change the schema to nullable automatically Key: SPARK-28651 URL: https://issues.apache.org/jira/browse/SPARK-28651 Project:

[jira] [Comment Edited] (SPARK-26152) Synchronize Worker Cleanup with Worker Shutdown

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902477#comment-16902477 ] Shixiong Zhu edited comment on SPARK-26152 at 8/7/19 9:07 PM: --

[jira] [Commented] (SPARK-26152) Synchronize Worker Cleanup with Worker Shutdown

2019-08-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902477#comment-16902477 ] Shixiong Zhu commented on SPARK-26152: -- [~ajithshetty] does your PR fix the flaky test? If I read

[jira] [Created] (SPARK-28650) Fix the guarantee of ForeachWriter

2019-08-07 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28650: Summary: Fix the guarantee of ForeachWriter Key: SPARK-28650 URL: https://issues.apache.org/jira/browse/SPARK-28650 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-28605) Performance regression in SS's foreach

2019-08-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899789#comment-16899789 ] Shixiong Zhu commented on SPARK-28605: -- By the way, this is not a critical regression. It's not

[jira] [Commented] (SPARK-28605) Performance regression in SS's foreach

2019-08-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899787#comment-16899787 ] Shixiong Zhu commented on SPARK-28605: -- This is a regression at all 2.4 branches. It's caused by 

[jira] [Updated] (SPARK-28605) Performance regression in SS's foreach

2019-08-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28605: - Affects Version/s: 2.4.0 2.4.1 2.4.2 >

[jira] [Resolved] (SPARK-28574) Allow to config different sizes for event queues

2019-08-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-28574. -- Resolution: Fixed > Allow to config different sizes for event queues >

[jira] [Assigned] (SPARK-28574) Allow to config different sizes for event queues

2019-08-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-28574: Assignee: Yun Zou > Allow to config different sizes for event queues >

[jira] [Updated] (SPARK-28605) Performance regression in SS's foreach

2019-08-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28605: - Labels: regresssion (was: ) > Performance regression in SS's foreach >

[jira] [Created] (SPARK-28605) Performance regression in SS's foreach

2019-08-02 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28605: Summary: Performance regression in SS's foreach Key: SPARK-28605 URL: https://issues.apache.org/jira/browse/SPARK-28605 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Docs Text: In Spark 3.0, the type of "error" parameter in the

[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Docs Text: In Spark 3.0, the type of "error" parameter in the

[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Labels: release-notes (was: ) > Error should also be sent to QueryExecutionListener.onFailure

[jira] [Created] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28556: Summary: Error should also be sent to QueryExecutionListener.onFailure Key: SPARK-28556 URL: https://issues.apache.org/jira/browse/SPARK-28556 Project: Spark

[jira] [Commented] (SPARK-16754) NPE when defining case class and searching Encoder in the same line

2019-07-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893125#comment-16893125 ] Shixiong Zhu commented on SPARK-16754: -- I think prepending

[jira] [Updated] (SPARK-28489) KafkaOffsetRangeCalculator.getRanges may drop offsets

2019-07-24 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28489: - Affects Version/s: 2.4.0 2.4.1 2.4.2 >

[jira] [Updated] (SPARK-28489) KafkaOffsetRangeCalculator.getRanges may drop offsets

2019-07-24 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28489: - Description: KafkaOffsetRangeCalculator.getRanges may drop offsets due to round off errors.  

[jira] [Created] (SPARK-28489) KafkaOffsetRangeCalculator.getRanges may drop offsets

2019-07-23 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28489: Summary: KafkaOffsetRangeCalculator.getRanges may drop offsets Key: SPARK-28489 URL: https://issues.apache.org/jira/browse/SPARK-28489 Project: Spark Issue

[jira] [Updated] (SPARK-28486) PythonBroadcast may delete the broadcast file while a Python worker still needs it

2019-07-23 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28486: - Issue Type: Bug (was: New Feature) > PythonBroadcast may delete the broadcast file while a

[jira] [Created] (SPARK-28486) PythonBroadcast may delete the broadcast file while a Python worker still needs it

2019-07-23 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28486: Summary: PythonBroadcast may delete the broadcast file while a Python worker still needs it Key: SPARK-28486 URL: https://issues.apache.org/jira/browse/SPARK-28486

[jira] [Updated] (SPARK-28456) Add a public API `Encoder.makeCopy` to allow creating Encoder without touching Scala reflections

2019-07-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28456: - Description: Because `Encoder` is not thread safe, the user cannot reuse an `Encoder` in

[jira] [Created] (SPARK-28456) Add a public API `Encoder.copyEncoder` to allow creating Encoder without touching Scala reflections

2019-07-19 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28456: Summary: Add a public API `Encoder.copyEncoder` to allow creating Encoder without touching Scala reflections Key: SPARK-28456 URL:

[jira] [Resolved] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

2019-05-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20547. -- Resolution: Fixed Fix Version/s: 3.0.0 > ExecutorClassLoader's findClass may not work

[jira] [Updated] (SPARK-27711) InputFileBlockHolder should be unset at the end of tasks

2019-05-23 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27711: - Component/s: PySpark > InputFileBlockHolder should be unset at the end of tasks >

[jira] [Updated] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

2019-05-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20547: - Labels: (was: bulk-closed) > ExecutorClassLoader's findClass may not work correctly when a

[jira] [Reopened] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-20547: -- Assignee: Shixiong Zhu > ExecutorClassLoader's findClass may not work correctly when a task

[jira] [Reopened] (SPARK-11095) Simplify Netty RPC implementation by using a separate thread pool for each endpoint

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-11095: -- > Simplify Netty RPC implementation by using a separate thread pool for each > endpoint >

[jira] [Resolved] (SPARK-11095) Simplify Netty RPC implementation by using a separate thread pool for each endpoint

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-11095. -- Resolution: Won't Do > Simplify Netty RPC implementation by using a separate thread pool for

[jira] [Reopened] (SPARK-17858) Provide option for Spark SQL to skip corrupt files

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-17858: -- Assignee: Shixiong Zhu > Provide option for Spark SQL to skip corrupt files >

[jira] [Resolved] (SPARK-17858) Provide option for Spark SQL to skip corrupt files

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17858. -- Resolution: Duplicate > Provide option for Spark SQL to skip corrupt files >

[jira] [Updated] (SPARK-10719) SQLImplicits.rddToDataFrameHolder is not thread safe when using Scala 2.10

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-10719: - Fix Version/s: 2.3.0 > SQLImplicits.rddToDataFrameHolder is not thread safe when using Scala

[jira] [Closed] (SPARK-10719) SQLImplicits.rddToDataFrameHolder is not thread safe when using Scala 2.10

2019-05-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu closed SPARK-10719. Assignee: Shixiong Zhu We can close this since Scala 2.10 has been dropped in Spark 2.3.0. >

[jira] [Created] (SPARK-27753) Support SQL expressions for interval parameter in Structured Streaming

2019-05-16 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27753: Summary: Support SQL expressions for interval parameter in Structured Streaming Key: SPARK-27753 URL: https://issues.apache.org/jira/browse/SPARK-27753 Project:

[jira] [Updated] (SPARK-27735) Interval string in upper case is not supported in Trigger

2019-05-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27735: - Description: Some APIs in Structured Streaming requires the user to specify an interval. Right

[jira] [Created] (SPARK-27735) Interval string in upper case is not supported in Trigger

2019-05-15 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27735: Summary: Interval string in upper case is not supported in Trigger Key: SPARK-27735 URL: https://issues.apache.org/jira/browse/SPARK-27735 Project: Spark

[jira] [Updated] (SPARK-27494) Null keys/values don't work in Kafka source v2

2019-04-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27494: - Description: Right now Kafka source v2 doesn't support null keys or values. * When processing

[jira] [Updated] (SPARK-27494) Null keys/values don't work in Kafka source v2

2019-04-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27494: - Summary: Null keys/values don't work in Kafka source v2 (was: Null values don't work in Kafka

[jira] [Updated] (SPARK-27494) Null values don't work in Kafka source v2

2019-04-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27494: - Labels: correctness (was: ) > Null values don't work in Kafka source v2 >

[jira] [Created] (SPARK-27496) RPC should send back the fatal errors

2019-04-17 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27496: Summary: RPC should send back the fatal errors Key: SPARK-27496 URL: https://issues.apache.org/jira/browse/SPARK-27496 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-17 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820460#comment-16820460 ] Shixiong Zhu commented on SPARK-27468: -- [~shahid] You need to use "--master

[jira] [Created] (SPARK-27494) Null values don't work in Kafka source v2

2019-04-17 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27494: Summary: Null values don't work in Kafka source v2 Key: SPARK-27494 URL: https://issues.apache.org/jira/browse/SPARK-27494 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-16 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27468: - Description: I ran the following unit test and checked the UI. {code} val conf = new

[jira] [Updated] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-16 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27468: - Description: I ran the following unit test and checked the UI. {code} val conf = new

[jira] [Created] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-15 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27468: Summary: "Storage Level" in "RDD Storage Page" is not correct Key: SPARK-27468 URL: https://issues.apache.org/jira/browse/SPARK-27468 Project: Spark Issue

[jira] [Updated] (SPARK-27394) The staleness of UI may last minutes or hours when no tasks start or finish

2019-04-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27394: - Fix Version/s: 2.4.2 > The staleness of UI may last minutes or hours when no tasks start or

[jira] [Resolved] (SPARK-27419) When setting spark.executor.heartbeatInterval to a value less than 1 seconds, it will always fail

2019-04-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-27419. -- Resolution: Fixed Fix Version/s: 2.4.2 > When setting spark.executor.heartbeatInterval

[jira] [Created] (SPARK-27419) When setting spark.executor.heartbeatInterval to a value less than 1 seconds, it will always fail

2019-04-09 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27419: Summary: When setting spark.executor.heartbeatInterval to a value less than 1 seconds, it will always fail Key: SPARK-27419 URL: https://issues.apache.org/jira/browse/SPARK-27419

[jira] [Commented] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812657#comment-16812657 ] Shixiong Zhu commented on SPARK-27348: -- [~sandeep.katta2007] I cannot reproduce this locally.

[jira] [Created] (SPARK-27394) The staleness of UI may last minutes or hours when no tasks start or finish

2019-04-04 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27394: Summary: The staleness of UI may last minutes or hours when no tasks start or finish Key: SPARK-27394 URL: https://issues.apache.org/jira/browse/SPARK-27394 Project:

[jira] [Assigned] (SPARK-27394) The staleness of UI may last minutes or hours when no tasks start or finish

2019-04-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-27394: Assignee: Shixiong Zhu > The staleness of UI may last minutes or hours when no tasks

[jira] [Updated] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27348: - Description: When a heartbeat timeout happens in HeartbeatReceiver, it doesn't remove lost

[jira] [Updated] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27348: - Description: When a heartbeat timeout happens in HeartbeatReceiver, it doesn't remove lost

[jira] [Created] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-02 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27348: Summary: HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend Key: SPARK-27348 URL: https://issues.apache.org/jira/browse/SPARK-27348

[jira] [Updated] (SPARK-27275) Potential corruption in EncryptedMessage.transferTo

2019-03-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27275: - Labels: correctness (was: ) > Potential corruption in EncryptedMessage.transferTo >

[jira] [Created] (SPARK-27275) Potential corruption in EncryptedMessage.transferTo

2019-03-25 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27275: Summary: Potential corruption in EncryptedMessage.transferTo Key: SPARK-27275 URL: https://issues.apache.org/jira/browse/SPARK-27275 Project: Spark Issue

[jira] [Resolved] (SPARK-27210) Cleanup incomplete output files in ManifestFileCommitProtocol if task is aborted

2019-03-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-27210. -- Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 3.0.0 > Cleanup

[jira] [Updated] (SPARK-27221) Improve the assert error message in TreeNode.parseToJson

2019-03-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-27221: - Summary: Improve the assert error message in TreeNode.parseToJson (was: Improve the assert

<    1   2   3   4   5   6   7   8   9   10   >