[jira] [Commented] (SPARK-19903) Watermark metadata is lost when using resolved attributes

[ https://issues.apache.org/jira/browse/SPARK-19903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610931#comment-16610931 ] Shixiong Zhu commented on SPARK-19903: -- Yes. I removed the target version. > Watermark metadata is

[jira] [Commented] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609595#comment-16609595 ] Shixiong Zhu commented on SPARK-23425: -- Added "release-note" label. Previously, when INPATH

[jira] [Updated] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-23425: - Labels: release-notes (was: release) > load data for hdfs file path with wild card usage is

[jira] [Updated] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-23425: - Labels: release (was: ) > load data for hdfs file path with wild card usage is not working

[jira] [Reopened] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

[ https://issues.apache.org/jira/browse/SPARK-24748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-24748: -- Assignee: (was: Arun Mahadevan) > Support for reporting custom metrics via Streaming

[jira] [Reopened] (SPARK-24863) Report offset lag as a custom metrics for Kafka structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-24863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-24863: -- Assignee: (was: Arun Mahadevan) > Report offset lag as a custom metrics for Kafka

[jira] [Updated] (SPARK-24863) Report offset lag as a custom metrics for Kafka structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-24863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24863: - Fix Version/s: (was: 2.4.0) > Report offset lag as a custom metrics for Kafka structured

[jira] [Updated] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

[ https://issues.apache.org/jira/browse/SPARK-24748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24748: - Fix Version/s: (was: 2.4.0) > Support for reporting custom metrics via Streaming Query

[jira] [Updated] (SPARK-25336) Revert SPARK-24863 and SPARK-24748

[ https://issues.apache.org/jira/browse/SPARK-25336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25336: - Summary: Revert SPARK-24863 and SPARK-24748 (was: Revert SPARK-24863 and SPARK 24748) >

[jira] [Updated] (SPARK-25336) Revert SPARK-24863 and SPARK-24748

[ https://issues.apache.org/jira/browse/SPARK-25336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25336: - Description: Revert SPARK-24863 and SPARK-24748. We will revisit them when the data source v2

[jira] [Assigned] (SPARK-25336) Revert SPARK-24863 and SPARK-24748

[ https://issues.apache.org/jira/browse/SPARK-25336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-25336: Assignee: Shixiong Zhu > Revert SPARK-24863 and SPARK-24748 >

[jira] [Created] (SPARK-25336) Revert SPARK-24863 and SPARK 24748

Shixiong Zhu created SPARK-25336: Summary: Revert SPARK-24863 and SPARK 24748 Key: SPARK-25336 URL: https://issues.apache.org/jira/browse/SPARK-25336 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-25257) v2 MicroBatchReaders can't resume from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-25257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599279#comment-16599279 ] Shixiong Zhu commented on SPARK-25257: -- [~mojodna] This issue has been fixed in SPARK-23092. > v2

[jira] [Resolved] (SPARK-25257) v2 MicroBatchReaders can't resume from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-25257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25257. -- Resolution: Duplicate > v2 MicroBatchReaders can't resume from checkpoints >

[jira] [Resolved] (SPARK-25288) Kafka transaction tests are flaky

[ https://issues.apache.org/jira/browse/SPARK-25288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25288. -- Resolution: Fixed Fix Version/s: 2.4.0 > Kafka transaction tests are flaky >

[jira] [Updated] (SPARK-25288) Kafka transaction tests are flaky

[ https://issues.apache.org/jira/browse/SPARK-25288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25288: - Description:

[jira] [Created] (SPARK-25288) Kafka transaction tests are flaky

Shixiong Zhu created SPARK-25288: Summary: Kafka transaction tests are flaky Key: SPARK-25288 URL: https://issues.apache.org/jira/browse/SPARK-25288 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-25005) Structured streaming doesn't support kafka transaction (creating empty offset with abort & markers)

[ https://issues.apache.org/jira/browse/SPARK-25005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25005. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.4.0 > Structured

[jira] [Resolved] (SPARK-25218) Potential resource leaks in TransportServer and SocketAuthHelper

[ https://issues.apache.org/jira/browse/SPARK-25218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25218. -- Resolution: Fixed Fix Version/s: 2.4.0 > Potential resource leaks in TransportServer

[jira] [Resolved] (SPARK-25106) A new Kafka consumer gets created for every batch

[ https://issues.apache.org/jira/browse/SPARK-25106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25106. -- Resolution: Duplicate Thanks for reporting this. I'm closing this as a duplicate of

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24987: - Affects Version/s: 2.2.2 2.3.0 > Kafka Cached Consumer Leaking File

[jira] [Updated] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24987: - Affects Version/s: (was: 2.3.0) > Kafka Cached Consumer Leaking File Descriptors >

[jira] [Resolved] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25214. -- Resolution: Fixed Fix Version/s: 2.4.0 > Kafka v2 source may return duplicated records

[jira] [Updated] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25214: - Affects Version/s: (was: 2.3.1) (was: 2.3.0)

[jira] [Created] (SPARK-25218) Potential resource leaks in TransportServer and SocketAuthHelper

Shixiong Zhu created SPARK-25218: Summary: Potential resource leaks in TransportServer and SocketAuthHelper Key: SPARK-25218 URL: https://issues.apache.org/jira/browse/SPARK-25218 Project: Spark

[jira] [Updated] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

[ https://issues.apache.org/jira/browse/SPARK-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25214: - Description: When there are missing offsets, Kafka v2 source may return duplicated records

[jira] [Created] (SPARK-25214) Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

Shixiong Zhu created SPARK-25214: Summary: Kafka v2 source may return duplicated records when `failOnDataLoss` is `false` Key: SPARK-25214 URL: https://issues.apache.org/jira/browse/SPARK-25214

[jira] [Resolved] (SPARK-25163) Flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuite.spilling with compression

[ https://issues.apache.org/jira/browse/SPARK-25163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25163. -- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.4.0 > Flaky

[jira] [Resolved] (SPARK-25181) Block Manager master and slave thread pools are unbounded

[ https://issues.apache.org/jira/browse/SPARK-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25181. -- Resolution: Fixed Assignee: Mukul Murthy Fix Version/s: 2.4.0 > Block Manager

[jira] [Created] (SPARK-25163) Flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuite.spilling with compression

Shixiong Zhu created SPARK-25163: Summary: Flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuite.spilling with compression Key: SPARK-25163 URL: https://issues.apache.org/jira/browse/SPARK-25163

[jira] [Resolved] (SPARK-25116) Fix the "exit code 1" error when terminating Kafka tests

[ https://issues.apache.org/jira/browse/SPARK-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25116. -- Resolution: Fixed Fix Version/s: 2.4.0 > Fix the "exit code 1" error when terminating

[jira] [Created] (SPARK-25116) Fix the "exit code 1" error when terminating Kafka tests

Shixiong Zhu created SPARK-25116: Summary: Fix the "exit code 1" error when terminating Kafka tests Key: SPARK-25116 URL: https://issues.apache.org/jira/browse/SPARK-25116 Project: Spark

[jira] [Updated] (SPARK-24699) Watermark / Append mode should work with Trigger.Once

[ https://issues.apache.org/jira/browse/SPARK-24699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24699: - Fix Version/s: 2.4.0 > Watermark / Append mode should work with Trigger.Once >

[jira] [Resolved] (SPARK-20919) Simplification of CachedKafkaConsumer.

[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20919. -- Resolution: Duplicate > Simplification of CachedKafkaConsumer. >

[jira] [Resolved] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25081. -- Resolution: Fixed > Nested spill in ShuffleExternalSorter may access a released memory page

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Affects Version/s: 2.3.2 > Nested spill in ShuffleExternalSorter may access a released memory

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Fix Version/s: (was: 2.3.2) 2.3.3 > Nested spill in

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Priority: Blocker (was: Major) > Nested spill in ShuffleExternalSorter may access a released

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Affects Version/s: 1.6.2 > Nested spill in ShuffleExternalSorter may access a released memory

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Affects Version/s: 1.6.0 1.6.1 2.0.0

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Affects Version/s: 1.6.3 2.0.2 2.1.3

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Fix Version/s: 2.4.0 2.3.2 > Nested spill in ShuffleExternalSorter may

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Affects Version/s: 2.3.0 > Nested spill in ShuffleExternalSorter may access a released memory

[jira] [Commented] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576636#comment-16576636 ] Shixiong Zhu commented on SPARK-25081: -- That's possible. That's why I added the "corrnectness"

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Labels: correctness (was: ) > Nested spill in ShuffleExternalSorter may access a released

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Description: This issue is pretty similar to SPARK-21907. "allocateArray" in

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Description: This issue is pretty similar to SPARK-21907. "allocateArray" in

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Description: This issue is pretty similar to SPARK-21907. "allocateArray" in

[jira] [Created] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

Shixiong Zhu created SPARK-25081: Summary: Nested spill in ShuffleExternalSorter may access a released memory page Key: SPARK-25081 URL: https://issues.apache.org/jira/browse/SPARK-25081 Project:

[jira] [Resolved] (SPARK-24161) Enable debug package feature on structured streaming

[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24161. -- Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 2.4.0 > Enable debug

[jira] [Resolved] (SPARK-24896) Uuid expression should produce different values in each execution under streaming query

[ https://issues.apache.org/jira/browse/SPARK-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24896. -- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.4.0 > Uuid

[jira] [Updated] (SPARK-24896) Uuid expression should produce different values in each execution under streaming query

[ https://issues.apache.org/jira/browse/SPARK-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24896: - Affects Version/s: (was: 2.4.0) 2.3.0 2.3.1 >

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 2.0.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567010#comment-16567010 ] Shixiong Zhu commented on SPARK-18057: -- [~srowen] Could you create a new ticket for DStreams Kafka?

[jira] [Updated] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 2.0.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18057: - Summary: Update structured streaming kafka from 0.10.0.1 to 2.0.0 (was: Update structured

[jira] [Resolved] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 2.0.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18057. -- Resolution: Fixed Assignee: Ted Yu Fix Version/s: 2.4.0 > Update structured

[jira] [Resolved] (SPARK-24880) Fix the group id for spark-kubernetes-integration-tests

[ https://issues.apache.org/jira/browse/SPARK-24880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24880. -- Resolution: Fixed Fix Version/s: 2.4.0 > Fix the group id for

[jira] [Updated] (SPARK-24880) Fix the group id for spark-kubernetes-integration-tests

[ https://issues.apache.org/jira/browse/SPARK-24880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24880: - Description: The correct group id should be `org.apache.spark`. This is causing the nightly

[jira] [Created] (SPARK-24880) Fix the group id for spark-kubernetes-integration-tests

Shixiong Zhu created SPARK-24880: Summary: Fix the group id for spark-kubernetes-integration-tests Key: SPARK-24880 URL: https://issues.apache.org/jira/browse/SPARK-24880 Project: Spark

[jira] [Resolved] (SPARK-24566) Fix spark.storage.blockManagerSlaveTimeoutMs default config

[ https://issues.apache.org/jira/browse/SPARK-24566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24566. -- Resolution: Fixed Assignee: xueyu Fix Version/s: 2.4.0 > Fix

[jira] [Updated] (SPARK-24566) Fix spark.storage.blockManagerSlaveTimeoutMs default config

[ https://issues.apache.org/jira/browse/SPARK-24566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24566: - Summary: Fix spark.storage.blockManagerSlaveTimeoutMs default config (was:

[jira] [Resolved] (SPARK-24451) Spark-Streaming-Kafka-1.6.3- KafkaUtils.createStream function uses the old Logging class

[ https://issues.apache.org/jira/browse/SPARK-24451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24451. -- Resolution: Not A Problem > Spark-Streaming-Kafka-1.6.3- KafkaUtils.createStream function

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525290#comment-16525290 ] Shixiong Zhu commented on SPARK-24630: -- Structured Streaming supports standard SQL as the batch

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24630: - Component/s: (was: SQL) Structured Streaming > SPIP: Support SQLStreaming

[jira] [Assigned] (SPARK-24578) Reading remote cache block behavior changes and causes timeout issue

[ https://issues.apache.org/jira/browse/SPARK-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24578: Assignee: Wenbo Zhao > Reading remote cache block behavior changes and causes timeout

[jira] [Resolved] (SPARK-24578) Reading remote cache block behavior changes and causes timeout issue

[ https://issues.apache.org/jira/browse/SPARK-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24578. -- Resolution: Fixed Fix Version/s: 2.3.2 Issue resolved by pull request 21593

[jira] [Updated] (SPARK-24578) Reading remote cache block behavior changes and causes timeout issue

[ https://issues.apache.org/jira/browse/SPARK-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24578: - Fix Version/s: 2.4.0 > Reading remote cache block behavior changes and causes timeout issue >

[jira] [Resolved] (SPARK-24565) Add API for in Structured Streaming for exposing output rows of each microbatch as a DataFrame

[ https://issues.apache.org/jira/browse/SPARK-24565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24565. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21571

[jira] [Assigned] (SPARK-24235) create the top-of-task RDD sending rows to the remote buffer

[ https://issues.apache.org/jira/browse/SPARK-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24235: Assignee: Jose Torres > create the top-of-task RDD sending rows to the remote buffer >

[jira] [Resolved] (SPARK-24235) create the top-of-task RDD sending rows to the remote buffer

[ https://issues.apache.org/jira/browse/SPARK-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24235. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21428

[jira] [Updated] (SPARK-24466) TextSocketMicroBatchReader no longer works with nc utility

[ https://issues.apache.org/jira/browse/SPARK-24466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24466: - Target Version/s: 2.4.0 > TextSocketMicroBatchReader no longer works with nc utility >

[jira] [Commented] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503630#comment-16503630 ] Shixiong Zhu commented on SPARK-24279: -- Just did a quick look at your pom.xml. I think it's missing

[jira] [Commented] (SPARK-24472) Orc RecordReaderFactory throws IndexOutOfBoundsException

[ https://issues.apache.org/jira/browse/SPARK-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502914#comment-16502914 ] Shixiong Zhu commented on SPARK-24472: -- cc [~dongjoon] > Orc RecordReaderFactory throws

[jira] [Commented] (SPARK-24472) Orc RecordReaderFactory throws IndexOutOfBoundsException

[ https://issues.apache.org/jira/browse/SPARK-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502609#comment-16502609 ] Shixiong Zhu commented on SPARK-24472: -- cc [~cloud_fan] > Orc RecordReaderFactory throws

[jira] [Created] (SPARK-24472) Orc RecordReaderFactory throws IndexOutOfBoundsException

Shixiong Zhu created SPARK-24472: Summary: Orc RecordReaderFactory throws IndexOutOfBoundsException Key: SPARK-24472 URL: https://issues.apache.org/jira/browse/SPARK-24472 Project: Spark

[jira] [Assigned] (SPARK-24351) offsetLog/commitLog purge thresholdBatchId should be computed with current committed epoch but not currentBatchId in CP mode

[ https://issues.apache.org/jira/browse/SPARK-24351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24351: Assignee: huangtengfei > offsetLog/commitLog purge thresholdBatchId should be computed

[jira] [Resolved] (SPARK-24351) offsetLog/commitLog purge thresholdBatchId should be computed with current committed epoch but not currentBatchId in CP mode

[ https://issues.apache.org/jira/browse/SPARK-24351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24351. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21400

[jira] [Commented] (SPARK-23649) CSV schema inferring fails on some UTF-8 chars

[ https://issues.apache.org/jira/browse/SPARK-23649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495843#comment-16495843 ] Shixiong Zhu commented on SPARK-23649: -- [~cloud_fan] looks like this is fixed? > CSV schema

[jira] [Resolved] (SPARK-24332) Fix places reading 'spark.network.timeout' as milliseconds

[ https://issues.apache.org/jira/browse/SPARK-24332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24332. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21382

[jira] [Created] (SPARK-24337) Improve the error message for invalid SQL conf value

Shixiong Zhu created SPARK-24337: Summary: Improve the error message for invalid SQL conf value Key: SPARK-24337 URL: https://issues.apache.org/jira/browse/SPARK-24337 Project: Spark Issue

[jira] [Created] (SPARK-24332) Fix places reading 'spark.network.timeout' as milliseconds

Shixiong Zhu created SPARK-24332: Summary: Fix places reading 'spark.network.timeout' as milliseconds Key: SPARK-24332 URL: https://issues.apache.org/jira/browse/SPARK-24332 Project: Spark

[jira] [Resolved] (SPARK-24159) Enable no-data micro batches for streaming mapGroupswithState

[ https://issues.apache.org/jira/browse/SPARK-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24159. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21345

[jira] [Assigned] (SPARK-24159) Enable no-data micro batches for streaming mapGroupswithState

[ https://issues.apache.org/jira/browse/SPARK-24159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24159: Assignee: Tathagata Das > Enable no-data micro batches for streaming mapGroupswithState >

[jira] [Assigned] (SPARK-20538) Dataset.reduce operator should use withNewExecutionId (as foreach or foreachPartition)

[ https://issues.apache.org/jira/browse/SPARK-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-20538: Assignee: Soham Aurangabadkar > Dataset.reduce operator should use withNewExecutionId (as

[jira] [Resolved] (SPARK-20538) Dataset.reduce operator should use withNewExecutionId (as foreach or foreachPartition)

[ https://issues.apache.org/jira/browse/SPARK-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20538. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21316

[jira] [Assigned] (SPARK-24040) support single partition aggregates

[ https://issues.apache.org/jira/browse/SPARK-24040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24040: Assignee: Jose Torres > support single partition aggregates >

[jira] [Resolved] (SPARK-24040) support single partition aggregates

[ https://issues.apache.org/jira/browse/SPARK-24040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24040. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21239

[jira] [Commented] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474628#comment-16474628 ] Shixiong Zhu commented on SPARK-24067: -- Okey. Since this is turned off by default, I'm okey to

[jira] [Commented] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474532#comment-16474532 ] Shixiong Zhu commented on SPARK-24067: -- [~srowen] this sounds more like a new feature to me. The PR

[jira] [Created] (SPARK-24246) Improve AnalysisException by setting the cause when it's available

Shixiong Zhu created SPARK-24246: Summary: Improve AnalysisException by setting the cause when it's available Key: SPARK-24246 URL: https://issues.apache.org/jira/browse/SPARK-24246 Project: Spark

[jira] [Resolved] (SPARK-24214) StreamingRelationV2/StreamingExecutionRelation/ContinuousExecutionRelation.toJSON should not fail

[ https://issues.apache.org/jira/browse/SPARK-24214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24214. -- Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Commented] (SPARK-14146) Imported implicits can't be found in Spark REPL in some cases

[ https://issues.apache.org/jira/browse/SPARK-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468094#comment-16468094 ] Shixiong Zhu commented on SPARK-14146: -- [~ashawley] Yeah, we can close this ticket when we upgrade

[jira] [Created] (SPARK-24214) StreamingRelationV2/StreamingExecutionRelation/ContinuousExecutionRelation.toJSON should not fail

Shixiong Zhu created SPARK-24214: Summary: StreamingRelationV2/StreamingExecutionRelation/ContinuousExecutionRelation.toJSON should not fail Key: SPARK-24214 URL:

[jira] [Assigned] (SPARK-24061) [SS]TypedFilter is not supported in Continuous Processing

[ https://issues.apache.org/jira/browse/SPARK-24061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24061: Assignee: Wang Yanlin > [SS]TypedFilter is not supported in Continuous Processing >

[jira] [Resolved] (SPARK-24061) [SS]TypedFilter is not supported in Continuous Processing

[ https://issues.apache.org/jira/browse/SPARK-24061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24061. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21136

[jira] [Assigned] (SPARK-23565) Improved error message for when the number of sources for a query changes

[ https://issues.apache.org/jira/browse/SPARK-23565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-23565: Assignee: Patrick McGloin > Improved error message for when the number of sources for a

[jira] [Resolved] (SPARK-23565) Improved error message for when the number of sources for a query changes

[ https://issues.apache.org/jira/browse/SPARK-23565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-23565. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20946

[jira] [Resolved] (SPARK-23860) SQLAppStatusListener should handle the cases that an accumulator may be GCed

[ https://issues.apache.org/jira/browse/SPARK-23860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-23860. -- Resolution: Duplicate already fixed in SPARK-20652 > SQLAppStatusListener should handle the

[jira] [Created] (SPARK-23860) SQLAppStatusListener should handle the cases that an accumulator may be GCed

Shixiong Zhu created SPARK-23860: Summary: SQLAppStatusListener should handle the cases that an accumulator may be GCed Key: SPARK-23860 URL: https://issues.apache.org/jira/browse/SPARK-23860

[jira] [Commented] (SPARK-23848) Structured Streaming fails with nested UDTs

[ https://issues.apache.org/jira/browse/SPARK-23848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423071#comment-16423071 ] Shixiong Zhu commented on SPARK-23848: -- To fix your codes, you can just change "case Row(a: Int, b:

[jira] [Commented] (SPARK-23848) Structured Streaming fails with nested UDTs

[ https://issues.apache.org/jira/browse/SPARK-23848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423070#comment-16423070 ] Shixiong Zhu commented on SPARK-23848: -- [~josephkb] this is unfortunately because DataFrame is not a

<    1   2   3   4   5   6   7   8   9   10   >