[jira] [Created] (SPARK-36455) Provide an example of complex session window

Jungtaek Lim created SPARK-36455: Summary: Provide an example of complex session window Key: SPARK-36455 URL: https://issues.apache.org/jira/browse/SPARK-36455 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-36354) EventLogFileReaders should not complain in case of no event log files

[ https://issues.apache.org/jira/browse/SPARK-36354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-36354. -- Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-36354) EventLogFileReaders should not complain in case of no event log files

[ https://issues.apache.org/jira/browse/SPARK-36354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-36354: Assignee: Dongjoon Hyun > EventLogFileReaders should not complain in case of no event

[jira] [Created] (SPARK-36314) Update Sessionization example to use native support of session window

Jungtaek Lim created SPARK-36314: Summary: Update Sessionization example to use native support of session window Key: SPARK-36314 URL: https://issues.apache.org/jira/browse/SPARK-36314 Project: Spark

[jira] [Commented] (SPARK-36240) Graceful termination of Spark Structured Streaming queries

[ https://issues.apache.org/jira/browse/SPARK-36240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384771#comment-17384771 ] Jungtaek Lim commented on SPARK-36240: -- Hi, just FYI, fix version is set when the PR is submitted

[jira] [Updated] (SPARK-36240) Graceful termination of Spark Structured Streaming queries

[ https://issues.apache.org/jira/browse/SPARK-36240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-36240: - Fix Version/s: (was: 3.2.0) > Graceful termination of Spark Structured Streaming queries >

[jira] [Commented] (SPARK-10816) EventTime based sessionization (session window)

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384626#comment-17384626 ] Jungtaek Lim commented on SPARK-10816: -- I don't think I could finalize this one without your

[jira] [Resolved] (SPARK-10816) EventTime based sessionization (session window)

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-10816. -- Fix Version/s: 3.2.0 Assignee: Jungtaek Lim Resolution: Fixed I'm resolving

[jira] [Resolved] (SPARK-36172) Document session window in Structured Streaming guide doc

[ https://issues.apache.org/jira/browse/SPARK-36172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-36172. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33433

[jira] [Assigned] (SPARK-36172) Document session window in Structured Streaming guide doc

[ https://issues.apache.org/jira/browse/SPARK-36172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-36172: Assignee: Jungtaek Lim > Document session window in Structured Streaming guide doc >

[jira] [Assigned] (SPARK-34893) Support native session window

[ https://issues.apache.org/jira/browse/SPARK-34893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34893: Assignee: Jungtaek Lim > Support native session window > - >

[jira] [Resolved] (SPARK-34893) Support native session window

[ https://issues.apache.org/jira/browse/SPARK-34893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34893. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33081

[jira] [Created] (SPARK-36172) Document session window in Structured Streaming guide doc

Jungtaek Lim created SPARK-36172: Summary: Document session window in Structured Streaming guide doc Key: SPARK-36172 URL: https://issues.apache.org/jira/browse/SPARK-36172 Project: Spark

[jira] [Created] (SPARK-36157) TimeWindow expression: apply filter before project

Jungtaek Lim created SPARK-36157: Summary: TimeWindow expression: apply filter before project Key: SPARK-36157 URL: https://issues.apache.org/jira/browse/SPARK-36157 Project: Spark Issue

[jira] [Assigned] (SPARK-34892) Introduce MergingSortWithSessionWindowStateIterator sorting input rows and rows in state efficiently

[ https://issues.apache.org/jira/browse/SPARK-34892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34892: Assignee: Jungtaek Lim > Introduce MergingSortWithSessionWindowStateIterator sorting

[jira] [Resolved] (SPARK-34892) Introduce MergingSortWithSessionWindowStateIterator sorting input rows and rows in state efficiently

[ https://issues.apache.org/jira/browse/SPARK-34892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34892. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33077

[jira] [Resolved] (SPARK-35988) The implementation for RocksDBStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-35988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35988. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33187

[jira] [Assigned] (SPARK-35988) The implementation for RocksDBStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-35988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35988: Assignee: Yuanjian Li > The implementation for RocksDBStateStoreProvider >

[jira] [Resolved] (SPARK-35788) Metrics support for RocksDB instance

[ https://issues.apache.org/jira/browse/SPARK-35788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35788. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32934

[jira] [Assigned] (SPARK-35788) Metrics support for RocksDB instance

[ https://issues.apache.org/jira/browse/SPARK-35788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35788: Assignee: Yuanjian Li > Metrics support for RocksDB instance >

[jira] [Commented] (SPARK-35993) Flaky test: org.apache.spark.sql.execution.streaming.state.RocksDBSuite.ensure that concurrent update and cleanup consistent versions

[ https://issues.apache.org/jira/browse/SPARK-35993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373855#comment-17373855 ] Jungtaek Lim commented on SPARK-35993: -- Thanks for reporting [~gsomogyi]! The test is marked as

[jira] [Updated] (SPARK-35993) Flaky test: org.apache.spark.sql.execution.streaming.state.RocksDBSuite.ensure that concurrent update and cleanup consistent versions

[ https://issues.apache.org/jira/browse/SPARK-35993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-35993: - Affects Version/s: (was: 3.1.2) 3.2.0 > Flaky test: >

[jira] [Updated] (SPARK-35785) Cleanup support for RocksDB instance

[ https://issues.apache.org/jira/browse/SPARK-35785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-35785: - Fix Version/s: 3.2.0 > Cleanup support for RocksDB instance >

[jira] [Comment Edited] (SPARK-35915) Kafka doesn't recover from data loss

[ https://issues.apache.org/jira/browse/SPARK-35915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371817#comment-17371817 ] Jungtaek Lim edited comment on SPARK-35915 at 6/30/21, 5:27 AM: OK the

[jira] [Commented] (SPARK-35915) Kafka doesn't recover from data loss

[ https://issues.apache.org/jira/browse/SPARK-35915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371817#comment-17371817 ] Jungtaek Lim commented on SPARK-35915: -- OK the code comment was referring the old behavior, and it

[jira] [Assigned] (SPARK-35896) [SS] Include more granular metrics for stateful operators in StreamingQueryProgress

[ https://issues.apache.org/jira/browse/SPARK-35896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35896: Assignee: Venki Korukanti > [SS] Include more granular metrics for stateful operators in

[jira] [Resolved] (SPARK-35896) [SS] Include more granular metrics for stateful operators in StreamingQueryProgress

[ https://issues.apache.org/jira/browse/SPARK-35896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35896. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33091

[jira] [Commented] (SPARK-35915) Kafka doesn't recover from data loss

[ https://issues.apache.org/jira/browse/SPARK-35915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371210#comment-17371210 ] Jungtaek Lim commented on SPARK-35915: -- releaseConsumer just returns the Kafka consumer to the

[jira] [Resolved] (SPARK-35880) [SS] Track the number of duplicates dropped in streaming dedupe operator

[ https://issues.apache.org/jira/browse/SPARK-35880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35880. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33065

[jira] [Assigned] (SPARK-35880) [SS] Track the number of duplicates dropped in streaming dedupe operator

[ https://issues.apache.org/jira/browse/SPARK-35880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35880: Assignee: Venki Korukanti > [SS] Track the number of duplicates dropped in streaming

[jira] [Resolved] (SPARK-35894) Introduce new style enforce to not import scala.collection.Seq/IndexedSeq

[ https://issues.apache.org/jira/browse/SPARK-35894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35894. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33085

[jira] [Assigned] (SPARK-35894) Introduce new style enforce to not import scala.collection.Seq/IndexedSeq

[ https://issues.apache.org/jira/browse/SPARK-35894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35894: Assignee: Jungtaek Lim > Introduce new style enforce to not import

[jira] [Updated] (SPARK-35894) Introduce new style enforce to not import scala.collection.Seq/IndexedSeq

[ https://issues.apache.org/jira/browse/SPARK-35894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-35894: - Summary: Introduce new style enforce to not import scala.collection.Seq/IndexedSeq (was:

[jira] [Created] (SPARK-35894) Introduce new style enforce to not import scala.collection.Seq

Jungtaek Lim created SPARK-35894: Summary: Introduce new style enforce to not import scala.collection.Seq Key: SPARK-35894 URL: https://issues.apache.org/jira/browse/SPARK-35894 Project: Spark

[jira] [Assigned] (SPARK-35628) RocksDBFileManager - load checkpoint from DFS

[ https://issues.apache.org/jira/browse/SPARK-35628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35628: Assignee: Yuanjian Li > RocksDBFileManager - load checkpoint from DFS >

[jira] [Resolved] (SPARK-35628) RocksDBFileManager - load checkpoint from DFS

[ https://issues.apache.org/jira/browse/SPARK-35628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35628. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32767

[jira] [Updated] (SPARK-34893) Support native session window

[ https://issues.apache.org/jira/browse/SPARK-34893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-34893: - Description: This issue tracks effort on supporting native session window, on both batch query

[jira] [Created] (SPARK-35861) Introduce "prefix match scan" feature on state store

Jungtaek Lim created SPARK-35861: Summary: Introduce "prefix match scan" feature on state store Key: SPARK-35861 URL: https://issues.apache.org/jira/browse/SPARK-35861 Project: Spark Issue

[jira] [Updated] (SPARK-35799) Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec

[ https://issues.apache.org/jira/browse/SPARK-35799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-35799: - Fix Version/s: 3.1.3 > Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec

[jira] [Assigned] (SPARK-35799) Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec

[ https://issues.apache.org/jira/browse/SPARK-35799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35799: Assignee: Venki Korukanti > Fix the allUpdatesTimeMs metric measuring in

[jira] [Resolved] (SPARK-35799) Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec

[ https://issues.apache.org/jira/browse/SPARK-35799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35799. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32952

[jira] [Resolved] (SPARK-35565) Add a config for ignoring metadata directory of file stream sink

[ https://issues.apache.org/jira/browse/SPARK-35565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35565. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32702

[jira] [Assigned] (SPARK-35763) Minor refactor of StateStoreCustomMetric

[ https://issues.apache.org/jira/browse/SPARK-35763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35763: Assignee: Venki Korukanti > Minor refactor of StateStoreCustomMetric >

[jira] [Resolved] (SPARK-35763) Minor refactor of StateStoreCustomMetric

[ https://issues.apache.org/jira/browse/SPARK-35763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35763. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32914

[jira] [Resolved] (SPARK-35689) Add logging for null value retrieval for SymmetricHashJoinStateManager

[ https://issues.apache.org/jira/browse/SPARK-35689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35689. -- Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed > Add logging for null

[jira] [Commented] (SPARK-35689) Add logging for null value retrieval for SymmetricHashJoinStateManager

[ https://issues.apache.org/jira/browse/SPARK-35689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362207#comment-17362207 ] Jungtaek Lim commented on SPARK-35689: -- The issue was resolved at

[jira] [Commented] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361310#comment-17361310 ] Jungtaek Lim commented on SPARK-24295: -- [~parshabani] Please follow up SPARK-27188. But there're

[jira] [Assigned] (SPARK-35436) RocksDBFileManager - save checkpoint to DFS

[ https://issues.apache.org/jira/browse/SPARK-35436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35436: Assignee: Yuanjian Li > RocksDBFileManager - save checkpoint to DFS >

[jira] [Resolved] (SPARK-35436) RocksDBFileManager - save checkpoint to DFS

[ https://issues.apache.org/jira/browse/SPARK-35436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35436. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32582

[jira] [Commented] (SPARK-35602) Job crashes with java.io.UTFDataFormatException: encoded string too long

[ https://issues.apache.org/jira/browse/SPARK-35602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359693#comment-17359693 ] Jungtaek Lim commented on SPARK-35602: -- Very interested that the json representation of the schema

[jira] [Assigned] (SPARK-35602) Job crashes with java.io.UTFDataFormatException: encoded string too long

[ https://issues.apache.org/jira/browse/SPARK-35602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35602: Assignee: Kousuke Saruta > Job crashes with java.io.UTFDataFormatException: encoded

[jira] [Resolved] (SPARK-35602) Job crashes with java.io.UTFDataFormatException: encoded string too long

[ https://issues.apache.org/jira/browse/SPARK-35602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35602. -- Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-35312) Introduce new Option in Kafka source to specify minimum number of records to read per trigger

[ https://issues.apache.org/jira/browse/SPARK-35312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35312: Assignee: Satish Gopalani > Introduce new Option in Kafka source to specify minimum

[jira] [Resolved] (SPARK-35312) Introduce new Option in Kafka source to specify minimum number of records to read per trigger

[ https://issues.apache.org/jira/browse/SPARK-35312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35312. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32653

[jira] [Created] (SPARK-35611) Introduce the strategy on mismatched offset for start offset timestamp on Kafka data source

Jungtaek Lim created SPARK-35611: Summary: Introduce the strategy on mismatched offset for start offset timestamp on Kafka data source Key: SPARK-35611 URL: https://issues.apache.org/jira/browse/SPARK-35611

[jira] [Resolved] (SPARK-35172) The implementation of RocksDBCheckpointMetadata

[ https://issues.apache.org/jira/browse/SPARK-35172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35172. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32272

[jira] [Resolved] (SPARK-29223) Kafka source: offset by timestamp - allow specifying timestamp for "all partitions"

[ https://issues.apache.org/jira/browse/SPARK-29223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29223. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32609

[jira] [Assigned] (SPARK-29223) Kafka source: offset by timestamp - allow specifying timestamp for "all partitions"

[ https://issues.apache.org/jira/browse/SPARK-29223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-29223: Assignee: Jungtaek Lim > Kafka source: offset by timestamp - allow specifying timestamp

[jira] [Resolved] (SPARK-34888) Introduce UpdatingSessionIterator adjusting session information on elements

[ https://issues.apache.org/jira/browse/SPARK-34888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34888. -- Fix Version/s: 3.2.0 Assignee: Jungtaek Lim Resolution: Fixed > Introduce

[jira] [Commented] (SPARK-34888) Introduce UpdatingSessionIterator adjusting session information on elements

[ https://issues.apache.org/jira/browse/SPARK-34888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346115#comment-17346115 ] Jungtaek Lim commented on SPARK-34888: -- Issue resolved by pull request 31986

[jira] [Resolved] (SPARK-35311) Add exposed SS UI state information metrics to the documentation

[ https://issues.apache.org/jira/browse/SPARK-35311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35311. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32433

[jira] [Assigned] (SPARK-35311) Add exposed SS UI state information metrics to the documentation

[ https://issues.apache.org/jira/browse/SPARK-35311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-35311: Assignee: Gabor Somogyi > Add exposed SS UI state information metrics to the

[jira] [Resolved] (SPARK-34526) Skip checking glob path in FileStreamSink.hasMetadata

[ https://issues.apache.org/jira/browse/SPARK-34526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34526. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31638

[jira] [Assigned] (SPARK-34526) Skip checking glob path in FileStreamSink.hasMetadata

[ https://issues.apache.org/jira/browse/SPARK-34526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34526: Assignee: Yuanjian Li > Skip checking glob path in FileStreamSink.hasMetadata >

[jira] [Resolved] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34854. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31944

[jira] [Assigned] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34854: Assignee: Yijia Cui > Report metrics for streaming source through progress reporter with

[jira] [Commented] (SPARK-28247) Flaky test: "query without test harness" in ContinuousSuite

[ https://issues.apache.org/jira/browse/SPARK-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332810#comment-17332810 ] Jungtaek Lim commented on SPARK-28247: -- UPDATE: The new PR (#32316) from [~zsxwing] is merged to

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326460#comment-17326460 ] Jungtaek Lim commented on SPARK-34198: -- Please note that the decision was taken from community

[jira] [Created] (SPARK-34893) Support native session window

Jungtaek Lim created SPARK-34893: Summary: Support native session window Key: SPARK-34893 URL: https://issues.apache.org/jira/browse/SPARK-34893 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-34892) Introduce MergingSortWithSessionWindowStateIterator sorting input rows and rows in state efficiently

Jungtaek Lim created SPARK-34892: Summary: Introduce MergingSortWithSessionWindowStateIterator sorting input rows and rows in state efficiently Key: SPARK-34892 URL:

[jira] [Created] (SPARK-34891) Introduce state store manager for session window in streaming query

Jungtaek Lim created SPARK-34891: Summary: Introduce state store manager for session window in streaming query Key: SPARK-34891 URL: https://issues.apache.org/jira/browse/SPARK-34891 Project: Spark

[jira] [Created] (SPARK-34889) Introduce MergingSessionsIterator merging elements directly which belong to the same session

Jungtaek Lim created SPARK-34889: Summary: Introduce MergingSessionsIterator merging elements directly which belong to the same session Key: SPARK-34889 URL: https://issues.apache.org/jira/browse/SPARK-34889

[jira] [Created] (SPARK-34888) Introduce UpdatingSessionIterator adjusting session information on elements

Jungtaek Lim created SPARK-34888: Summary: Introduce UpdatingSessionIterator adjusting session information on elements Key: SPARK-34888 URL: https://issues.apache.org/jira/browse/SPARK-34888 Project:

[jira] [Commented] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309833#comment-17309833 ] Jungtaek Lim commented on SPARK-31754: -- Did anyone make a backup of problematic state with

[jira] [Commented] (SPARK-31427) Spark Structure streaming read data twice per every micro-batch.

[ https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301291#comment-17301291 ] Jungtaek Lim commented on SPARK-31427: -- If you have to use foreachBatch to enable specific

[jira] [Resolved] (SPARK-34471) Document DataStreamReader/Writer table APIs in Structured Streaming Programming Guide

[ https://issues.apache.org/jira/browse/SPARK-34471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34471. -- Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 31590

[jira] [Assigned] (SPARK-34471) Document DataStreamReader/Writer table APIs in Structured Streaming Programming Guide

[ https://issues.apache.org/jira/browse/SPARK-34471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34471: Assignee: Bo Zhang > Document DataStreamReader/Writer table APIs in Structured Streaming

[jira] [Assigned] (SPARK-34446) Add documentation for full outer and left semi stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-34446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34446: Assignee: Cheng Su > Add documentation for full outer and left semi stream-stream join >

[jira] [Resolved] (SPARK-34446) Add documentation for full outer and left semi stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-34446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34446. -- Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 31572

[jira] [Resolved] (SPARK-34426) Add driver and executors POD logs to integration tests log when the test fails

[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34426. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31561

[jira] [Assigned] (SPARK-34426) Add driver and executors POD logs to integration tests log when the test fails

[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34426: Assignee: Attila Zsolt Piros > Add driver and executors POD logs to integration tests

[jira] [Commented] (SPARK-32883) Stream-stream join improvement

[ https://issues.apache.org/jira/browse/SPARK-32883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285071#comment-17285071 ] Jungtaek Lim commented on SPARK-32883: -- Thanks for the head-up. [~chengsu] please ping me on the PR

[jira] [Comment Edited] (SPARK-10816) EventTime based sessionization

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284964#comment-17284964 ] Jungtaek Lim edited comment on SPARK-10816 at 2/15/21, 11:29 PM: -

[jira] [Commented] (SPARK-10816) EventTime based sessionization

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284964#comment-17284964 ] Jungtaek Lim commented on SPARK-10816: -- Probably the code size is different because the design is

[jira] [Commented] (SPARK-10816) EventTime based sessionization

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284959#comment-17284959 ] Jungtaek Lim commented on SPARK-10816: -- cc. [~XuanYuan] > EventTime based sessionization >

[jira] [Commented] (SPARK-10816) EventTime based sessionization

[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284950#comment-17284950 ] Jungtaek Lim commented on SPARK-10816: -- First of all, I'd like to say thanks for finding the value

[jira] [Comment Edited] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284507#comment-17284507 ] Jungtaek Lim edited comment on SPARK-34427 at 2/15/21, 12:50 AM: - OK I

[jira] [Comment Edited] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284507#comment-17284507 ] Jungtaek Lim edited comment on SPARK-34427 at 2/15/21, 12:49 AM: - OK I

[jira] [Commented] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284507#comment-17284507 ] Jungtaek Lim commented on SPARK-34427: -- OK I agree it's going to meaningless argue. I should have

[jira] [Commented] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284496#comment-17284496 ] Jungtaek Lim commented on SPARK-34427: -- This assignee case is quite different from what I've seen

[jira] [Comment Edited] (SPARK-34198) Add RocksDB StateStore as external module

[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284491#comment-17284491 ] Jungtaek Lim edited comment on SPARK-34198 at 2/14/21, 8:02 PM: Thanks

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284491#comment-17284491 ] Jungtaek Lim commented on SPARK-34198: -- Thanks for considering it. I think it would be the best

[jira] [Comment Edited] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284362#comment-17284362 ] Jungtaek Lim edited comment on SPARK-34427 at 2/14/21, 7:28 AM: I'd say

[jira] [Commented] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284363#comment-17284363 ] Jungtaek Lim commented on SPARK-34427: -- If you'd like to say assignee is to avoid stepping on

[jira] [Commented] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284362#comment-17284362 ] Jungtaek Lim commented on SPARK-34427: -- I'd say SPARK-10816 has no active progress so far because

[jira] [Commented] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284342#comment-17284342 ] Jungtaek Lim commented on SPARK-34427: -- Just to make clear, SPARK-10816 has SPIP docs from two

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284336#comment-17284336 ] Jungtaek Lim commented on SPARK-34198: -- Now we have consensus on having this, which is great.

[jira] [Assigned] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34427: Assignee: (was: L. C. Hsieh) > Session window support in SS >

[jira] [Resolved] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34427. -- Resolution: Duplicate > Session window support in SS > > >

[jira] [Commented] (SPARK-34427) Session window support in SS

[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284334#comment-17284334 ] Jungtaek Lim commented on SPARK-34427: -- We can close this one and continue from SPARK-10816. This

<    4   5   6   7   8   9   10   11   12   13   >