[jira] [Created] (SPARK-35800) Improving testability of GroupState in streaming flatMapGroupsWithState

Tathagata Das created SPARK-35800: - Summary: Improving testability of GroupState in streaming flatMapGroupsWithState Key: SPARK-35800 URL: https://issues.apache.org/jira/browse/SPARK-35800 Project:

[jira] [Created] (SPARK-34962) Explicit representation of star in MergeIntoTable's Update and Insert action

Tathagata Das created SPARK-34962: - Summary: Explicit representation of star in MergeIntoTable's Update and Insert action Key: SPARK-34962 URL: https://issues.apache.org/jira/browse/SPARK-34962

[jira] [Updated] (SPARK-34720) Incorrect star expansion logic MERGE INSERT * / UPDATE *

[ https://issues.apache.org/jira/browse/SPARK-34720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-34720: -- Summary: Incorrect star expansion logic MERGE INSERT * / UPDATE * (was: Incorrect star

[jira] [Created] (SPARK-34720) Incorrect star expansion logic MERGE INSERT / UPDATE *

Tathagata Das created SPARK-34720: - Summary: Incorrect star expansion logic MERGE INSERT / UPDATE * Key: SPARK-34720 URL: https://issues.apache.org/jira/browse/SPARK-34720 Project: Spark

[jira] [Resolved] (SPARK-32585) Support scala enumeration in ScalaReflection

[ https://issues.apache.org/jira/browse/SPARK-32585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-32585. --- Fix Version/s: 3.1.0 Resolution: Done > Support scala enumeration in ScalaReflection

[jira] [Resolved] (SPARK-32794) Rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 streaming sources

[ https://issues.apache.org/jira/browse/SPARK-32794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-32794. --- Fix Version/s: (was: 2.4.8) 2.4.7 Resolution: Fixed Issue

[jira] [Updated] (SPARK-32794) Rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 streaming sources

[ https://issues.apache.org/jira/browse/SPARK-32794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-32794: -- Fix Version/s: 3.0.2 3.1.0 2.4.8 > Rare corner case

[jira] [Created] (SPARK-32794) Rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 streaming sources

Tathagata Das created SPARK-32794: - Summary: Rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 streaming sources Key: SPARK-32794 URL:

[jira] [Updated] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

[ https://issues.apache.org/jira/browse/SPARK-32017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-32017: -- Fix Version/s: (was: 3.0.1) > Make Pyspark Hadoop 3.2+ Variant available in PyPI >

[jira] [Commented] (SPARK-30657) Streaming limit after streaming dropDuplicates can throw error

[ https://issues.apache.org/jira/browse/SPARK-30657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027910#comment-17027910 ] Tathagata Das commented on SPARK-30657: --- This fix by itself (separate from the fix for

[jira] [Commented] (SPARK-30658) Limit after on streaming dataframe before streaming agg returns wrong results

[ https://issues.apache.org/jira/browse/SPARK-30658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027906#comment-17027906 ] Tathagata Das commented on SPARK-30658: --- I am a little afraid to backport this because this is

[jira] [Commented] (SPARK-30658) Limit after on streaming dataframe before streaming agg returns wrong results

[ https://issues.apache.org/jira/browse/SPARK-30658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027907#comment-17027907 ] Tathagata Das commented on SPARK-30658: --- Fixed in this PR

[jira] [Resolved] (SPARK-30657) Streaming limit after streaming dropDuplicates can throw error

[ https://issues.apache.org/jira/browse/SPARK-30657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-30657. --- Resolution: Fixed > Streaming limit after streaming dropDuplicates can throw error >

[jira] [Resolved] (SPARK-30658) Limit after on streaming dataframe before streaming agg returns wrong results

[ https://issues.apache.org/jira/browse/SPARK-30658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-30658. --- Resolution: Fixed > Limit after on streaming dataframe before streaming agg returns wrong

[jira] [Assigned] (SPARK-30658) Limit after on streaming dataframe before streaming agg returns wrong results

[ https://issues.apache.org/jira/browse/SPARK-30658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-30658: - Assignee: Tathagata Das > Limit after on streaming dataframe before streaming agg

[jira] [Updated] (SPARK-30658) Limit after on streaming dataframe before streaming agg returns wrong results

[ https://issues.apache.org/jira/browse/SPARK-30658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-30658: -- Description: Limit before a streaming aggregate (i.e. {{df.limit(5).groupBy().count()}}) in

[jira] [Resolved] (SPARK-29438) Failed to get state store in stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-29438. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26162

[jira] [Assigned] (SPARK-29438) Failed to get state store in stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-29438: - Assignee: Jungtaek Lim > Failed to get state store in stream-stream join >

[jira] [Created] (SPARK-30658) Limit after on streaming dataframe before streaming agg returns wrong results

Tathagata Das created SPARK-30658: - Summary: Limit after on streaming dataframe before streaming agg returns wrong results Key: SPARK-30658 URL: https://issues.apache.org/jira/browse/SPARK-30658

[jira] [Created] (SPARK-30657) Streaming limit after streaming dropDuplicates can throw error

Tathagata Das created SPARK-30657: - Summary: Streaming limit after streaming dropDuplicates can throw error Key: SPARK-30657 URL: https://issues.apache.org/jira/browse/SPARK-30657 Project: Spark

[jira] [Resolved] (SPARK-30609) Allow default merge command resolution to be bypassed by DSv2 sources

[ https://issues.apache.org/jira/browse/SPARK-30609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-30609. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27326

[jira] [Assigned] (SPARK-30609) Allow default merge command resolution to be bypassed by DSv2 sources

[ https://issues.apache.org/jira/browse/SPARK-30609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-30609: - Assignee: Tathagata Das > Allow default merge command resolution to be bypassed by

[jira] [Created] (SPARK-30609) Allow default merge command resolution to be bypassed by DSv2 sources

Tathagata Das created SPARK-30609: - Summary: Allow default merge command resolution to be bypassed by DSv2 sources Key: SPARK-30609 URL: https://issues.apache.org/jira/browse/SPARK-30609 Project:

[jira] [Resolved] (SPARK-27453) DataFrameWriter.partitionBy is Silently Dropped by DSV1

[ https://issues.apache.org/jira/browse/SPARK-27453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-27453. --- Resolution: Fixed Fix Version/s: 2.4.2 3.0.0 Issue resolved by

[jira] [Created] (SPARK-26629) Error with multiple file stream in a query + restart on a batch that has no data for one file stream

Tathagata Das created SPARK-26629: - Summary: Error with multiple file stream in a query + restart on a batch that has no data for one file stream Key: SPARK-26629 URL:

[jira] [Created] (SPARK-26425) Add more constraint checks in file streaming source to avoid checkpoint corruption

Tathagata Das created SPARK-26425: - Summary: Add more constraint checks in file streaming source to avoid checkpoint corruption Key: SPARK-26425 URL: https://issues.apache.org/jira/browse/SPARK-26425

[jira] [Created] (SPARK-25752) Add trait to easily whitelist logical operators that produce named output from CleanupAliases

Tathagata Das created SPARK-25752: - Summary: Add trait to easily whitelist logical operators that produce named output from CleanupAliases Key: SPARK-25752 URL: https://issues.apache.org/jira/browse/SPARK-25752

[jira] [Assigned] (SPARK-25639) Add documentation on foreachBatch, and multiple watermark policy

[ https://issues.apache.org/jira/browse/SPARK-25639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-25639: - Assignee: Tathagata Das > Add documentation on foreachBatch, and multiple watermark

[jira] [Resolved] (SPARK-25639) Add documentation on foreachBatch, and multiple watermark policy

[ https://issues.apache.org/jira/browse/SPARK-25639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-25639. --- Resolution: Fixed Fix Version/s: 2.4.1 Issue resolved by pull request 22627

[jira] [Created] (SPARK-25639) Add documentation on foreachBatch, and multiple watermark policy

Tathagata Das created SPARK-25639: - Summary: Add documentation on foreachBatch, and multiple watermark policy Key: SPARK-25639 URL: https://issues.apache.org/jira/browse/SPARK-25639 Project: Spark

[jira] [Resolved] (SPARK-25399) Reusing execution threads from continuous processing for microbatch streaming can result in correctness issues

[ https://issues.apache.org/jira/browse/SPARK-25399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-25399. --- Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Issue resolved by

[jira] [Assigned] (SPARK-25399) Reusing execution threads from continuous processing for microbatch streaming can result in correctness issues

[ https://issues.apache.org/jira/browse/SPARK-25399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-25399: - Assignee: Mukul Murthy > Reusing execution threads from continuous processing for

[jira] [Commented] (SPARK-25106) A new Kafka consumer gets created for every batch

[ https://issues.apache.org/jira/browse/SPARK-25106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591170#comment-16591170 ] Tathagata Das commented on SPARK-25106: --- This is interesting! I dont know how this could be

[jira] [Resolved] (SPARK-25204) rate source test is flaky

[ https://issues.apache.org/jira/browse/SPARK-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-25204. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22191

[jira] [Assigned] (SPARK-25204) rate source test is flaky

[ https://issues.apache.org/jira/browse/SPARK-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-25204: - Assignee: Jose Torres > rate source test is flaky > - > >

[jira] [Resolved] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

[ https://issues.apache.org/jira/browse/SPARK-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-25184. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22182

[jira] [Assigned] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

[ https://issues.apache.org/jira/browse/SPARK-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-25184: - Assignee: Tathagata Das > Flaky test: FlatMapGroupsWithState "streaming with

[jira] [Created] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

Tathagata Das created SPARK-25184: - Summary: Flaky test: FlatMapGroupsWithState "streaming with processing time timeout" Key: SPARK-25184 URL: https://issues.apache.org/jira/browse/SPARK-25184

[jira] [Commented] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588234#comment-16588234 ] Tathagata Das commented on SPARK-24763: --- They will be. The merge script always puts the major

[jira] [Assigned] (SPARK-24441) Expose total estimated size of states in HDFSBackedStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24441: - Assignee: Jungtaek Lim > Expose total estimated size of states in

[jira] [Resolved] (SPARK-24441) Expose total estimated size of states in HDFSBackedStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24441. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21469

[jira] [Assigned] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24763: - Assignee: Jungtaek Lim (was: Tathagata Das) > Remove redundant key data from value in

[jira] [Assigned] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24763: - Assignee: Tathagata Das > Remove redundant key data from value in streaming

[jira] [Resolved] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24763. --- Resolution: Done Fix Version/s: 3.0.0 2.4.0 > Remove redundant

[jira] [Resolved] (SPARK-24699) Watermark / Append mode should work with Trigger.Once

[ https://issues.apache.org/jira/browse/SPARK-24699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24699. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21746

[jira] [Assigned] (SPARK-24699) Watermark / Append mode should work with Trigger.Once

[ https://issues.apache.org/jira/browse/SPARK-24699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24699: - Assignee: Tathagata Das > Watermark / Append mode should work with Trigger.Once >

[jira] [Resolved] (SPARK-22187) Update unsaferow format for saved state such that we can set timeouts when state is null

[ https://issues.apache.org/jira/browse/SPARK-22187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-22187. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21739

[jira] [Resolved] (SPARK-24717) Split out min retain version of state for memory in HDFSBackedStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-24717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24717. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21700

[jira] [Assigned] (SPARK-24717) Split out min retain version of state for memory in HDFSBackedStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-24717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24717: - Assignee: Jungtaek Lim > Split out min retain version of state for memory in >

[jira] [Resolved] (SPARK-24697) Fix the reported start offsets in streaming query progress

[ https://issues.apache.org/jira/browse/SPARK-24697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24697. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21744

[jira] [Assigned] (SPARK-24697) Fix the reported start offsets in streaming query progress

[ https://issues.apache.org/jira/browse/SPARK-24697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24697: - Assignee: Tathagata Das > Fix the reported start offsets in streaming query progress >

[jira] [Resolved] (SPARK-24730) Add policy to choose max as global watermark when streaming query has multiple watermarks

[ https://issues.apache.org/jira/browse/SPARK-24730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24730. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21701

[jira] [Resolved] (SPARK-24662) Structured Streaming should support LIMIT

[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24662. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21662

[jira] [Assigned] (SPARK-24662) Structured Streaming should support LIMIT

[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24662: - Assignee: Mukul Murthy > Structured Streaming should support LIMIT >

[jira] [Created] (SPARK-24730) Add policy to choose max as global watermark when streaming query has multiple watermarks

Tathagata Das created SPARK-24730: - Summary: Add policy to choose max as global watermark when streaming query has multiple watermarks Key: SPARK-24730 URL: https://issues.apache.org/jira/browse/SPARK-24730

[jira] [Resolved] (SPARK-24386) implement continuous processing coalesce(1)

[ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24386. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21560

[jira] [Assigned] (SPARK-24386) implement continuous processing coalesce(1)

[ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24386: - Assignee: Jose Torres > implement continuous processing coalesce(1) >

[jira] [Resolved] (SPARK-24396) Add Structured Streaming ForeachWriter for python

[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24396. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21477

[jira] [Updated] (SPARK-24565) Add API for in Structured Streaming for exposing output rows of each microbatch as a DataFrame

[ https://issues.apache.org/jira/browse/SPARK-24565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-24565: -- Description: Currently, the micro-batches in the MicroBatchExecution is not exposed to the

[jira] [Created] (SPARK-24565) Add API for in Structured Streaming for exposing output rows of each microbatch as a DataFrame

Tathagata Das created SPARK-24565: - Summary: Add API for in Structured Streaming for exposing output rows of each microbatch as a DataFrame Key: SPARK-24565 URL: https://issues.apache.org/jira/browse/SPARK-24565

[jira] [Resolved] (SPARK-24453) Fix error recovering from the failure in a no-data batch

[ https://issues.apache.org/jira/browse/SPARK-24453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24453. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21491

[jira] [Created] (SPARK-24453) Fix error recovering from the failure in a no-data batch

Tathagata Das created SPARK-24453: - Summary: Fix error recovering from the failure in a no-data batch Key: SPARK-24453 URL: https://issues.apache.org/jira/browse/SPARK-24453 Project: Spark

[jira] [Resolved] (SPARK-24397) Add TaskContext.getLocalProperties in Python

[ https://issues.apache.org/jira/browse/SPARK-24397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24397. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21437

[jira] [Commented] (SPARK-24396) Add Structured Streaming ForeachWriter for python

[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491415#comment-16491415 ] Tathagata Das commented on SPARK-24396: --- TaskContext.getLocalProperty in Python is needed for

[jira] [Comment Edited] (SPARK-24396) Add Structured Streaming ForeachWriter for python

[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491415#comment-16491415 ] Tathagata Das edited comment on SPARK-24396 at 5/26/18 12:07 AM: -

[jira] [Updated] (SPARK-24397) Add TaskContext.getLocalProperties in Python

[ https://issues.apache.org/jira/browse/SPARK-24397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-24397: -- Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-24396) > Add

[jira] [Created] (SPARK-24397) Add TaskContext.getLocalProperties in Python

Tathagata Das created SPARK-24397: - Summary: Add TaskContext.getLocalProperties in Python Key: SPARK-24397 URL: https://issues.apache.org/jira/browse/SPARK-24397 Project: Spark Issue Type:

[jira] [Created] (SPARK-24396) Add Structured Streaming ForeachWriter for python

Tathagata Das created SPARK-24396: - Summary: Add Structured Streaming ForeachWriter for python Key: SPARK-24396 URL: https://issues.apache.org/jira/browse/SPARK-24396 Project: Spark Issue

[jira] [Resolved] (SPARK-23416) Flaky test: KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-23416. --- Resolution: Fixed Fix Version/s: (was: 2.3.0) 3.0.0 Issue

[jira] [Assigned] (SPARK-23416) Flaky test: KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-23416: - Assignee: Jose Torres > Flaky test: KafkaSourceStressForDontFailOnDataLossSuite.stress

[jira] [Assigned] (SPARK-24234) create the bottom-of-task RDD with row buffer

[ https://issues.apache.org/jira/browse/SPARK-24234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24234: - Assignee: Jose Torres > create the bottom-of-task RDD with row buffer >

[jira] [Resolved] (SPARK-24234) create the bottom-of-task RDD with row buffer

[ https://issues.apache.org/jira/browse/SPARK-24234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24234. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21337

[jira] [Resolved] (SPARK-23503) continuous execution should sequence committed epochs

[ https://issues.apache.org/jira/browse/SPARK-23503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-23503. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 20936

[jira] [Resolved] (SPARK-24158) Enable no-data micro batches for streaming joins

[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24158. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21253

[jira] [Resolved] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24157. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21220

[jira] [Assigned] (SPARK-24039) remove restarting iterators hack

[ https://issues.apache.org/jira/browse/SPARK-24039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24039: - Assignee: Jose Torres > remove restarting iterators hack >

[jira] [Resolved] (SPARK-24039) remove restarting iterators hack

[ https://issues.apache.org/jira/browse/SPARK-24039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24039. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21200

[jira] [Created] (SPARK-24159) Enable no-data micro batches for streaming mapGroupswithState

Tathagata Das created SPARK-24159: - Summary: Enable no-data micro batches for streaming mapGroupswithState Key: SPARK-24159 URL: https://issues.apache.org/jira/browse/SPARK-24159 Project: Spark

[jira] [Created] (SPARK-24158) Enable no-data micro batches for streaming joins

Tathagata Das created SPARK-24158: - Summary: Enable no-data micro batches for streaming joins Key: SPARK-24158 URL: https://issues.apache.org/jira/browse/SPARK-24158 Project: Spark Issue

[jira] [Assigned] (SPARK-24158) Enable no-data micro batches for streaming joins

[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24158: - Assignee: Tathagata Das > Enable no-data micro batches for streaming joins >

[jira] [Created] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

Tathagata Das created SPARK-24156: - Summary: Enable no-data micro batches for more eager streaming state clean up Key: SPARK-24156 URL: https://issues.apache.org/jira/browse/SPARK-24156 Project:

[jira] [Created] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

Tathagata Das created SPARK-24157: - Summary: Enable no-data micro batches for streaming aggregation and deduplication Key: SPARK-24157 URL: https://issues.apache.org/jira/browse/SPARK-24157 Project:

[jira] [Resolved] (SPARK-18791) Stream-Stream Joins

[ https://issues.apache.org/jira/browse/SPARK-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-18791. --- Resolution: Done Fix Version/s: 2.3.0 > Stream-Stream Joins > --- > >

[jira] [Resolved] (SPARK-24094) Change description strings of v2 streaming sources to reflect the change

[ https://issues.apache.org/jira/browse/SPARK-24094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24094. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21160

[jira] [Created] (SPARK-24094) Change description strings of v2 streaming sources to reflect the change

Tathagata Das created SPARK-24094: - Summary: Change description strings of v2 streaming sources to reflect the change Key: SPARK-24094 URL: https://issues.apache.org/jira/browse/SPARK-24094 Project:

[jira] [Resolved] (SPARK-24050) StreamingQuery does not calculate input / processing rates in some cases

[ https://issues.apache.org/jira/browse/SPARK-24050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24050. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21126

[jira] [Resolved] (SPARK-24038) refactor continuous write exec to its own class

[ https://issues.apache.org/jira/browse/SPARK-24038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24038. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21116

[jira] [Assigned] (SPARK-24038) refactor continuous write exec to its own class

[ https://issues.apache.org/jira/browse/SPARK-24038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24038: - Assignee: Jose Torres > refactor continuous write exec to its own class >

[jira] [Resolved] (SPARK-24056) Make consumer creation lazy in Kafka source for Structured streaming

[ https://issues.apache.org/jira/browse/SPARK-24056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24056. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21134

[jira] [Created] (SPARK-24056) Make consumer creation lazy in Kafka source for Structured streaming

Tathagata Das created SPARK-24056: - Summary: Make consumer creation lazy in Kafka source for Structured streaming Key: SPARK-24056 URL: https://issues.apache.org/jira/browse/SPARK-24056 Project:

[jira] [Resolved] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-23004. --- Resolution: Fixed Fix Version/s: 2.3.1 3.0.0 Issue resolved by

[jira] [Updated] (SPARK-24050) StreamingQuery does not calculate input / processing rates in some cases

[ https://issues.apache.org/jira/browse/SPARK-24050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-24050: -- Description: In some streaming queries, the input and processing rates are not calculated at

[jira] [Created] (SPARK-24050) StreamingQuery does not calculate input / processing rates in some cases

Tathagata Das created SPARK-24050: - Summary: StreamingQuery does not calculate input / processing rates in some cases Key: SPARK-24050 URL: https://issues.apache.org/jira/browse/SPARK-24050 Project:

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Target Version/s: 2.3.1, 2.4.0 (was: 2.3.1) > Structured Streaming raise

[jira] [Comment Edited] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447440#comment-16447440 ] Tathagata Das edited comment on SPARK-23004 at 4/23/18 1:16 AM:

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Component/s: (was: Input/Output) Structured Streaming > Structured

[jira] [Assigned] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-23004: - Assignee: Tathagata Das > Structured Streaming raise "llegalStateException: Cannot

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Target Version/s: 2.3.1 > Structured Streaming raise "llegalStateException: Cannot remove

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Affects Version/s: 2.1.0 2.1.1 2.1.2

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Description: {{A structured streaming query with streaming aggregations can throw the

  1   2   3   4   5   6   7   8   9   10   >