Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17179
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106736186
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala
---
@@ -0,0 +1,270 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106569878
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logical.L
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106569867
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logical.L
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106568435
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logical.L
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106568427
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/streaming/KeyedStateTimeout.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Softwar
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106568386
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -92,27 +121,33 @@ import
org.apache.spark.sql.catalyst.plans.logical.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106568279
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/KeyedStateImpl.scala
---
@@ -60,6 +82,45 @@ private[sql] class KeyedStateImpl[S](op
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106567105
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala
---
@@ -0,0 +1,270 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106566487
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
---
@@ -50,6 +50,8 @@ trait StateStore {
/** Get the
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106566096
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -361,18 +435,20 @@ class KeyValueGroupedDataset[K, V] private[sql](
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106566078
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -284,6 +322,38 @@ class KeyValueGroupedDataset[K, V] private[sql](
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106566062
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -249,6 +250,43 @@ class KeyValueGroupedDataset[K, V] private[sql](
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106566088
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -298,12 +368,14 @@ class KeyValueGroupedDataset[K, V] private[sql](
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055705
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -298,12 +368,14 @@ class KeyValueGroupedDataset[K, V] private[sql
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106045620
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/KeyedStateImpl.scala
---
@@ -60,6 +82,45 @@ private[sql] class KeyedStateImpl[S]
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106051316
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala
---
@@ -0,0 +1,270 @@
+/*
+ * Licensed
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106054568
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
---
@@ -50,6 +50,8 @@ trait StateStore {
/** Get t
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055666
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -361,18 +435,20 @@ class KeyValueGroupedDataset[K, V] private[sql
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106045218
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -92,27 +121,33 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055599
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -284,6 +322,38 @@ class KeyValueGroupedDataset[K, V] private[sql]
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106036389
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/streaming/KeyedStateTimeout.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Soft
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055580
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -249,6 +250,43 @@ class KeyValueGroupedDataset[K, V] private[sql]
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105991219
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105990971
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105990594
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -249,6 +250,43 @@ class KeyValueGroupedDataset[K, V] private[sql
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105989379
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala
---
@@ -0,0 +1,270 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105989232
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logical.L
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105989084
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logical.L
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105988058
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -298,12 +368,14 @@ class KeyValueGroupedDataset[K, V] private[sql](
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105987532
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -249,6 +250,43 @@ class KeyValueGroupedDataset[K, V] private[sql](
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105822080
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105821698
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -298,12 +368,14 @@ class KeyValueGroupedDataset[K, V] private[sq
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105823059
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala
---
@@ -0,0 +1,270 @@
+/*
+ * Licensed
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105822317
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105822109
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,50 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105821496
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -249,6 +250,43 @@ class KeyValueGroupedDataset[K, V] private[sql
Github user amitsela commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105396443
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,49 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user amitsela commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105395680
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,49 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105363636
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala
---
@@ -256,94 +257,6 @@ case class StateStoreSaveExec(
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105315322
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/MapGroupsWithStateSuite.scala
---
@@ -83,6 +88,34 @@ class MapGroupsWithStateSuite extends
S
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105315158
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -230,16 +231,19 @@ class KeyValueGroupedDataset[K, V] private[sql](
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105315079
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,49 @@ import
org.apache.spark.sql.catalyst.plans.logical.L
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105314636
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,49 @@ import
org.apache.spark.sql.catalyst.plans.logical.L
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105306350
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala
---
@@ -317,6 +317,9 @@ case class MapGroups(
/** Internal
Github user amitsela commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105230719
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/MapGroupsWithStateSuite.scala
---
@@ -83,6 +88,34 @@ class MapGroupsWithStateSuite extend
Github user amitsela commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105228392
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,49 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user amitsela commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105227799
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -61,25 +65,49 @@ import
org.apache.spark.sql.catalyst.plans.logic
Github user amitsela commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105224804
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -230,16 +231,19 @@ class KeyValueGroupedDataset[K, V] private[sq
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/17179
[SPARK-19067][SS] Processing-time-based timeout in MapGroupsWithState
## What changes were proposed in this pull request?
When a key does not get any new data in `mapGroupsWithState`, the mapp
50 matches
Mail list logo
