Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206726280
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206727984
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala
---
@@ -53,7 +53,35 @@ class StreamingAggregationSuite
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206726108
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206732233
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/MemoryStateStore.scala
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206726327
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206732204
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/MemoryStateStore.scala
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206732522
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatefulOperatorsHelperSuite.scala
---
@@ -0,0 +1,121
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206730542
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206730845
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206729618
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206725943
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206732676
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatefulOperatorsHelperSuite.scala
---
@@ -0,0 +1,121
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206725731
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206728051
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatefulOperatorsHelperSuite.scala
---
@@ -0,0 +1,121
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206726027
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulOperatorsHelper.scala
---
@@ -0,0 +1,137 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206724964
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -871,6 +871,16 @@ object SQLConf {
.intConf
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206727384
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala
---
@@ -201,33 +200,37 @@ object WatermarkSupport
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206725041
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -871,6 +871,16 @@ object SQLConf {
.intConf
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
ping ^^^
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
hey @c-horn , I am ready to merge your PR, and to add you as coauthor i
think i need to know your email address i the github account. Can you provide
me
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21700
LGTM! I am merging it! Thank you for all the hard work. And my apologies
for not being able to give it time earlier to review
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r203573819
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
---
@@ -270,11 +273,43 @@ private[state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r203573621
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +66,143 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r203573306
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +66,143 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202927242
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +64,122 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202919419
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +64,122 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202925739
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -99,43 +102,84 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202926467
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +64,122 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202920067
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +64,122 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202922272
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +64,122 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202927538
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -99,43 +102,84 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202917836
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
---
@@ -270,11 +273,42 @@ private[state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202920190
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,21 +64,122 @@ class StateStoreSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r202918769
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
---
@@ -270,11 +273,42 @@ private[state
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21739
@zsxwing
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
Here is my solution based on my suggestion -
https://github.com/apache/spark/pull/21746
I stole your unit test from this PR :) Thank you! I will add you as a
co-author in that PR
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/21746
[WIP][SPARK-24699] [SS]Make watermarks work with Trigger.Once by saving
updated watermark to commit log
## What changes were proposed in this pull request?
Streaming queries with watermarks
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21739#discussion_r201597291
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/FlatMapGroupsWithStateExecHelper.scala
---
@@ -0,0 +1,225
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21744
@arunmahadevan I agree that this can be refactored later. I was trying to
do that, and then realized that it does not make sense to do that in the same
PR as this bug fix. thank you for reviewing
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21744
jenkins retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21744
jenkins retest this
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21744
@arunmahadevan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
The offset log contains the watermark value that is going to be used in the
batch corresponding to that offset. For example, "checkpoint/offsets/10" will
contain the watermark value
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21673
@arunmahadevan I made this PR as an attempt to incrementally improve the
control flow in ProgressReporter while fixing the bug here.
https://github.com/apache/spark/pull/21744
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/21744
Fix reporting of offsets in StreamExecution
## What changes were proposed in this pull request?
In ProgressReporter for streams, we use the `committedOffsets` as the
startOffset
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21739
jenkins retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21662#discussion_r201097931
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingGlobalLimitExec.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21662#discussion_r200532848
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
---
@@ -136,6 +137,11 @@ class IncrementalExecution
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21662#discussion_r200532939
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -354,6 +355,27 @@ abstract class SparkStrategies extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21739#discussion_r201092627
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/ObjectType.scala ---
@@ -43,7 +43,7 @@ case class ObjectType(cls: Class[_]) extends DataType
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/21739
[SPARK-22187][SS] Update unsaferow format for saved state such that we can
set timeouts when state is null
## What changes were proposed in this pull request?
Currently, the group state
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21701
@zsxwing @brkyvz
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21673
Thanks @arunmahadevan for making this PR. However, I dont like the solution
of adding another field as a workaround thus making the control flow harder to
reason about. I think the fundamental problem
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
I think the right solution is to record the updateat watermark in the
commit log, so that the updated watermark can be read back from the commit log
next time the stream is started
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21701
jenkins retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/21701
[SPARK-24730][SS] Add policy to choose max as global watermark when
streaming query has multiple watermarks
## What changes were proposed in this pull request?
Currently, when a streaming
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21560
LGTM assuming tests pass.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198380164
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousAggregationSuite.scala
---
@@ -50,6 +51,42 @@ class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198053471
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,118
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198052028
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -51,7 +51,7 @@ class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198052377
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceExec.scala
---
@@ -0,0 +1,54
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198055297
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,118
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198055537
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,108
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198056760
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousAggregationSuite.scala
---
@@ -50,6 +51,42 @@ class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198050994
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -98,6 +98,10 @@ class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21587#discussion_r196616725
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
---
@@ -68,50 +68,42 @@ case object AllTuples extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21587#discussion_r196615808
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
---
@@ -186,9 +180,8 @@ case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196586217
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,93
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196238635
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/shuffle/ContinuousShuffleReadRDD.scala
---
@@ -21,22 +21,25 @@ import
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196607009
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -51,7 +51,7 @@ class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196582424
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -350,7 +350,14 @@ object
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196586798
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,93
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196606185
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,93
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196584760
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceExec.scala
---
@@ -0,0 +1,57
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196580603
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -350,7 +350,14 @@ object
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196589311
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,93
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196609584
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,93
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196589618
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala
---
@@ -0,0 +1,93
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196581248
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -350,7 +350,14 @@ object
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21571
jenkins retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21571
Jenkins retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195917161
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
---
@@ -322,6 +338,45 @@ final class DataStreamWriter[T] private[sql
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195917011
--- Diff: python/pyspark/sql/streaming.py ---
@@ -1016,6 +1018,35 @@ def func_with_open_process_close(partition_id,
iterator):
self
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195882373
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
---
@@ -322,6 +338,45 @@ final class DataStreamWriter[T] private[sql
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195862257
--- Diff: dev/sparktestsupport/modules.py ---
@@ -389,19 +389,19 @@ def __hash__(self):
"python/pyspar
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195861903
--- Diff: python/pyspark/java_gateway.py ---
@@ -145,3 +145,26 @@ def do_server_auth(conn, auth_secret):
if reply != "ok":
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21477
Thank you every one. I merging this.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21571
@zsxwing @HyukjinKwon @HeartSaVioR @JoshRosen
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21477
@JoshRosen
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21477
@zsxwing @HyukjinKwon @HeartSaVioR
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21477#discussion_r195653451
--- Diff: python/pyspark/sql/tests.py ---
@@ -1885,6 +1885,263 @@ def test_query_manager_await_termination(self):
q.stop
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195597051
--- Diff: python/pyspark/sql/tests.py ---
@@ -269,6 +269,7 @@ def test_struct_field_type_name(self):
struct_field = StructField("a", I
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195597031
--- Diff: python/pyspark/sql/streaming.py ---
@@ -854,6 +856,20 @@ def trigger(self, processingTime=None, once=None,
continuous=None):
self
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21571#discussion_r195596984
--- Diff: dev/sparktestsupport/modules.py ---
@@ -389,19 +389,19 @@ def __hash__(self):
"python/pyspar
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/21571
[WIP][SPARK-24565][SS] Add API for in Structured Streaming for exposing
output rows of each microbatch as a DataFrame
## What changes were proposed in this pull request?
Currently, the micro
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20677
I will have to spend some time to look into the issue. I can do it later
next week. If there is a mistake, I apologize for it!! And I will fix
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21477#discussion_r193892652
--- Diff: python/pyspark/sql/tests.py ---
@@ -1884,7 +1885,164 @@ def test_query_manager_await_termination(self):
finally:
q.stop
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21477#discussion_r193892565
--- Diff: python/pyspark/sql/streaming.py ---
@@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None,
continuous=None):
self
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21477#discussion_r193892571
--- Diff: python/pyspark/sql/streaming.py ---
@@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None,
continuous=None):
self
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21477#discussion_r193892514
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonForeachWriter.scala
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed
101 - 200 of 6589 matches
Mail list logo
