Github user tdas commented on the issue:
https://github.com/apache/spark/pull/23060
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22627#discussion_r223479786
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -2709,6 +2935,78 @@ write.stream(aggDF, "memory", outputMode =
"complete"
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22627#discussion_r223479414
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -1989,22 +2026,211 @@ head(sql("select * from aggregates"))
-#
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22627
@holdenk yeah, i intend to backport this to 2.4
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22627#discussion_r223456294
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -2709,6 +2935,78 @@ write.stream(aggDF, "memory", outputMode =
"complete"
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22627#discussion_r223456079
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -1989,22 +2026,211 @@ head(sql("select * from aggregates"))
-#
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22627
@zsxwing
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/22627
[SPARK-25639] [DOCS] Added docs for foreachBatch, python foreach and
multiple watermarks
## What changes were proposed in this pull request?
Added
- Python foreach
- Scala, Java
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22507
LGTM. Thanks for the explanation.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22507#discussion_r220061053
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -874,6 +874,57 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22507#discussion_r220059919
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -874,6 +874,57 @@ abstract class
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22476
LGTM.
Please make sure to merge it to 2.4
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22386
LGTM. Just one super nit.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22386#discussion_r216781599
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala
---
@@ -74,9 +75,14 @@ class StateStoreRDD[T: ClassTag
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22293
I was afraid that this would be flaky. Glad you found a solution quickly.
just one comment to improve code readability
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22293#discussion_r214208589
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -652,62 +654,67 @@ abstract class
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22042
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22230
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22207#discussion_r212706003
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala
---
@@ -0,0 +1,281
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22207#discussion_r212706908
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala
---
@@ -0,0 +1,281
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22207#discussion_r212706859
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala
---
@@ -0,0 +1,281
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22207#discussion_r212526645
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala
---
@@ -0,0 +1,281
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22207#discussion_r212526373
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala
---
@@ -0,0 +1,281
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r212507190
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -160,6 +160,23 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r212522664
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -161,6 +161,22 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r212521083
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -239,56 +335,74 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r212522432
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -161,6 +161,22 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r212504622
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReadSupport.scala
---
@@ -331,6 +331,7 @@ private[kafka010
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22191
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/22182
[SPARK-25184][SS] Fixed race condition in StreamExecution that caused flaky
test in FlatMapGroupsWithState
## What changes were proposed in this pull request?
The race condition that caused
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211805733
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -597,6 +614,254 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211801632
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -346,11 +437,40 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211802112
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -250,33 +341,39 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211801676
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -250,33 +341,39 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211801968
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -250,33 +341,39 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211795985
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -80,6 +90,72 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211805275
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -597,6 +614,254 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211801549
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -250,33 +341,39 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211804454
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -597,6 +614,254 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211802489
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -288,7 +385,7 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211801254
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -250,33 +341,39 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211804879
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -597,6 +614,254 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211803267
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala
---
@@ -337,6 +338,7 @@ private[kafka010] case
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211805821
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -597,6 +614,254 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211805409
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -597,6 +614,254 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211804704
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -160,6 +160,23 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211803763
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
---
@@ -160,6 +160,23 @@ abstract class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r211805993
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
---
@@ -327,6 +332,14 @@ class KafkaTestUtils
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21469
I did. Fixed the import
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22175
Merged as compilation passed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/22175
jenkins retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/22175
[MINOR] Added import to fix compilation
## What changes were proposed in this pull request?
Two back to PRs implicitly conflicted by one PR removing an existing import
that the other PR
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21469
Unfortunately this PR broke the master build. Looks like some import that
probably got removed in the other PR I merged, which didnt create any direct
conflict
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21469
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21469
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21469
@HeartSaVioR I think I agree with a second approach that you suggested. So
`memoryUsedBytes` => `size for total memory usage of loaded versions` and
`customMetric` => `size for memory
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21733
Good point. That can be minor Pr.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21733
LGTM. Will merge when tests pass. :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21733
This looks good!! Only one comment, please don't add the .crc files. They
are useless and adds unnecessarily clutter
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r210422755
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -31,22 +31,21 @@ import
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r210423180
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -91,6 +90,17 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r210985375
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -346,11 +385,40 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r210422521
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -31,22 +31,21 @@ import
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209479417
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -251,32 +274,53 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209479551
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala
---
@@ -77,44 +77,6 @@ private[kafka010] class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209475048
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -251,32 +274,53 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209476712
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -347,9 +391,12 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209473392
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -31,6 +31,17 @@ import
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209477156
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -251,32 +274,53 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209476548
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -251,32 +274,53 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209478033
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -251,32 +274,53 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209473432
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -31,6 +31,17 @@ import
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209474755
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -251,32 +274,53 @@ private[kafka010] case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209473316
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -95,6 +106,10 @@ private[kafka010] case class
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21469
I am having a second thoughts about this. Exposing the entire memory usage
of all the loaded maps as another custom metric just adds more confusion.
Rather the point of the the main state metric
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208485230
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208483760
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208482355
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208491512
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208492158
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala
---
@@ -167,6 +165,18 @@ trait WatermarkSupport extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208488566
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208491168
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StreamingAggregationStateManagerSuite.scala
---
@@ -0,0 +1,132
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208487198
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208486837
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208496351
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala
---
@@ -201,33 +211,37 @@ object WatermarkSupport
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208490526
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StreamingAggregationStateManagerSuite.scala
---
@@ -0,0 +1,132
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208490973
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StreamingAggregationStateManagerSuite.scala
---
@@ -0,0 +1,132
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208485192
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208486899
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208491615
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208483242
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208481983
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208489469
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -871,6 +871,16 @@ object SQLConf {
.intConf
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208482479
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala
---
@@ -81,4 +85,221 @@ package object state
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r208492352
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala
---
@@ -167,6 +165,18 @@ trait WatermarkSupport extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21622#discussion_r206761192
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetricsReporter.scala
---
@@ -39,6 +42,23 @@ class MetricsReporter
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21469#discussion_r206739252
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
---
@@ -81,10 +81,10 @@ class SQLMetric(val metricType: String
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21469#discussion_r206738287
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala ---
@@ -48,12 +49,24 @@ class StateOperatorProgress private[sql](
def
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21469#discussion_r206738919
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala ---
@@ -48,12 +49,24 @@ class StateOperatorProgress private[sql](
def
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21357
While the new code looks a bit cleaner, I am bit hesistant to this complete
rewriting of the entire critical code. We generally do refactor of the code
only if there is some ultimate benefit, e.g
1 - 100 of 6589 matches
Mail list logo
