Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98777503
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/State.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98780383
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/objects.scala ---
@@ -144,6 +145,12 @@ object ObjectOperator {
(i: InternalRow
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98779599
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulAggregate.scala
---
@@ -235,3 +240,86 @@ case class StateStoreSaveExec
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98779015
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
---
@@ -90,6 +93,14 @@ class IncrementalExecution
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98775825
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -219,6 +219,160 @@ class KeyValueGroupedDataset[K, V] private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98780164
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulAggregate.scala
---
@@ -184,7 +189,7 @@ case class StateStoreSaveExec
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98778823
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/StateImpl.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98779264
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulAggregate.scala
---
@@ -54,6 +55,18 @@ trait StatefulOperator extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98777585
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/State.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98779142
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulAggregate.scala
---
@@ -54,6 +55,18 @@ trait StatefulOperator extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98777860
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/State.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98778773
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/StateImpl.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98776628
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -219,6 +219,160 @@ class KeyValueGroupedDataset[K, V] private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98777095
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -219,6 +219,160 @@ class KeyValueGroupedDataset[K, V] private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98776799
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -219,6 +219,160 @@ class KeyValueGroupedDataset[K, V] private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98776538
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -219,6 +219,160 @@ class KeyValueGroupedDataset[K, V] private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98778649
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/StateImpl.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98777806
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/State.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16758#discussion_r98777964
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/State.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16745
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16329
Sorry for the delay. This LGTM, but I'm currently away from my Apache SSH
keys. Other committers should feel free to merge if you get there before I do.
---
If your project is set up
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16664
/cc @liancheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16564
Hmm, I'm not sure that I agree with the solution from #15427. I do not
think that it should be valid to have to different expressions that have the
same expression id. There are many case where
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16553#discussion_r95655634
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -488,219 +488,241 @@ class UDFRegistration private[sql
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16240#discussion_r94888245
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLImplicits.scala
---
@@ -99,33 +96,96 @@ abstract class SQLImplicits {
// Seqs
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16240
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16240
For future reference: https://github.com/apache/spark/blob/master/dev/mima
(script to run mima)
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16371
+1 I think we can move forward.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16322
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16360#discussion_r93508333
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
---
@@ -219,7 +221,13 @@ final class DataStreamWriter[T] private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16360#discussion_r93507397
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalOutputModes.scala
---
@@ -15,7 +15,7 @@
* limitations under the License
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16360#discussion_r93509028
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
---
@@ -219,7 +221,13 @@ final class DataStreamWriter[T] private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16304#discussion_r93355210
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -387,7 +387,7 @@ class StreamExecution
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16322#discussion_r93121938
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -206,6 +201,36 @@ class StreamExecution
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16329
This is great! Thanks for taking the time to write up such complete
examples. I think this was a big gap in the existing docs. One other ask.
The screen-shot is great, but I'd like to see
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16329#discussion_r93119785
--- Diff:
examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedTypedAggregation.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16329#discussion_r93121147
--- Diff:
examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedTypedAggregation.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16329#discussion_r93118975
--- Diff:
examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedTypedAggregation.java
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16329#discussion_r93118905
--- Diff:
examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedTypedAggregation.java
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16240
/cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16322#discussion_r93113059
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -206,6 +201,36 @@ class StreamExecution
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16304
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16304#discussion_r92904102
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala
---
@@ -124,6 +137,29 @@ class WatermarkSuite extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16304#discussion_r92902755
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala
---
@@ -124,6 +137,29 @@ class WatermarkSuite extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16304#discussion_r92753969
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -572,6 +572,10 @@ class Dataset[T] private[sql](
val parsedDelay
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16304#discussion_r92753590
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -572,6 +572,10 @@ class Dataset[T] private[sql](
val parsedDelay
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16289
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16258
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16258#discussion_r92077840
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala ---
@@ -33,27 +34,6 @@ import
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16258#discussion_r92073271
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryStatusAndProgressSuite.scala
---
@@ -38,13 +38,18 @@ class
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16258#discussion_r92071196
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -360,6 +360,24 @@ class StreamExecution
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16238#discussion_r91803533
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/util/DefaultSource.scala
---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16238
This LGTM, I was just talking with @tdas about how I think that all of this
initialization stuff should be lazy and happen on the stream execution thread.
I think this can simplify what
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16238#discussion_r91803089
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/util/DefaultSource.scala
---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16182
/cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/16182
[SPARK-18754][SS] Rename recentProgresses to recentProgress
Based on an informal survey, users find this option easier to understand /
remember.
You can merge this pull request into a Git
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16178#discussion_r91190778
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1760,25 +1760,24 @@ class SparkContext(config: SparkConf) extends
Logging
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16138
This will be very convenient! Looking forward to the whole patch. For SQL
I think you should look at
[`RuntimeReplaceable`](https://github.com/apache/spark/blob
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16113#discussion_r90741307
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -605,34 +629,64 @@ class StreamExecution
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16113#discussion_r90741239
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala ---
@@ -32,21 +32,33 @@ import org.apache.spark.sql.SparkSession
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/16113#discussion_r90741067
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
---
@@ -54,6 +61,26 @@ object OffsetSeq {
* `nulls
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15918
I don't think you can limit the implicit. What type would pick up case
classes, but not case classes that contain invalid things? I think you would
need a macros for this kind of introspection
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16094
No worries, thanks for working on this! It's great to ensure our Python
APIs aren't lagging behind the Scala ones.
---
If your project is set up for it, you can reply to this email and have your
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13147#discussion_r90560927
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
---
@@ -27,6 +27,14 @@ import
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/16094
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15918
We should probably add a flag (maybe even off by default). The error
message can tell you to turn on the flag if you are okay with the fallback.
---
If your project is set up for it, you can
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15918
I agree that the only change in behavior is that things that used to throw
an error will now not throw an error. If done right (I haven't looked deeply
at the PR itself yet), no case
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13147#discussion_r90317826
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/QueryPlanner.scala
---
@@ -27,6 +27,14 @@ import
Repository: spark
Updated Branches:
refs/heads/master 9a02f6821 -> c3d08e2f2
http://git-wip-us.apache.org/repos/asf/spark/blob/c3d08e2f/sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala
--
diff --git
Repository: spark
Updated Branches:
refs/heads/branch-2.1 045ae299c -> 28b57c8a1
http://git-wip-us.apache.org/repos/asf/spark/blob/28b57c8a/sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala
--
diff --git
"3" : 0,
"0" : 1
}
},
"numRecords" : 3,
"inputRowsPerSecond" : 230.76923076923077,
"processedRowsPerSecond" : 10.869565217391303
} ]
}
```
Additionally, in order to make it possible to correlate progress update
"3" : 0,
"0" : 1
}
},
"numRecords" : 3,
"inputRowsPerSecond" : 230.76923076923077,
"processedRowsPerSecond" : 10.869565217391303
} ]
}
```
Additionally, in order to make it possible to correlate progress updates across
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15954
LGTM, merging to master and 2.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90133572
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala
---
@@ -59,13 +62,20 @@ class StreamingQueryManager private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90133503
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala ---
@@ -64,23 +68,26 @@ trait StreamingQuery
cks without
worrying about batch file name formats. marmbrus zsxwing
Existing tests already ensure this API faithfully support core functionality
i.e., creation of batch files.
Author: Tyson Condie <tcon...@gmail.com>
Closes #15924 from tcondie/SPARK-18498.
Signed-off-by: Michael Armbr
cks without
worrying about batch file name formats. marmbrus zsxwing
Existing tests already ensure this API faithfully support core functionality
i.e., creation of batch files.
Author: Tyson Condie <tcon...@gmail.com>
Closes #15924 from tcondie/SPARK-18498.
Signed-off-by: Michael Armbr
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15924#discussion_r90090753
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
---
@@ -129,48 +129,18 @@ class HDFSMetadataLog[T
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90083413
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala ---
@@ -38,11 +40,11 @@ trait StreamingQuery {
def name
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90082045
--- Diff: python/pyspark/sql/streaming.py ---
@@ -87,6 +88,24 @@ def awaitTermination(self, timeout=None):
else:
return self
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90084842
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala
---
@@ -279,3 +287,8 @@ class StreamingQueryManager private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90085518
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryProgress.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90084420
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryListener.scala
---
@@ -81,30 +83,30 @@ object StreamingQueryListener
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90085377
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryProgress.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90084970
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryProgress.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90083627
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala ---
@@ -51,7 +53,7 @@ trait StreamingQuery {
def sparkSession
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90086100
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala ---
@@ -669,55 +658,48 @@ trait StreamTest extends QueryTest
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r90084872
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala
---
@@ -279,3 +287,8 @@ class StreamingQueryManager private
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r89035938
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryProgress.scala
---
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r89032237
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala ---
@@ -64,23 +66,26 @@ trait StreamingQuery
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r89032104
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/SourceProgress.scala ---
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r89032039
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryProgress.scala
---
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/15954#discussion_r89031940
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/SourceProgress.scala ---
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15962
Do you have any performance comparisons to show that we need to do this?
On the driver, I think we want it to be as small as possible, because we
don't want to pull any data down (it would
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15954
/cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/15954
[WIP][SPARK-18516][SQL] Split state and progress in streaming
This PR separates the status of a `StreamingQuery` into two separate APIs:
- `status` - describes the status of a `StreamingQuery
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15933
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15934
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15921
No, I don't think we need to throw any exceptions. Watermarks are defined
at batch boundaries, so it would just have no affect for a batch job.
We should make sure that the batch planner
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15908
Oh sorry... I'm realizing I'm commenting on a PR for branch-2.0. We don't
have to address these comments here, but we should make sure we are happy with
all the naming before 2.1 is released
301 - 400 of 8258 matches
Mail list logo
