Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20096
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160774506
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochCoordinator.scala
---
@@ -39,6 +39,15 @@ private[continuous]
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160640668
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochCoordinator.scala
---
@@ -39,6 +39,15 @@ private[continuous]
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160570055
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -49,28 +52,37 @@ abstract class
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160569877
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -237,85 +378,67 @@ class KafkaSourceSuite
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160569556
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -977,20 +971,8 @@ class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160552554
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,135 @@
+/*
--- End diff --
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160550486
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -237,85 +378,67 @@ class KafkaSourceSuite
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160546427
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160559942
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSinkSuite.scala
---
@@ -0,0 +1,348 @@
+/*
+ * Licensed
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160546593
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160546455
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160550392
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -237,85 +378,67 @@ class KafkaSourceSuite
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160549136
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -49,28 +52,37 @@ abstract class KafkaSourceTest
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160516280
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160552160
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
---
@@ -977,20 +971,8 @@ class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160552695
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160516177
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160009573
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceOffset.scala
---
@@ -19,7 +19,8 @@ package
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160004815
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousReader.scala
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160007884
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -418,11 +418,16 @@ abstract class StreamExecution(
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160006676
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousWriter.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160005666
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaWriteTask.scala
---
@@ -74,8 +56,49 @@ private[kafka010] class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159985929
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
---
@@ -145,6 +149,19 @@ private[kafka010] class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r160008897
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala
---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159984700
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousReader.scala
---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159983463
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala
---
@@ -261,6 +261,10 @@ private[kafka010] case
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159797164
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -418,11 +418,16 @@ abstract class
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159796525
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSinkV2.scala
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159796099
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159795887
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159795747
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159795735
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159569390
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159569305
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159569599
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159568911
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159728249
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSinkV2.scala
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159568750
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159568798
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159569925
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159571068
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSinkV2.scala
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159566309
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159570879
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159568788
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159568756
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159503021
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159571114
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159566461
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159569751
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159731324
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -181,26 +223,22 @@ private[kafka010] class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159733170
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/ContinuousKafkaSuite.scala
---
@@ -0,0 +1,264 @@
+/*
+ * Licensed to
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159504846
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceOffset.scala
---
@@ -31,10 +32,14 @@ case class
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/20096#discussion_r159563923
--- Diff:
external/kafka-0-10-sql/src/main/scala/ContinuousKafkaReader.scala ---
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation
GitHub user jose-torres opened a pull request:
https://github.com/apache/spark/pull/20096
[SPARK-22908] Add kafka source and sink for continuous processing.
## What changes were proposed in this pull request?
Add kafka source and sink for continuous processing. This
55 matches
Mail list logo
