Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19431
@koeninger done!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19431
@koeninger resolved the conflict
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r173719300
--- Diff:
external/kafka-0-8/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala
---
@@ -456,6 +455,60 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167395493
--- Diff:
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
---
@@ -22,6 +22,7 @@ import java.lang
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167395482
--- Diff:
external/kafka-0-8/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala
---
@@ -539,6 +456,58 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167395492
--- Diff:
external/kafka-0-8/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala
---
@@ -21,6 +21,7 @@ import java.io.File
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167395487
--- Diff:
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
---
@@ -687,6 +618,51 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167242320
--- Diff:
external/kafka-0-8/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala
---
@@ -387,6 +387,89 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167242353
--- Diff:
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
---
@@ -551,6 +551,76 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167242329
--- Diff:
external/kafka-0-8/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala
---
@@ -387,6 +387,89 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r167242186
--- Diff:
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
---
@@ -551,6 +551,76 @@ class
Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19431
> Related the doc I thought it's kafka specific but it's not so fine like
that
Yes, it was implemented only in Kafka Streams but doc doesnt limit usage of
this parameter to Ka
Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19431
Latest rate means rate of previous batch. Is it possible that in alive
system 0 events were processed? Only if there is no backlog and no new events
came during last batch. Completely possible
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r166606906
--- Diff:
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
---
@@ -126,7 +129,10 @@ private[spark
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r166605640
--- Diff:
external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala
---
@@ -91,9 +91,16 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r166605671
--- Diff:
external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala
---
@@ -108,7 +115,9 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r166605578
--- Diff:
external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala
---
@@ -91,9 +91,16 @@ class
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r166605547
--- Diff:
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
---
@@ -126,7 +129,10 @@ private[spark
Github user akonopko commented on a diff in the pull request:
https://github.com/apache/spark/pull/19431#discussion_r166603416
--- Diff:
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
---
@@ -126,7 +129,10 @@ private[spark
Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19431
@gaborgsomogyi
`spark.streaming.backpressure.initialRate` is already documented in here:
https://spark.apache.org/docs/latest/configuration.html
But was mistakenly not included
Github user akonopko commented on the issue:
https://github.com/apache/spark/pull/19430
@vanzin plz take a look at https://github.com/apache/spark/pull/19431
I am deleting this PR cause it was mistakenly created against branch-2.2
Github user akonopko closed the pull request at:
https://github.com/apache/spark/pull/19430
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
GitHub user akonopko opened a pull request:
https://github.com/apache/spark/pull/19431
Add spark.streaming.backpressure.initialRate to direct Kafka streams
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/akonopko/spark SPARK
GitHub user akonopko reopened a pull request:
https://github.com/apache/spark/pull/19430
Spark 18580
## What changes were proposed in this pull request?
Add `spark.streaming.backpressure.initialRate` to direct Kafka Streams for
Kafka 0.8 and 0.10
This is required
Github user akonopko closed the pull request at:
https://github.com/apache/spark/pull/19430
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
GitHub user akonopko opened a pull request:
https://github.com/apache/spark/pull/19430
Spark 18580
## What changes were proposed in this pull request?
Add `spark.streaming.backpressure.initialRate` to direct Kafka Streams for
Kafka 0.8 and 0.10
This is required
26 matches
Mail list logo
