[GitHub] spark issue #14981: [SPARK-17418] Remove Kinesis artifacts from Spark releas...

2016-09-09 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14981 Isn't that mostly down to whether someone wants to just put the whole assembly on their classpath, vs install the project and depend on it in their build tool? I can see why someone would

[GitHub] spark issue #14981: [SPARK-17418] Remove Kinesis artifacts from Spark releas...

2016-09-09 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14981 My 2 cents are that we should make things as easy as possible for users, within the bounds of what ASF legal is willing to tolerate ;) Which probably means having it exist, but not published to

[GitHub] spark issue #14606: [SPARK-17023][BUILD] Upgrade to Kafka 0.10.0.1 release

2016-08-12 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14606 Did you actually test this against a production sized workload? As Sean said, probably fine, but i'm curious what bugs you think are especially relevant to the spark connector. -

[GitHub] spark pull request #14502: [SPARK-16909][Spark Core] - Streaming for postgre...

2016-08-05 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14502#discussion_r73713830 --- Diff: core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala --- @@ -79,12 +79,18 @@ class JdbcRDD[T: ClassTag]( val conn = getConnection

[GitHub] spark pull request #14502: [SPARK-16909][Spark Core] - Streaming for postgre...

2016-08-05 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14502#discussion_r73681837 --- Diff: core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala --- @@ -79,12 +79,18 @@ class JdbcRDD[T: ClassTag]( val conn = getConnection

[GitHub] spark issue #14385: [SPARK-16312][STREAMING][KAFKA][DOC] Doc for Kafka 0.10 ...

2016-08-01 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14385 @tdas @srowen @zsxwing Anyone have time to look at this? Because of the issues building scaladoc for the 0.10 integration, it seems like it'd be good to have at least some kind of document

[GitHub] spark issue #14279: [SPARK-16216][SQL] Write Timestamp and Date in ISO 8601 ...

2016-08-01 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14279 +1 to default formats that are unambiguous regarding timezone --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...

2016-07-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14416 LGTM, minor comments about map notwithstanding. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #14416: [SPARK-16776][STREAMING] Replace deprecated API i...

2016-07-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14416#discussion_r72888085 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala --- @@ -178,8 +178,9 @@ private[kafka010] class

[GitHub] spark issue #14343: [SPARK-16705] Kafka Direct Stream is not experimental an...

2016-07-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14343 I've got a separate PR for 0.10 docs that also fixes all the experimental mentions I could find, SPARK-16312 https://github.com/apache/spark/pull/14385 On Jul 29, 2016 6:42 AM,

[GitHub] spark pull request #14385: [SPARK-16312][STREAMING][KAFKA][DOC] Doc for Kafk...

2016-07-27 Thread koeninger
GitHub user koeninger opened a pull request: https://github.com/apache/spark/pull/14385 [SPARK-16312][STREAMING][KAFKA][DOC] Doc for Kafka 0.10 integration ## What changes were proposed in this pull request? Doc for the Kafka 0.10 integration ## How was this patch

[GitHub] spark issue #14372: Make offsetRanges variable volatile

2016-07-26 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14372 I generally try to write my code the way Sean suggested (single foreachRDD) whenever possible. Sounds like if the transformation isn't guaranteed to run before the foreachRDD lat

[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...

2016-07-26 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14340 @jerryshao agreed that this is probably as good as possible for the Python api and the limitations make sense Generally LGTM, the only remaining comments I had were minor wording things

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-26 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72251359 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -286,6 +285,30 @@ private[kafka010] class

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-26 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72248712 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -221,22 +222,20 @@ private[kafka010] class

[GitHub] spark issue #14361: [TEST][STREAMING] Fix flaky Kafka rate controlling test

2016-07-26 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14361 - This is testing RateEstimator, not maxRatePerPartition. I didn't write the rate estimator code, but my understanding of the rate expressed there is that it is on a per-stream basis, not

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72177258 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -177,3 +182,172 @@ object KafkaUtils extends

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72123421 --- Diff: python/pyspark/streaming/kafka010.py --- @@ -0,0 +1,370 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...

2016-07-25 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14340 In terms of feature parity, are we OK with telling python users they are out of luck if they need something not covered by Assign / Subscribe / SubscribePattern? For scala / java users they can

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72122440 --- Diff: python/pyspark/streaming/kafka010.py --- @@ -0,0 +1,370 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72122274 --- Diff: python/pyspark/streaming/kafka010.py --- @@ -0,0 +1,370 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72121957 --- Diff: python/pyspark/streaming/kafka010.py --- @@ -0,0 +1,370 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72120535 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -177,3 +182,172 @@ object KafkaUtils extends

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72118143 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala --- @@ -177,3 +182,172 @@ object KafkaUtils extends

[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...

2016-07-25 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14340#discussion_r72115899 --- Diff: examples/src/main/python/streaming/direct_kafka010_wordcount.py --- @@ -0,0 +1,59 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] spark issue #14343: [SPARK-16705] Kafka Direct Stream is not experimental an...

2016-07-25 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14343 Agreed that only removing the word experimental should be necessary. I thought there were more mentions, but can double check. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-08 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14026#discussion_r70158520 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -79,8 +81,71 @@ private case class

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-06 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14026#discussion_r69830774 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -79,8 +81,71 @@ private case class

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-06 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14026#discussion_r69829237 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -79,8 +81,71 @@ private case class

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-06 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14026#discussion_r69829009 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -79,8 +81,71 @@ private case class

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-06 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14026#discussion_r69828832 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -79,8 +81,71 @@ private case class

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-06 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14026#discussion_r69828549 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -79,8 +81,71 @@ private case class

[GitHub] spark issue #14073: [SPARK-16212][STREAMING][KAFKA] apply test tweaks from 0...

2016-07-06 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14073 @srowen regarding your message on the mailing list about seeing DirectKafkaStreamSuite failures, can you see if this is more reliable for you? --- If your project is set up for it, you can reply

[GitHub] spark pull request #14073: [SPARK-16212][STREAMING][KAFKA] apply test tweaks...

2016-07-06 Thread koeninger
GitHub user koeninger opened a pull request: https://github.com/apache/spark/pull/14073 [SPARK-16212][STREAMING][KAFKA] apply test tweaks from 0-10 to 0-8 as well ## What changes were proposed in this pull request? Bring the kafka-0-8 subproject up to date with some test

[GitHub] spark issue #14041: [SPARK-16359][STREAMING][KAFKA] unidoc skip kafka 0.10

2016-07-05 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14041 From this + looking at jenkins, it seems like master is broken --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14041: [SPARK-16359][STREAMING][KAFKA] unidoc skip kafka 0.10

2016-07-05 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14041 That build error looks hive related... I can try merging latest master @tdas yes, the reason unidoc is failing is because it's throwing all dependencies from all subprojects int

[GitHub] spark issue #14026: [SPARK-13569][STREAMING][KAFKA] pattern based topic subs...

2016-07-05 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14026 ping @tdas @zsxwing these poll fixes need to go in for a release / release candidate, even if SubscribePattern doesn't make it for some reason. --- If your project is set up for it, yo

[GitHub] spark issue #14041: [SPARK-16359][STREAMING][KAFKA] unidoc skip kafka 0.10

2016-07-04 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14041 Keeping the 0.10 classes might work if we want to skip publishing 0.8, but trying to skip publishing 0.10 did not work until I modified the classpath for the unidoc task. 0.8 would error

[GitHub] spark pull request #14041: [SPARK-16359][STREAMING][KAFKA] unidoc skip kafka...

2016-07-03 Thread koeninger
GitHub user koeninger opened a pull request: https://github.com/apache/spark/pull/14041 [SPARK-16359][STREAMING][KAFKA] unidoc skip kafka 0.10 ## What changes were proposed in this pull request? during sbt unidoc task, skip the streamingKafka010 subproject and filter kafka 0.10

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-02 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/14026#discussion_r69383724 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -79,8 +81,71 @@ private case class

[GitHub] spark issue #14026: [SPARK-13569][STREAMING][KAFKA] pattern based topic subs...

2016-07-01 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14026 @tdas @zsxwing This should be the last ConsumerStrategy implementation to have basic parity with what's offered by the kafka consumer, anything else should probably be handled by user subcl

[GitHub] spark pull request #14026: [SPARK-13569][STREAMING][KAFKA] pattern based top...

2016-07-01 Thread koeninger
GitHub user koeninger opened a pull request: https://github.com/apache/spark/pull/14026 [SPARK-13569][STREAMING][KAFKA] pattern based topic subscription ## What changes were proposed in this pull request? Allow for kafka topic subscriptions based on a regex pattern

[GitHub] spark issue #14018: [SPARK-16212][STREAMING][KAFKA] use random port for embe...

2016-07-01 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14018 @tdas sorry, this was one more thing that came out of testing on 0.10 that needs to be done in 0.8 as well. Let me know if you think a different jira ticket is needed. --- If your project is

[GitHub] spark pull request #14018: [SPARK-16212][STREAMING][KAFKA] use random port f...

2016-07-01 Thread koeninger
GitHub user koeninger opened a pull request: https://github.com/apache/spark/pull/14018 [SPARK-16212][STREAMING][KAFKA] use random port for embedded kafka ## What changes were proposed in this pull request? Testing for 0.10 uncovered an issue with a fixed port number being

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 Added the doc links and my best guess on that first test failure. I think the second is going to take more work to figure out. Your call on whether you want to wait and see if I find something

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 Are those failures repeatable for those particular hadoop versions, or is this an intermittent thing? May need to look at redoing how kafkatestutils is working, since it's intera

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 @zsxwing made a tweak in this PR that may affect that first failure. LMK if we should do a separate pull just for that. Looking at the second one. --- If your project is set up for it

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 @zsxwing looking --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 Sure... the most trivial example of how that would break at runtime is that I already created an implementation that defines that method, with the opposite semantics. Is that particularly likely

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 Basically, if you're adding a new method to an interface, it's because you need new behavior. This is the fragile base class problem inherent in, well, inheriting behavior

[GitHub] spark issue #13998: [SPARK-12177][Streaming][Kafka] limit api surface area

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13998 Copying from the other PR about the trait vs abstract class and adding methods after people have already implented or subclassed: You said "When abstract class, you can later add de

[GitHub] spark pull request #13998: [SPARK-12177][Streaming][Kafka] limit api surface...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13998#discussion_r69202911 --- Diff: external/kafka-0-10/src/test/java/org/apache/spark/streaming/kafka010/JavaConsumerStrategySuite.java --- @@ -44,37 +44,39 @@ public void

[GitHub] spark pull request #13998: [SPARK-12177][Streaming][Kafka] limit api surface...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13998#discussion_r69196487 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -68,8 +67,7 @@ trait ConsumerStrategy[K, V

[GitHub] spark pull request #13998: [SPARK-12177][Streaming][Kafka] limit api surface...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13998#discussion_r69182103 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala --- @@ -68,8 +67,7 @@ trait ConsumerStrategy[K, V

[GitHub] spark issue #13996: [SPARK-12177] Refactored some API in Kafka 0.10 to make ...

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13996 1. "When abstract class, you can later add defined methods which the users can override if needed, but does not break compatibility of existing implementations." This is what I'

[GitHub] spark pull request #13996: [SPARK-12177] Refactored some API in Kafka 0.10 t...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13996#discussion_r69179469 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/MapConverter.scala --- @@ -0,0 +1,30 @@ +/* + * Licensed to the

[GitHub] spark issue #13996: [SPARK-12177] Refactored some API in Kafka 0.10 to make ...

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13996 See the linked pr https://github.com/apache/spark/pull/13998 for an example of what I'm proposing. I'll make sure the same thing works for consumer strategy and update that pr.

[GitHub] spark pull request #13998: [SPARK-12177][Streaming][Kafka] limit api surface...

2016-06-30 Thread koeninger
GitHub user koeninger opened a pull request: https://github.com/apache/spark/pull/13998 [SPARK-12177][Streaming][Kafka] limit api surface area ## What changes were proposed in this pull request? This is an alternative to the refactoring proposed by https://github.com/apache

[GitHub] spark issue #13996: [SPARK-12177] Refactored some API in Kafka 0.10 to make ...

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13996 So another concrete reason I'm against moving the strategy interfaces to java is that LocationStrategy is no longer sealed, so we lose exhaustivity checking on the

[GitHub] spark issue #13908: [SPARK-16212][STREAMING][KAFKA] code cleanup from review...

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13908 I don't plan to make any more changes... I kind of doubt feedback on your https://github.com/apache/spark/pull/13996 is going to result in any changes that would also apply to 0.8 --- If

[GitHub] spark pull request #13996: [SPARK-12177] Refactored some API in Kafka 0.10 t...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13996#discussion_r69156162 --- Diff: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala --- @@ -547,20 +559,20 @@ class

[GitHub] spark pull request #13996: [SPARK-12177] Refactored some API in Kafka 0.10 t...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13996#discussion_r69155792 --- Diff: external/kafka-0-10/src/test/java/org/apache/spark/streaming/kafka010/JavaKafkaRDDSuite.java --- @@ -96,14 +96,14 @@ public String call

[GitHub] spark pull request #13996: [SPARK-12177] Refactored some API in Kafka 0.10 t...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13996#discussion_r69154443 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/MapConverter.scala --- @@ -0,0 +1,30 @@ +/* + * Licensed to the

[GitHub] spark pull request #13996: [SPARK-12177] Refactored some API in Kafka 0.10 t...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13996#discussion_r69153657 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala --- @@ -71,7 +64,7 @@ private[spark] class

[GitHub] spark pull request #13996: [SPARK-12177] Refactored some API in Kafka 0.10 t...

2016-06-30 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13996#discussion_r69153231 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala --- @@ -44,13 +42,8 @@ import

[GitHub] spark issue #13996: [SPARK-12177] Refactored some API in Kafka 0.10 to make ...

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13996 I'll leave line comments about specific things, but my major overarching concern is about moving the interface to a Java abstract class. As far as I can tell, nothing being done in

[GitHub] spark issue #13996: [SPARK-12177] Refactored some API in Kafka 0.10 to make ...

2016-06-30 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/13996 I've got concerns about this, please don't merge these refactorings until I get a chance to look at it today On Jun 30, 2016 6:32 AM, "Tathagata Das" wrote:

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 I'll do the scaladoc fix and the package rename. I think the package rename is fine even if it did work with docs, just to disambiguate things. Will start a separate ticke

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 You do need CanCommitOffsets because DirectKafkaInputDstream is now private, so otherwise you have nothing to cast to to access that method. On Jun 29, 2016 7:11 PM, "Tathagata Das&qu

[GitHub] spark issue #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStreams to ...

2016-06-29 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/11863 @zsxwing Thanks for the fixes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #11863: [SPARK-12177][Streaming][Kafka] Update KafkaDStre...

2016-06-29 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/11863#discussion_r69035388 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -0,0 +1,321

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68851832 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68848509 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68848341 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68848046 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68847747 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68846951 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68846804 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68845808 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68846168 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68845448 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68843786 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68844514 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68844121 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68844003 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68843447 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68843392 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68842976 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68842804 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68842633 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68842326 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68841970 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68841644 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68841415 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68840717 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68840643 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68840069 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68839964 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

[GitHub] spark pull request #13945: [SPARK-16256][SQL][STREAMING] Added Structured St...

2016-06-28 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/13945#discussion_r68839851 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -0,0 +1,888 @@ +--- +layout: global +displayTitle: Structured Streaming

<    1   2   3   4   5   6   7   >