Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2065
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53546318
Alright, tested this. Merging it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53542841
This looks real good now. Thanks @harishreedharan for all changes and the
wonderful refactoring. I am going to quickly test this in my local flume set up
for double confirma
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53537410
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19298/consoleFull)
for PR 2065 at commit
[`f93a07c`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53533488
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19298/consoleFull)
for PR 2065 at commit
[`f93a07c`](https://github.com/ap
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53530019
This now looks quite good to me. There were a few more formatting issues,
should take 5 minutes to solve :)
---
If your project is set up for it, you can reply to this ema
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16758820
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache So
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16758798
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache So
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16758766
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache So
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16758756
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala
---
@@ -225,4 +237,16 @@ private class Transaction
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16758729
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -63,18 +68,34 @@ private[flume] class S
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16758724
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -45,7 +47,8 @@ private[flume] class Spa
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16758698
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -63,18 +68,34 @@ private[flume] class S
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53527293
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19283/consoleFull)
for PR 2065 at commit
[`d7427cc`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53526713
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19280/consoleFull)
for PR 2065 at commit
[`a0a8852`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53524905
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19283/consoleFull)
for PR 2065 at commit
[`d7427cc`](https://github.com/ap
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756511
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to th
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53524470
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19280/consoleFull)
for PR 2065 at commit
[`a0a8852`](https://github.com/ap
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756449
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -88,59 +87,7 @@ private[streaming] class FlumePol
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756411
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache So
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756404
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache So
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756399
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache So
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756389
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFetcher.scala
---
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache So
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756336
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +67,30 @@ private[flume] class S
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756314
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -45,7 +47,8 @@ private[flume] class Spa
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756273
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +67,30 @@ private[flume] class S
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756261
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +67,30 @@ private[flume] class S
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756253
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +67,30 @@ private[flu
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16756236
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +67,30 @@ private[flume] class S
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16755731
--- Diff:
yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -28,6 +28,7 @@ import org.apache.hadoop.yarn.conf.YarnConfigurat
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16755614
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -124,6 +155,13 @@ private[fl
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16755513
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16755514
--- Diff:
yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -28,6 +28,7 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration
im
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754872
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754609
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754580
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754560
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754551
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754471
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754413
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754361
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingRunnable.scala
---
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754182
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +74,30 @@ private[flume] class S
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16754092
--- Diff: external/flume-sink/pom.xml ---
@@ -71,6 +71,10 @@
test
+ org.scala-lang
--- End diff --
The thing
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16753967
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -124,6 +155,13 @@ private[flume] class
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53515670
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19250/consoleFull)
for PR 2065 at commit
[`445e700`](https://github.com/a
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16752978
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +74,30 @@ private[flume] class S
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16752958
--- Diff: external/flume-sink/pom.xml ---
@@ -71,6 +71,10 @@
test
+ org.scala-lang
--- End diff --
Yes, so
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16752853
--- Diff: external/flume-sink/pom.xml ---
@@ -71,6 +71,10 @@
test
+ org.scala-lang
--- End diff --
@srowen Yo
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-53512084
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19250/consoleFull)
for PR 2065 at commit
[`445e700`](https://github.com/ap
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16728148
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +70,24 @@ private[flume] class S
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16727964
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +70,24 @@ private[flu
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16687067
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] class FlumePo
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16685628
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] class FlumePo
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16685565
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] class FlumePo
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16685491
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] class FlumePo
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16683745
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] cl
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16683738
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] cl
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16683530
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -124,6 +139,8 @@ private[flu
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16683523
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] class FlumePo
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16683481
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +93,77 @@ private[streaming] class FlumePo
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16683031
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -124,6 +139,8 @@ private[flume] class S
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16682810
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,19 +70,24 @@ private[flume] class S
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16682789
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -46,6 +48,10 @@ private[flume] class Sp
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52973226
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19064/consoleFull)
for PR 2065 at commit
[`9001d26`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52969414
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19063/consoleFull)
for PR 2065 at commit
[`e7b8d82`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52966251
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19064/consoleFull)
for PR 2065 at commit
[`9001d26`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52962158
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19063/consoleFull)
for PR 2065 at commit
[`e7b8d82`](https://github.com/ap
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16553665
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -144,6 +168,21 @@ private[streaming] c
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52943823
Mostly nits left; looks OK after you look at the `EventBatch` issue I
pointed out.
---
If your project is set up for it, you can reply to this email and have your
reply a
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16549316
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -144,6 +168,21 @@ private[streaming] class Flum
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16549291
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -144,6 +168,21 @@ private[streaming] class Flum
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16549232
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -18,23 +18,27 @@ package org.apache.spark.strea
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16549194
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -18,23 +18,27 @@ package org.apache.spark.strea
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16549128
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala
---
@@ -88,6 +90,11 @@ private class Transaction
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16549093
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,12 +70,16 @@ private[flume] class
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52873308
Some errors (like the ChannelClosedException) getting logged on the sink
side are inevitable. This is because Avro throws that when the other side
closes the conn
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16518442
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala
---
@@ -115,7 +122,7 @@ private class T
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52866621
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19020/consoleFull)
for PR 2065 at commit
[`598efa7`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52862309
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19020/consoleFull)
for PR 2065 at commit
[`598efa7`](https://github.com/ap
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16512127
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +95,76 @@ private[streaming] cl
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16512056
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -144,6 +172,23 @@ private[streaming] class Flum
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16512028
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -90,53 +95,76 @@ private[streaming] class Flume
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16511761
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala
---
@@ -47,11 +52,11 @@ import org.apache.spark.stream
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16511729
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala
---
@@ -115,7 +122,7 @@ private class Transactio
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2065#discussion_r16511500
--- Diff:
external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
---
@@ -62,12 +70,16 @@ private[flume] class
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52839517
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18984/consoleFull)
for PR 2065 at commit
[`ed608c8`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2065#issuecomment-52832628
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18984/consoleFull)
for PR 2065 at commit
[`ed608c8`](https://github.com/ap
GitHub user harishreedharan opened a pull request:
https://github.com/apache/spark/pull/2065
[SPARK-3154][STREAMING] Make FlumePollingInputDStream shutdown cleaner.
Currently lot of errors get thrown from Avro IPC layer when the dstream
or sink is shutdown. This PR cleans it up.
88 matches
Mail list logo
