[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196574750 OK apparently the recent mima failures on the master branch is caused by the mqtt dependency not being in maven. We can either fix that with some build hack, or just merge this pr. Since these are the least contentious choices, I'm going to merge this pull request. We should discuss kafka/kinesis separately. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196574785 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196555837 Thanks @rxin. Opened [SPARK-11806](https://issues.apache.org/jira/browse/SPARK-11806) to discuss this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196553861 Can we have a JIRA ticket to discuss that? I am not sure this github pr is the place to discuss moving kafka out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196553283 Hi @zsxwing, I think the discussion on supporting Kafka 0.9 should happen **if** we decide to keep Kafka in Spark itself. At this point, I think the piece that benefits the most out of moving out of Spark is the kafka integration - since that is the one where more of the API compatibility issues are. I really think we should discuss moving Kafka out and come to an agreement on that as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196539603 @JoshRosen could you take a look at this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196537867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53095/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196537861 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196537552 **[Test build #53095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53095/consoleFull)** for PR 11672 at commit [`806dba8`](https://github.com/apache/spark/commit/806dba859d03ca0db92991f6e5baf27aa15f7322). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class ShuffleServiceHeartbeat extends BlockTransferMessage ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196489486 **[Test build #53095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53095/consoleFull)** for PR 11672 at commit [`806dba8`](https://github.com/apache/spark/commit/806dba859d03ca0db92991f6e5baf27aa15f7322). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196487964 Merge conflict is probably my fault (I modified a file that this PR deleted). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196485585 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196485589 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53078/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196485239 **[Test build #53078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53078/consoleFull)** for PR 11672 at commit [`01dadc5`](https://github.com/apache/spark/commit/01dadc568240cc688163aeb7048510079b083d54). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196412519 **[Test build #53078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53078/consoleFull)** for PR 11672 at commit [`01dadc5`](https://github.com/apache/spark/commit/01dadc568240cc688163aeb7048510079b083d54). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196175509 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196175510 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53047/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196175359 **[Test build #53047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53047/consoleFull)** for PR 11672 at commit [`97fcb46`](https://github.com/apache/spark/commit/97fcb46152bb8a02e861a7a91443781f4cc3b595). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196141331 **[Test build #53047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53047/consoleFull)** for PR 11672 at commit [`97fcb46`](https://github.com/apache/spark/commit/97fcb46152bb8a02e861a7a91443781f4cc3b595). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196115717 I agree with @ksakellis on this one. It would be great if we can pull Kafka out as well. I understand that there are a lot of users who might find it difficult, but if you think about it, most people use the plugins via mvn anyway (since we don't actually package them in our assembly). I am not sure what the policy is if we pull it into a different repo and if we can keep the same groupId and artifactId, but that could be an alternative and most likely will not break too many users. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196111340 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196111343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53043/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196111225 **[Test build #53043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53043/consoleFull)** for PR 11672 at commit [`563614f`](https://github.com/apache/spark/commit/563614f03b6ddb90a1b60c5d569e0045f617ae04). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196087049 **[Test build #53043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53043/consoleFull)** for PR 11672 at commit [`563614f`](https://github.com/apache/spark/commit/563614f03b6ddb90a1b60c5d569e0045f617ae04). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-196086887 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-195913903 **[Test build #53023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53023/consoleFull)** for PR 11672 at commit [`563614f`](https://github.com/apache/spark/commit/563614f03b6ddb90a1b60c5d569e0045f617ae04). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-195913913 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53023/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-195913912 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-195911951 **[Test build #53023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53023/consoleFull)** for PR 11672 at commit [`563614f`](https://github.com/apache/spark/commit/563614f03b6ddb90a1b60c5d569e0045f617ae04). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-195890923 @ksakellis Most of Streaming users use Kafka. It may affect a lot of people. I think it's better to discuss how to support Kafka 0.9 ([SPARK-13252](https://issues.apache.org/jira/browse/SPARK-13252)) before deciding whether moving Kafka out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-195889408 cc @rxin @tdas @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13843][Streaming]Remove streaming-flume...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11672#issuecomment-195889378 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
