[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15367 Thanks! I'm going to merge this one since the concern from @koeninger is addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15367 No, if we backport this I would plan to continue to backport changes (that are safe) until the next release. Either way this should not affect what goes into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15367 Does backporting reduce the likelihood of change if user feedback indicates we got it wrong? My technical concerns were largely addressed, that's my big remaining organizational concern. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/15367 @marmbrus @zsxwing I agree its experimental and we should have more flexibility here with backports. I also very much agree that structured streaming in its current state on 2.0 isn't usable - but I'm not super sure that backporting fixes is the best way to do this? Honestly I spend most of my time focused on Python & ML (and I've only really been looking at structured streaming with those two hats on). I'm really cautious about the idea 2k+ line backport which hasn't even been released otherwise but I don't have any specific objections to the changes its just making me nervous. The fact the whats being backported seems to still be under development is also concerning since doing this backport now puts us in a position of backporting more (not yet merged into mainline) fixes. Of course - If the people with the most experience in this area all agree (and most of y'all [ @marmbrus @zsxwing @tdas but maybe missing @koeninger ] seem to already be on this PR so I'll leave you to it) that this backport reasonable that is great - it would probably be good to follow up to the original backport mailing list thread and update the wiki as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15367 > @zsxwing Have you tested the maven build? > Test build #66433 has finished for PR 15367 at commit 0915826. > > This patch passes all tests. > This patch merges cleanly. > This patch adds no public classes. This one is maven build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15367 @zsxwing Have you test the maven build? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15367 We should definitly vet this PR carefully to make sure its safe. One thing that is missing from that guide, that I do believe is accepted practice, is more leeway when the feature is marked experimental (i.e. during Spark 1.0-1.3 I used to backport pretty much everything that went into SQL). Since I think many people are simply not using structured streaming due to lack of kafka support, I'd error on the side of more users sooner. If you think any part of this PR is unsafe with respect stable components though, please do raise those concerns. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15367 @holdenk this is basically adding an external project. Without the Kafka source, it's hard for people to test Structured Streaming with their real data. Considering 2.0.2 will be out before 2.1.0, it's better to add this one into branch 2.0 in order to test it early. Since Structured Streaming is experimental, it won't break any production usage and should be safe. /cc @rxin @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/15367 This is rather a lot of functionality to backport, our backport guide doesn't seem to quite line up with this (based on https://www.mail-archive.com/dev@spark.apache.org/msg10284.html / https://cwiki.apache.org/confluence/display/SPARK/Committers#Committers-PolicyonBackportingBugFixes ) but @pwendell will know better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15367 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66433/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15367 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15367 **[Test build #66433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66433/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15367 **[Test build #66433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66433/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15367 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15367 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15367 **[Test build #66432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66432/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15367 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66432/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15367 **[Test build #66432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66432/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15367 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
