[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19789 @daroo thanks for checking it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 @HyukjinKwon I've looked at the latest changes in the code and I think this problem shouldn't happen (at least in practice) anymore. So the JIRA can be closed. One minor issue I've noticed is

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 sure :-) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19789 @daroo, mind reopening this if you have some time to update? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19789 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19789 cc @marmbrus @zsxwing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 Hi @HyukjinKwon It seems that @zsxwing is not really interested in this PR. Should I cancel it and close https://issues.apache.org/jira/browse/SPARK-22562 ? BTW. Why AppVeyor CI

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #90551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90551/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90551/ Test PASSed. ---

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #90551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90551/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #85575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85575/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85575/ Test PASSed. ---

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #85575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85575/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19789 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 Hey @zsxwing Could you please have a look at my PR? It's been a month almost since I opened it. @koeninger said that it looks good to him, but everyone else is waiting for your feedback.

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19789 I think we should wait for @zsxwing's feedback. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 @HyukjinKwon @zsxwing Any chance to go ahead with this? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 I don't know what are the plans for the structured streaming which @zsxwing has in mind, but before I created the PR I actually had seen how this problem is currently solved in kafka-0-10-sql module

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19789 Seems reasonable to me but you should probably ask zsxwing if it fits in with plans for the structured streaming kafka code. On Thu, Nov 23, 2017 at 10:23 PM, Hyukjin Kwon

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84162/ Test PASSed. ---

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #84162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84162/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #84162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84162/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19789 cc @zsxwing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84143/ Test PASSed. ---

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #84143 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84143/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19789 **[Test build #84143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84143/testReport)** for PR 19789 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19789 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 Hey Spark commiters @jiangxb1987 @srowen @cloud-fan @HyukjinKwon Could you please authorize my PR for testing? Thanks Darek ---

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19789 You'll need to get a commiter's attention to merge it anyway On Nov 23, 2017 01:48, "Daroo" wrote: It seems that your "magic spell" didn't work. No

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 It seems that your "magic spell" didn't work. No build was triggered --- - To unsubscribe, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19789 ok to test On Wed, Nov 22, 2017 at 2:49 PM, Daroo wrote: > Cool. Could you please authorize it for testing? > > — > You are receiving this

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 Cool. Could you please authorize it for testing? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19789 Seems reasonable. On Wed, Nov 22, 2017 at 1:52 PM, Daroo wrote: > It fails on the current master branch and doesn't after the patch > > —

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 It fails on the current master branch and doesn't after the patch --- - To unsubscribe, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19789 What are you actually asserting in that test and/or does it reliably fail if run on the version of your code before the patch? On Wed, Nov 22, 2017 at 1:33 PM, Daroo

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 I've added a test. @koeninger is it something you had in mind? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19789 Yeah, subscribepattern could definitely be an issue. As far as unit testing, have you tried anything along the lines of setting the cache size artificially low and then introducing new

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 I see you point, but it's kind of difficult to size it properly when you use SubscribePattern (i.e. dynamic number of topics/partitions) consumer strategy ---

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19789 My main comment is that if you have a situation where there's actually contention on the size of the cache, chances are things are going to be screwed up anyway due to consumers being recreated

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user daroo commented on the issue: https://github.com/apache/spark/pull/19789 Hey @koeninger , could you please have a look at my PR? Thanx a lot, Darek --- - To unsubscribe, e-mail:

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19789 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional