[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 Thanks very much --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/21690 LGTM, merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 After applying this patch, my application can be running successfully. This issue could happen in the case of many topics(hundreds of ) consumed. ---

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/21690 What results are you seeing? On Thu, Jul 12, 2018, 6:53 AM Yuanbo Liu wrote: > @koeninger Sorry to interrupt, could you > take a look at my

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 @koeninger Sorry to interrupt, could you take a look at my patch? --- - To unsubscribe, e-mail:

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92709/ Test PASSed. ---

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21690 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21690 **[Test build #92709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92709/testReport)** for PR 21690 at commit

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21690 **[Test build #92709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92709/testReport)** for PR 21690 at commit

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 @koeninger Thanks for your reply. Agree with you. there is no need to to use pause repeatedly. This is my test without any pause, and the app master stuck for a long time without any process

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/21690 @yuanboliu What I'm suggesting is more like this: https://github.com/apache/spark/compare/master...koeninger:SPARK-24713?expand=1 ---

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 @koeninger Thanks for your details. Sorry quite busy this week. I will delete the last pause, test the patch on my own cluster this weekend and give feedback asap. ---

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/21690 @yuanboliu From reading KafkaConsumer code, and from testing, I don't see where consumer.position() alone would un-pause topicpartitions. See below. Can you give a counter-example? I

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 The first pause is used to stop poll() in the method paranoidPoll The second one is attached because of p.partition(). I'm not sure whether the state of pause will be rewritten after these

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21690 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92528/ Test PASSed. ---

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21690 **[Test build #92528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92528/testReport)** for PR 21690 at commit

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/21690 @yuanboliu Can you clarify why repeated pause is necessary? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21690 **[Test build #92528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92528/testReport)** for PR 21690 at commit

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user koeninger commented on the issue: https://github.com/apache/spark/pull/21690 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 @koeninger Sorry to interrupt, would you please review my patch? Thanks in advance. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21690 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21690 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21690 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional