Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/20767
@tdas @zsxwing @koeninger @tedyu do you think it makes sense to make
similar step in the DStream area like this and then later follow with the
mentioned Apache Common Pool?
---
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
Just to be clear, I am not saying that we *have to* move to this pool
stuff. I am just saying that if we want to make this more robust, then we
should try to use existing tools (after careful
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/20767
Interesting.
https://commons.apache.org/proper/commons-pool/apidocs/org/apache/commons/pool2/impl/BaseGenericObjectPool.html#getBorrowedCount()
---
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
@tedyu It was indeed hard to find :) But apache commons pool does expose
metrics on idle/active counts. See
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/20767
I did a quick search for 'apache commons pool metrics' which didn't show up
directly related links.
---
-
To unsubscribe, e-mail:
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
@tedyu @zsxwing My thoughts on this is that we should consider migrating to
something like Apache Common Pool (assuming it does not require additional
maven libraries), which might be less maintenance
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20767
@tedyu that's a good idea. Could you create a ticket? Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/20767
@tdas
Do you think a follow on JIRA can be logged for adding metrics for the
cache operations ?
Thanks
---
-
To
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20767
@tdas this is much simpler!!! LGTM. Merging to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
The idea is good. But how do you propose exposing that information?
Periodic print out in the log?
From a different angle, I would rather not do feature creep in this PR that
is
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88285 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88285/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88285/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88285 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88285/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
@tedyu @zsxwing thank you very much for catching the bugs. I have
simplified the logic quite a bit. Note that I removed the invariant that I had
introduced earlier. Additionally, I locally ran the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1545/
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
@koeninger good question Cody! I think we should fix this limitation
eventually. The only reason I am not doing that in this PR is to keep the
changes minimum for backporting to 2.3.x. Eventually, we
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/20767
Can you clarify why you want to allow only 1 cached consumer per
topicpartition, closing any others at task end?
It seems like opening and closing consumers would be less efficient than
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88140 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88140/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88140/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1439/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88140 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88140/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88109/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88109 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88109/testReport)**
for PR 20767 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88109 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88109/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1414/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88082/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88082 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88082/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88081 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88081/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88081/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88082 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88082/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1388/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
jenkins retest this
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88081 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88081/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1387/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88070 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88070/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88070/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20767
**[Test build #88070 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88070/testReport)**
for PR 20767 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20767
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1376/
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20767
@zsxwing @brkyvz PTAL.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
51 matches
Mail list logo
