[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
Github user holdenk closed the pull request at: https://github.com/apache/spark/pull/11287 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11287#issuecomment-186900836 I did some digging into why its 32 in jenkins and 4 on my machine - digging into the Scala internals it seems that the (private) FutureTasks doesn't actually look at the provided threadpool when determining parallelism rather it looks at the system. I'll close this for now and look at switching this over to ForkJoinTaskSupport (although it makes it pretty difficult to name the thread pool something meaningful :( ). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11287#issuecomment-186856227 This looks conceptually fine, though the test failed. I am not sure why. I stared at this for a while and don't see what would cause it to not be limited to 8 threads in the test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11287#issuecomment-186781248 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51616/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11287#issuecomment-186781246 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11287#issuecomment-186781164 **[Test build #51616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51616/consoleFull)** for PR 11287 at commit [`f2e6a25`](https://github.com/apache/spark/commit/f2e6a25ae2581e9b6cc43950c1302c26a8602c2a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11287#issuecomment-186775743 **[Test build #51616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51616/consoleFull)** for PR 11287 at commit [`f2e6a25`](https://github.com/apache/spark/commit/f2e6a25ae2581e9b6cc43950c1302c26a8602c2a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13398][STREAMING] Change FileWriteAhead...
GitHub user holdenk opened a pull request: https://github.com/apache/spark/pull/11287 [SPARK-13398][STREAMING] Change FileWriteAheadLogger to not depend on deprecated ThreadPoolTaskSupport ## What changes were proposed in this pull request? Replace deprecated usage of ThreadPoolTaskSupport with ExecutionContextTaskSupport and incidentally update the comment to indicate that the number of concurrent objects is bounded by the threadpool size or 8 whichever is larger (updating the documentation to match code). ## How was the this patch tested? unit tests, streaming/testOnly org.apache.spark.streaming.util.WriteAheadLogUtilsSuite You can merge this pull request into a Git repository by running: $ git pull https://github.com/holdenk/spark SPARK-13398-move-away-from-ThreadPoolTaskSupport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11287.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11287 commit 68df46a1093132ad3d5eef3e7a3d669105083236 Author: Holden Karau Date: 2016-02-21T07:08:45Z Change FileWriteAheadLogger to not depend on deprecated ThreadPoolContext commit 9991ebce07b6fabe122719ba1c4690e733297b18 Author: Holden Karau Date: 2016-02-21T07:13:40Z Remove now unused import commit f2e6a25ae2581e9b6cc43950c1302c26a8602c2a Author: Holden Karau Date: 2016-02-21T07:19:27Z Update the comment to point out if the thread pool is smaller than 8, that is the number of objects kept in memory. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
