[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16387 **[Test build #71068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71068/testReport)** for PR 16387 at commit [`5645533`](https://github.com/apache/spark/commit/5645533ede8bb37aa316afbc0d003d042bf5d52d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARK-19133][SPARKR][ML] fix glm for Gamma, clarify glm...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16511 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARK-19133][SPARKR][ML] fix glm for Gamma, clarify glm...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16511 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71066/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARK-19133][SPARKR][ML] fix glm for Gamma, clarify glm...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16511 **[Test build #71066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71066/testReport)** for PR 16511 at commit [`0688aa4`](https://github.com/apache/spark/commit/0688aa4691e630e22a2016ded7e002221a1ec16a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16486: [SPARK-13610][ML] Create a Transformer to disassemble ve...
Github user leonfl commented on the issue: https://github.com/apache/spark/pull/16486 @jkbradley, Could you also help to check this patch cause you are familiar with this defect, Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16513: [SPARK-12757][CORE] lower "block locks were not released...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16513 **[Test build #71067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71067/testReport)** for PR 16513 at commit [`dc81625`](https://github.com/apache/spark/commit/dc816254226baa2ff3cbdbdf421a3ac517f5a2c8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16513: [SPARK-12757][CORE] lower "block locks were not released...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16513 @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16513: [SPARK-12757][CORE] lower "block locks were not r...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/16513 [SPARK-12757][CORE] lower "block locks were not released" log to info level ## What changes were proposed in this pull request? lower "block locks were not released" log to info level You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark blocklockswarn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16513 commit dc816254226baa2ff3cbdbdf421a3ac517f5a2c8 Author: Felix Cheung Date: 2017-01-09T07:23:39Z lower log to info --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15671 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15671 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71064/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15671 **[Test build #71064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71064/testReport)** for PR 15671 at commit [`e6b4615`](https://github.com/apache/spark/commit/e6b46153ee82f722ab95021dc82ab81d34dfd4ff). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARK-19133][SPARKR][DOCS] fix glm for Gamma, clarify g...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16511 **[Test build #71066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71066/testReport)** for PR 16511 at commit [`0688aa4`](https://github.com/apache/spark/commit/0688aa4691e630e22a2016ded7e002221a1ec16a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16511 turns out Gamma is broken. will be working on adding tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71065/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16432 **[Test build #71065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71065/testReport)** for PR 16432 at commit [`2673af2`](https://github.com/apache/spark/commit/2673af288c0c9472af3aa8e68782d6e572d12346). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15539 It could return incorrect results, but I need to prove it using a use case. We always call [refresh0](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala#L50) in [`filterPartitions`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala#L78-L79). Thus, it covers most cases. Let me try more in the next few days. Tonight, let me first fix the cache inconsistence issue caused by `SET LOCATION`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16510 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71062/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16510 **[Test build #71062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71062/testReport)** for PR 16510 at commit [`a706c15`](https://github.com/apache/spark/commit/a706c15ea699e41dd720e3c1f4c76e880ea7adc8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16510 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15539 Hm, what use cases are we trying to address? As I understand, the worst that can happen if the cache size flag is toggled at runtime is that the old settings might still apply. And when the filesource partition management flag is changed, it only takes effect after the table entry in `cachedDataSourceTables` in HiveMetastoreCatalog is refreshed. If we document the above, is that enough? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16512 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71061/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16512 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16512 **[Test build #71061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71061/testReport)** for PR 16512 at commit [`b66a0ac`](https://github.com/apache/spark/commit/b66a0ac4748bbf14dfb992aeff95028122b6d7a9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16497: [SPARK-19118] [SQL] Percentile support for frequency dis...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/16497 @tanejagagan Would you please revert the unnecessary code changes? BTW: You can test the scala style on local environment by running `./dev/lint-scala` from console. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16501: [SPARK-19117][TESTS] Skip the tests using script transfo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16501 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71058/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16501: [SPARK-19117][TESTS] Skip the tests using script transfo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16501 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16501: [SPARK-19117][TESTS] Skip the tests using script transfo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16501 **[Test build #71058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71058/testReport)** for PR 16501 at commit [`0f654f3`](https://github.com/apache/spark/commit/0f654f347627d18a65d748a2ec9d60023568a531). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16432 **[Test build #71065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71065/testReport)** for PR 16432 at commit [`2673af2`](https://github.com/apache/spark/commit/2673af288c0c9472af3aa8e68782d6e572d12346). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16511 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16511 **[Test build #71060 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71060/testReport)** for PR 16511 at commit [`3403567`](https://github.com/apache/spark/commit/3403567704946a546ab229bceb8ea28e91e0d759). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16511 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71060/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16474 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71055/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16474 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16474 **[Test build #71055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71055/testReport)** for PR 16474 at commit [`6b562eb`](https://github.com/apache/spark/commit/6b562eba8b7d2a508cfd6f972bc18a108c3ed044). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16233 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16233 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71057/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15671 **[Test build #71064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71064/testReport)** for PR 15671 at commit [`e6b4615`](https://github.com/apache/spark/commit/e6b46153ee82f722ab95021dc82ab81d34dfd4ff). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16474 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16233 **[Test build #71057 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71057/testReport)** for PR 16233 at commit [`ad8957c`](https://github.com/apache/spark/commit/ad8957cbdb408e3c18f65f1bff5d14df447bd39c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16474 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71053/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16474 **[Test build #71053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71053/testReport)** for PR 16474 at commit [`d6878e1`](https://github.com/apache/spark/commit/d6878e1087eb9d7c32c1084e908a967326c75087). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16401 OK I'll submit a pr to fix this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16401 hm concrete suggestion: 1. pass the conf and cache the computed statistics the first time 2. have a simple invalidateStatsCache method that can be called manually to invalidate. Basically what I suggested in https://github.com/apache/spark/pull/16401#issuecomment-270841604 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16401 So you mean we still pass the conf and do caching, and also add the invalidation method, although the cache invalidation logic has some problem in current stage? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16411: [SPARK-17984][YARN][Mesos][Deploy][WIP] add executor lau...
Github user xiaochang-wu commented on the issue: https://github.com/apache/spark/pull/16411 @tgravescs Thanks for your comments. There are two things we have tried. 1) To add a prefix command on executor launch I agree this opens a door for user to do anything for launching the executor. This patch is intended for profiling and debugging. May not fit for production. I am not sure it's the best form to implement, it fixed our problem quickly. 2) NUMA The script attached about NUMA is only an example to show how to use this patch. User can customize it to fulfill their specific needs. Automatic NUMA balancing is by default enabled on our system. As mentioned in the original Redhat slides, It can only deal with certain cases and still can not beat manual pinning. From our experiments, not all cases have big NUMA penalties. We should use some platform tools such as Intel VTune to identify if there is a NUMA problem and tune case by case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16395 @ron8hu Can you update the test cases based on the latest master? We have a new test infrastructure now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12414: [SPARK-14657] [ML] RFormula w/o intercept should output ...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/12414 hi - where are we on this? @yanboliang could you add `[SPARKR]` to the title --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16510 **[Test build #71062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71062/testReport)** for PR 16510 at commit [`a706c15`](https://github.com/apache/spark/commit/a706c15ea699e41dd720e3c1f4c76e880ea7adc8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15480 **[Test build #71063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71063/testReport)** for PR 15480 at commit [`3d31cb3`](https://github.com/apache/spark/commit/3d31cb3e6309950bb723a1b83a004caab493c5e7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16512 **[Test build #71061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71061/testReport)** for PR 16512 at commit [`b66a0ac`](https://github.com/apache/spark/commit/b66a0ac4748bbf14dfb992aeff95028122b6d7a9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16401 But you can't do a lazy val unless you have the conf passed in, since you will be adding knobs to the CBO stats soon wouldn't you? Also there is really nothing basic vs CBO here. Both are returning the same set of statistics, just using a different way to estimate. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/16432#discussion_r95104135 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala --- @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider } // Get the time of next renewal. -tokenRenewalInterval.map { interval => - creds.getAllTokens.asScala -.filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND) +tokenRenewalInterval.flatMap { interval => + val nextRenewalDates = creds.getAllTokens.asScala + .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier]) --- End diff -- Yeah, that's the issue, let me think about how to handle it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16512: [SPARK-18335][SPARKR] createDataFrame to support ...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/16512 [SPARK-18335][SPARKR] createDataFrame to support numPartitions parameter ## What changes were proposed in this pull request? To allow specifying number of partitions when the DataFrame is created ## How was this patch tested? manual, unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rnumpart Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16512.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16512 commit b66a0ac4748bbf14dfb992aeff95028122b6d7a9 Author: Felix Cheung Date: 2017-01-09T05:16:39Z add numPartitions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/16432#discussion_r95104079 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala --- @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider } // Get the time of next renewal. -tokenRenewalInterval.map { interval => - creds.getAllTokens.asScala -.filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND) +tokenRenewalInterval.flatMap { interval => + val nextRenewalDates = creds.getAllTokens.asScala + .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier]) .map { t => - val identifier = new DelegationTokenIdentifier() - identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier))) + val identifier = t.decodeIdentifier().asInstanceOf[AbstractDelegationTokenIdentifier] identifier.getIssueDate + interval - }.foldLeft(0L)(math.max) +} + if (nextRenewalDates.isEmpty) None else Some(nextRenewalDates.min) --- End diff -- I think the logic was incorrect previously, we should get the minimum renewal date, that's why I changed here. But the code was probably OK if only we only have one default HDFS configured. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16511 **[Test build #71060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71060/testReport)** for PR 16511 at commit [`3403567`](https://github.com/apache/spark/commit/3403567704946a546ab229bceb8ea28e91e0d759). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16511: [SPARKR][DOCS] clarify glm family supported
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16511#discussion_r95103899 --- Diff: R/pkg/R/mllib_regression.R --- @@ -52,6 +52,8 @@ setClass("IsotonicRegressionModel", representation(jobj = "jobj")) #' This can be a character string naming a family function, a family function or #' the result of a call to a family function. Refer R family at #' \url{https://stat.ethz.ch/R-manual/R-devel/library/stats/html/family.html}. +#' Currently these families are supported: \code{binomial}, \code{gaussian}, +#' \code{Gamma}, and \code{poisson}. --- End diff -- in R, `Gamma` family is capital `R` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16511 @yanboliang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16511: [SPARKR][DOCS] clarify glm family supported
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/16511 [SPARKR][DOCS] clarify glm family supported ## What changes were proposed in this pull request? R family is a longer list than what Spark supports. ## How was this patch tested? manual You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rdocglmfamily Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16511.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16511 commit 3403567704946a546ab229bceb8ea28e91e0d759 Author: Felix Cheung Date: 2017-01-09T05:31:06Z doc family supported --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16432#discussion_r95103785 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala --- @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider } // Get the time of next renewal. -tokenRenewalInterval.map { interval => - creds.getAllTokens.asScala -.filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND) +tokenRenewalInterval.flatMap { interval => + val nextRenewalDates = creds.getAllTokens.asScala + .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier]) --- End diff -- Can we make sure all instances of `AbstractDelegationTokenIdentifier` are tokens for Hadoop file systems? Isn't Hive's `DelegationTokenIdentifier` also an `AbstractDelegationTokenIdentifier`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16431: [SPARK-19020] [SQL] Cardinality estimation of aggregate ...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16431 OK, I'll update this pr today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16401 If our goal is to fix two sets of stats issue first, is it ok to just use two `lazy val`s now, and leave the configuration and cache invalidation issues in the future? In current implementation, we don't need any configurations except the cbo switch conf. ``` def statistics(conf: CatalystConf): Statistics = { if (conf.cboEnabled) cboStats else basicStats } lazy val cboStats... lazy val basicStats... ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16432#discussion_r95103643 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala --- @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider } // Get the time of next renewal. -tokenRenewalInterval.map { interval => - creds.getAllTokens.asScala -.filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND) +tokenRenewalInterval.flatMap { interval => + val nextRenewalDates = creds.getAllTokens.asScala + .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier]) .map { t => - val identifier = new DelegationTokenIdentifier() - identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier))) + val identifier = t.decodeIdentifier().asInstanceOf[AbstractDelegationTokenIdentifier] identifier.getIssueDate + interval - }.foldLeft(0L)(math.max) +} + if (nextRenewalDates.isEmpty) None else Some(nextRenewalDates.min) --- End diff -- Why previously we use math.max and now changed to minimum value? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16431: [SPARK-19020] [SQL] Cardinality estimation of agg...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16431#discussion_r95103316 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/AggregateEstimation.scala --- @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.catalyst.plans.logical.estimation + +import org.apache.spark.sql.catalyst.expressions.Attribute +import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Statistics} + + +object AggregateEstimation { + import EstimationUtils._ + + def estimate(agg: Aggregate): Option[Statistics] = { +val childStats = agg.child.statistics +// Check if we have column stats for all group-by columns. +val colStatsExist = agg.groupingExpressions.forall { e => + e.isInstanceOf[Attribute] && childStats.attributeStats.contains(e.asInstanceOf[Attribute]) +} +if (rowCountsExist(agg.child) && colStatsExist) { + // Initial value for agg without group expressions + var outputRows: BigInt = 1 + agg.groupingExpressions.map(_.asInstanceOf[Attribute]).foreach { attr => +val colStat = childStats.attributeStats(attr) +// Multiply distinct counts of group by columns. This is an upper bound, which assumes +// the data contains all combinations of distinct values of group by columns. +outputRows *= colStat.distinctCount + } + + // The number of output rows must not be larger than child's number of rows. + // Note that this also covers the case of uniqueness of column. If one of the group-by columns --- End diff -- OK I don't think you need this explanation here -- it simply makes it more confusing. You are just putting an upper bound on cardinality, and that explains everything. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16431: [SPARK-19020] [SQL] Cardinality estimation of aggregate ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16431 Can you update the pull request and the test cases to use the new test infra? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16430 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16510 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16510 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71059/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16510 **[Test build #71059 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71059/testReport)** for PR 16510 at commit [`f90a841`](https://github.com/apache/spark/commit/f90a8414bfc71f69b5785e689e719b6d45c45024). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16430: [SPARK-17077] [SQL] Cardinality estimation for project o...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16430 Alright I'm going to merge this since this patch introduces test infrastructure that can be used by other tests. Please submit a follow-up PR to add more test cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Github user gczsjdy commented on a diff in the pull request: https://github.com/apache/spark/pull/16476#discussion_r95102912 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -340,3 +341,91 @@ object CaseKeyWhen { CaseWhen(cases, elseValue) } } + +/** + * A function that returns the index of str in (str1, str2, ...) list or 0 if not found. + * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType. + */ +@ExpressionDescription( + usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.", + extended = """ +Examples: + > SELECT _FUNC_(10, 9, 3, 10, 4); + 3 + """) +case class Field(children: Seq[Expression]) extends Expression { + + override def nullable: Boolean = false + override def foldable: Boolean = children.forall(_.foldable) + + private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType) + + override def checkInputDataTypes(): TypeCheckResult = { +if (children.length <= 1) { + TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments") +} else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) { + TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType") +} else + TypeCheckResult.TypeCheckSuccess + } + + override def dataType: DataType = IntegerType + + override def eval(input: InternalRow): Any = { +val target = children.head.eval(input) +val targetDataType = children.head.dataType +def findEqual(target: Any, params: Seq[Expression], index: Int): Int = { + params.toList match { --- End diff -- @chenghao-intel I have added that annotation and remove the `toList`. Still working on the rule for folding not match datatypes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Github user gczsjdy commented on a diff in the pull request: https://github.com/apache/spark/pull/16476#discussion_r95102842 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -340,3 +341,91 @@ object CaseKeyWhen { CaseWhen(cases, elseValue) } } + +/** + * A function that returns the index of str in (str1, str2, ...) list or 0 if not found. + * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType. + */ +@ExpressionDescription( + usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.", + extended = """ +Examples: + > SELECT _FUNC_(10, 9, 3, 10, 4); + 3 + """) +case class Field(children: Seq[Expression]) extends Expression { + + override def nullable: Boolean = false + override def foldable: Boolean = children.forall(_.foldable) + + private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType) + + override def checkInputDataTypes(): TypeCheckResult = { +if (children.length <= 1) { + TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments") +} else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) { + TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType") +} else + TypeCheckResult.TypeCheckSuccess + } + + override def dataType: DataType = IntegerType + + override def eval(input: InternalRow): Any = { +val target = children.head.eval(input) +val targetDataType = children.head.dataType +def findEqual(target: Any, params: Seq[Expression], index: Int): Int = { + params.toList match { +case Nil => 0 +case head::tail if targetDataType == head.dataType + && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index +case _ => findEqual(target, params.tail, index + 1) + } +} +if(target == null) + 0 +else + findEqual(target, children.tail, 1) + } + + protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { --- End diff -- Yes, because in `checkEvaluation` function there is : `checkEvaluationWithGeneratedMutableProjection` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16507: [SPARK-18903][SPARKR][BACKPORT-2.1] Add API to ge...
Github user felixcheung closed the pull request at: https://github.com/apache/spark/pull/16507 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16507: [SPARK-18903][SPARKR][BACKPORT-2.1] Add API to get Spark...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16507 merged to branch-2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16504: [SPARK-19126][Docs] Update Join Documentation Acr...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16504 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16504: [SPARK-19126][Docs] Update Join Documentation Across Lan...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16504 merged to master & branch-2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16504: [SPARK-19126][Docs] Update Join Documentation Across Lan...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16504 @anabranch the failure is not from Jenkins but from (R only) AppVeyor tests. It looks like it is nothing related - it failed at access github. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16510 **[Test build #71059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71059/testReport)** for PR 16510 at commit [`f90a841`](https://github.com/apache/spark/commit/f90a8414bfc71f69b5785e689e719b6d45c45024). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16510: [SPARK-19130][SPARKR] Support setting literal val...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/16510 [SPARK-19130][SPARKR] Support setting literal value as column implicitly ## What changes were proposed in this pull request? ``` df$foo <- 1 ``` instead of ``` df$foo <- lit(1) ``` ## How was this patch tested? unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rlitcol Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16510.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16510 commit f90a8414bfc71f69b5785e689e719b6d45c45024 Author: Felix Cheung Date: 2017-01-09T04:22:15Z setting literal value --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71056/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16432 **[Test build #71056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71056/testReport)** for PR 16432 at commit [`3b4c4b0`](https://github.com/apache/spark/commit/3b4c4b0bed43a04fbbb1f3de7e45e34f1007a222). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71054/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16432 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16501: [WIP][SPARK-19117][TESTS] Skip the tests using script tr...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16501 Build started: [TESTS] `org.apache.spark.sql.hive.execution.HiveQuerySuite` [![PR-16501](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=072A48DF-9BFF-488E-9510-4FE37B211F68&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/072A48DF-9BFF-488E-9510-4FE37B211F68) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16432 **[Test build #71054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71054/testReport)** for PR 16432 at commit [`86c76a6`](https://github.com/apache/spark/commit/86c76af0bdb4fda15e420d1103dffcb375e3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16501: [WIP][SPARK-19117][TESTS] Skip the tests using script tr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16501 **[Test build #71058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71058/testReport)** for PR 16501 at commit [`0f654f3`](https://github.com/apache/spark/commit/0f654f347627d18a65d748a2ec9d60023568a531). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15539 Yeah, I think we should document the behavior issues when different sessions are using different conf values. Will do it. I think we also need to evict all the cache that are associated with the specific client when users make a switch. Do you think this makes sense? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16443: [SPARK-19042] spark executor can't download the jars whe...
Github user hustfxj commented on the issue: https://github.com/apache/spark/pull/16443 @srowen I can't fix the description at this PR, so I close the PR and recreate the new PR for this bug. You can review it at https://github.com/apache/spark/pull/16509. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16443: [SPARK-19042] spark executor can't download the j...
Github user hustfxj closed the pull request at: https://github.com/apache/spark/pull/16443 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15539 `(ClientId, Path), Array[FileStatus]` uh... `FileStatusCache` does not share any entries with any other client, but does share memory resources for the purpose of cache eviction. Sorry, I misunderstood the code. I think we need to evict all the cache that are associated with the specific client when users make a switch. Do you think this makes sense? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16509: [SPARK-19042] spark executor can't download the jars whe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16509 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16509: [SPARK-19042] spark executor can't download the j...
GitHub user hustfxj opened a pull request: https://github.com/apache/spark/pull/16509 [SPARK-19042] spark executor can't download the jars when uber jar's http url conta⦠You can merge this pull request into a Git repository by running: $ git pull https://github.com/hustfxj/spark bug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16509.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16509 commit 1a0bd73fa891f8449e2c86f756522323426c37d9 Author: xiaojian.fxj Date: 2017-01-09T03:45:59Z spark executor can't download the jars when uber jar's http url contains query string --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16233 **[Test build #71057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71057/testReport)** for PR 16233 at commit [`ad8957c`](https://github.com/apache/spark/commit/ad8957cbdb408e3c18f65f1bff5d14df447bd39c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16474 **[Test build #71055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71055/testReport)** for PR 16474 at commit [`6b562eb`](https://github.com/apache/spark/commit/6b562eba8b7d2a508cfd6f972bc18a108c3ed044). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16432 **[Test build #71056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71056/testReport)** for PR 16432 at commit [`3b4c4b0`](https://github.com/apache/spark/commit/3b4c4b0bed43a04fbbb1f3de7e45e34f1007a222). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16508: [spark-18806] [core] the processors DriverWrapper and Co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16508 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16508: [spark-18806] [core] the processors DriverWrapper...
GitHub user liujianhuiouc opened a pull request: https://github.com/apache/spark/pull/16508 [spark-18806] [core] the processors DriverWrapper and CoarseGrainedExecutorBackend should be exit when worker exit ## What changes were proposed in this pull request? modify the method `isWorker` by adding hostname equals ## How was this patch tested? test by manual You can merge this pull request into a Git repository by running: $ git pull https://github.com/liujianhuiouc/spark-1 spark-18806 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16508.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16508 commit 665921411606fcbad40de1b7f571aa323c5a0fa7 Author: liujianhui Date: 2017-01-09T03:20:14Z [spark-18806] whether the remote address is worker according to the ip --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/16432 @tgravescs thanks for your comments, I also tested with webhdfs with configuration like: ``` --conf spark.yarn.access.namenodes=webhdfs://c6402.ambari.apache.org:50070/ ``` And the related logs are: ``` 17/01/09 03:15:49 INFO HadoopFSCredentialProvider: getting token for: webhdfs://c6402.ambari.apache.org:50070/ 17/01/09 03:15:50 INFO HadoopFSCredentialProvider: getting token for: hdfs://c6402.ambari.apache.org:8020/user/spark 17/01/09 03:15:50 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 138 for spark on 192.168.64.102:8020 17/01/09 03:15:50 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 140 for spark on 192.168.64.102:8020 17/01/09 03:15:50 INFO HadoopFSCredentialProvider: Renewal interval is 86400112 for token WEBHDFS delegation 17/01/09 03:15:50 INFO HadoopFSCredentialProvider: Renewal interval is 86400146 for token HDFS_DELEGATION_TOKEN ``` As you could see also get tokens and token renewal interval from webhdfs token. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16474 **[Test build #71053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71053/testReport)** for PR 16474 at commit [`d6878e1`](https://github.com/apache/spark/commit/d6878e1087eb9d7c32c1084e908a967326c75087). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org