[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS

2016-08-20 Thread Sherry302
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @steveloughran Thank you very much for the comments. I have created an Hadoop jira [HADOOP-13527 ](https://issues.apache.org/jira/browse/HADOOP-13527) and attached the patch, could you please

[GitHub] spark issue #14738: [MINOR][ML]Add expert param support to SharedParamsCodeG...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14738 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14738: [MINOR][ML]Add expert param support to SharedParamsCodeG...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14738 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64159/ Test PASSed. ---

[GitHub] spark issue #14738: [MINOR][ML]Add expert param support to SharedParamsCodeG...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14738 **[Test build #64159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64159/consoleFull)** for PR 14738 at commit

[GitHub] spark issue #14738: [MINOR][ML]Add expert param support to SharedParamsCodeG...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14738 **[Test build #64159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64159/consoleFull)** for PR 14738 at commit

[GitHub] spark pull request #14738: [MINOR][ML]Add expert param support to SharedPara...

2016-08-20 Thread hqzizania
GitHub user hqzizania opened a pull request: https://github.com/apache/spark/pull/14738 [MINOR][ML]Add expert param support to SharedParamsCodeGen ## What changes were proposed in this pull request? Add expert param support to SharedParamsCodeGen where aggregationDepth a

[GitHub] spark pull request #14625: [SPARK-17045] [SQL] Build/move Join-related test ...

2016-08-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14625#discussion_r75588865 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala --- @@ -245,6 +245,10 @@ class SQLQueryTestSuite extends QueryTest with

[GitHub] spark pull request #14625: [SPARK-17045] [SQL] Build/move Join-related test ...

2016-08-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14625#discussion_r75588856 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala --- @@ -245,6 +245,10 @@ class SQLQueryTestSuite extends QueryTest with

[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...

2016-08-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14712 Spark SQL already has its own metastore: `InMemoryCatalog`. And we do have an abstraction for metasotre: `ExternalCatalog`. We have 2 targets here: 1. add table statistics in Spark SQL

[GitHub] spark pull request #14625: [SPARK-17045] [SQL] Build/move Join-related test ...

2016-08-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14625#discussion_r75588814 --- Diff: sql/core/src/test/resources/sql-tests/inputs/join.sql --- @@ -0,0 +1,225 @@ +-- join nested table expressions (auto_join0.q) --- End

[GitHub] spark pull request #14601: [SPARK-13979][Core] Killed executor is re spawned...

2016-08-20 Thread agsachin
Github user agsachin commented on a diff in the pull request: https://github.com/apache/spark/pull/14601#discussion_r75588799 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -107,6 +107,14 @@ class SparkHadoopUtil extends Logging {

[GitHub] spark pull request #14601: [SPARK-13979][Core] Killed executor is re spawned...

2016-08-20 Thread agsachin
Github user agsachin commented on a diff in the pull request: https://github.com/apache/spark/pull/14601#discussion_r75588790 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -102,11 +102,19 @@ class SparkHadoopUtil extends Logging {

[GitHub] spark issue #14737: [Spark-17171][WEB UI] DAG will list all partitions in th...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14737 **[Test build #64158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64158/consoleFull)** for PR 14737 at commit

[GitHub] spark pull request #14737: [Spark-17171][WEB UI] DAG will list all partition...

2016-08-20 Thread cenyuhai
GitHub user cenyuhai opened a pull request: https://github.com/apache/spark/pull/14737 [Spark-17171][WEB UI] DAG will list all partitions in the graph ## What changes were proposed in this pull request? DAG will list all partitions in the graph, it is too slow and hard to see

[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...

2016-08-20 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/14712 I suggest in the current stage, we still follow hive's convention. When spark sql has its own metastore, we can bridge between these two metastores by a mapping between two different sets of

[GitHub] spark pull request #14717: [SPARK-17090][ML]Make tree aggregation level in l...

2016-08-20 Thread hqzizania
Github user hqzizania commented on a diff in the pull request: https://github.com/apache/spark/pull/14717#discussion_r75588359 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala --- @@ -389,4 +389,21 @@ private[ml] trait HasSolver extends Params {

[GitHub] spark pull request #14717: [SPARK-17090][ML]Make tree aggregation level in l...

2016-08-20 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/14717#discussion_r75588252 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala --- @@ -389,4 +389,21 @@ private[ml] trait HasSolver extends Params {

[GitHub] spark issue #14682: [SPARK-17104][SQL] LogicalRelation.newInstance should fo...

2016-08-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/14682 @cloud-fan Thank you for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #14712: [SPARK-17072] [SQL] support table-level statistic...

2016-08-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/14712#discussion_r75588230 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTableCommand.scala --- @@ -33,7 +34,7 @@ import

[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...

2016-08-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/14712 If it is a hive table, I think we should respect hive's statistics. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread sarutak
Github user sarutak commented on the issue: https://github.com/apache/spark/pull/14719 @cloud-fan Of course. I'll write a design doc soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #14625: [SPARK-17045] [SQL] Build/move Join-related test ...

2016-08-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14625#discussion_r75588146 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala --- @@ -245,6 +245,10 @@ class SQLQueryTestSuite extends QueryTest with

[GitHub] spark pull request #14625: [SPARK-17045] [SQL] Build/move Join-related test ...

2016-08-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14625#discussion_r75588118 --- Diff: sql/core/src/test/resources/sql-tests/inputs/join.sql --- @@ -0,0 +1,225 @@ +-- join nested table expressions (auto_join0.q) --- End diff

[GitHub] spark issue #14717: [SPARK-17090][ML]Make tree aggregation level in linear/l...

2016-08-20 Thread hqzizania
Github user hqzizania commented on the issue: https://github.com/apache/spark/pull/14717 Thanks for the reviews :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14719 It's really a hard problem and we have discussed it many times but can't reach a consensus. Do you mind sending a design doc first so that it's easy for other people to review and

[GitHub] spark pull request #14717: [SPARK-17090][ML]Make tree aggregation level in l...

2016-08-20 Thread hqzizania
Github user hqzizania commented on a diff in the pull request: https://github.com/apache/spark/pull/14717#discussion_r75588057 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala --- @@ -389,4 +389,21 @@ private[ml] trait HasSolver extends Params {

[GitHub] spark pull request #14717: [SPARK-17090][ML]Make tree aggregation level in l...

2016-08-20 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/14717#discussion_r75587929 --- Diff: mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala --- @@ -389,4 +389,21 @@ private[ml] trait HasSolver extends Params {

[GitHub] spark pull request #14717: [SPARK-17090][ML]Make tree aggregation level in l...

2016-08-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14717 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #14717: [SPARK-17090][ML]Make tree aggregation level in linear/l...

2016-08-20 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/14717 LGTM. Merge into master. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #14717: [SPARK-17090][ML]Make tree aggregation level in l...

2016-08-20 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/14717#discussion_r75587723 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -256,6 +256,17 @@ class LogisticRegression @Since("1.2.0")

[GitHub] spark pull request #14717: [SPARK-17090][ML]Make tree aggregation level in l...

2016-08-20 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/14717#discussion_r75587709 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -48,7 +48,7 @@ import

[GitHub] spark issue #14723: [SQL][WIP][Test] Supports object-based aggregation funct...

2016-08-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14723 Can you create a jira? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586776 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/AggregateWithObjectAggregateBufferSuite.scala --- @@ -0,0 +1,156 @@ +/* + * Licensed to the

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586764 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586760 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586661 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586622 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark issue #14674: [SPARK-17002][CORE]: Document that spark.ssl.protocol. i...

2016-08-20 Thread wangmiao1981
Github user wangmiao1981 commented on the issue: https://github.com/apache/spark/pull/14674 @srowen Do you have any suggestions on our discussion above? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586350 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586238 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586232 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586233 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark pull request #14723: [SQL][WIP][Test] Supports object-based aggregatio...

2016-08-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14723#discussion_r75586183 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala --- @@ -389,3 +389,89 @@ abstract class

[GitHub] spark issue #14705: [SPARK-16508][SparkR] Fix CRAN undocumented/duplicated a...

2016-08-20 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/14705 Thanks. Reviewing each change, I think we need this PR (14705) and PR #14734 in 2.0.1 - so maybe only a few lines of conflicts. --- If your project is set up for it, you can

[GitHub] spark issue #14705: [SPARK-16508][SparkR] Fix CRAN undocumented/duplicated a...

2016-08-20 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14705 Yeah so we can do a couple of things. One is we try to cherry-pick this PR to branch-2.0 and then fix all the merge conflicts that are thrown. I think that should handle cases where the method

[GitHub] spark issue #14731: [SPARK-17159] [streaming]: optimise check for new files ...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14731 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14731: [SPARK-17159] [streaming]: optimise check for new files ...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14731 **[Test build #64156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64156/consoleFull)** for PR 14731 at commit

[GitHub] spark issue #14731: [SPARK-17159] [streaming]: optimise check for new files ...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14731 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64156/ Test PASSed. ---

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14735 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14735 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64157/ Test PASSed. ---

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14735 **[Test build #64157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64157/consoleFull)** for PR 14735 at commit

[GitHub] spark issue #14705: [SPARK-16508][SparkR] Fix CRAN undocumented/duplicated a...

2016-08-20 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/14705 I think a subset of this should go to 2.0.1 as well (as requirement to fix warning for CRAN in 2.0.x), but it's a non-trivial port: mllib isoreg are new in 2.1.0 only. What's the best

[GitHub] spark issue #14735: [SPARK-17173][SPARKR] R MLlib refactor, cleanup, reforma...

2016-08-20 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/14735 This also tighten the signature for mllib by removing the previously unused `...`: ``` "summary", signature(object = "GeneralizedLinearRegressionModel")

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14735 **[Test build #64157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64157/consoleFull)** for PR 14735 at commit

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14719 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64155/ Test PASSed. ---

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14719 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14719 **[Test build #64155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64155/consoleFull)** for PR 14719 at commit

[GitHub] spark issue #13428: [SPARK-12666][CORE] SparkSubmit packages fix for when 'd...

2016-08-20 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/13428 Merged to master and branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...

2016-08-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13428 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #14731: [SPARK-17159] [streaming]: optimise check for new files ...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14731 **[Test build #64156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64156/consoleFull)** for PR 14731 at commit

[GitHub] spark pull request #14601: [SPARK-13979][Core] Killed executor is re spawned...

2016-08-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/14601#discussion_r75584298 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -107,6 +107,14 @@ class SparkHadoopUtil extends Logging {

[GitHub] spark issue #12695: [SPARK-14914] Normalize Paths/URIs for windows.

2016-08-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/12695 As #13868 does adopt `org.apache.hadoop.io.Path`, I don't see this patch being needed —though it may highlight some places where the new code may need applying --- If your project is set

[GitHub] spark issue #12695: [SPARK-14914] Normalize Paths/URIs for windows.

2016-08-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/12695 If you are working with windows paths; Hadoop's Path class contains the code to do this, stabilised and addressing the corner cases --- If your project is set up for it, you can reply to

[GitHub] spark pull request #14601: [SPARK-13979][Core] Killed executor is re spawned...

2016-08-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/14601#discussion_r75584303 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala --- @@ -102,11 +102,19 @@ class SparkHadoopUtil extends Logging {

[GitHub] spark issue #14718: [SPARK-16711] YarnShuffleService doesn't re-init properl...

2016-08-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14718 Moving the jackson/leveldb dependencies isn't going to create problems on the yarn shuffle CP are they? Given the versions aren't changing, I'm not too worried —I just want to make sure

[GitHub] spark issue #14736: [SPARK-17024][SQL] Weird behaviour of the DataFrame when...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14736 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS

2016-08-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14659 That Caller context doesn't list Spark as one of the users in its LimitedPrivate scope. Add a Hadoop patch there and I'll get it in. This avoids arguments later when someone breaks the API,

[GitHub] spark pull request #14736: [SPARK-17024][SQL] Weird behaviour of the DataFra...

2016-08-20 Thread izeigerman
GitHub user izeigerman opened a pull request: https://github.com/apache/spark/pull/14736 [SPARK-17024][SQL] Weird behaviour of the DataFrame when a column name contains dots. ## What changes were proposed in this pull request? The Spark SQL doesn’t support field names

[GitHub] spark issue #14038: [SPARK-16317][SQL] Add a new interface to filter files i...

2016-08-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14038 Path filtering in Hadoop FS calls on anything other than filename is very suboptimal; in #14731 you can see where the filtering has been postoned until after the listing, when the full

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2016-08-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r75584026 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -293,8 +290,8 @@ class FileInputDStream[K, V, F

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2016-08-20 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r75584030 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -241,16 +233,21 @@ class FileInputDStream[K, V, F

[GitHub] spark issue #14732: [SPARK-16320] [DOC] Document G1 heap region's effect on ...

2016-08-20 Thread petermaxlee
Github user petermaxlee commented on the issue: https://github.com/apache/spark/pull/14732 Looks good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2016-08-20 Thread petermaxlee
Github user petermaxlee commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r75583457 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -293,8 +290,8 @@ class FileInputDStream[K, V, F <:

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/14735 This seems a big enough change that it might be good to have a JIRA for this ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2016-08-20 Thread petermaxlee
Github user petermaxlee commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r75583446 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -241,16 +233,21 @@ class FileInputDStream[K, V, F

[GitHub] spark pull request #14731: [SPARK-17159] [streaming]: optimise check for new...

2016-08-20 Thread petermaxlee
Github user petermaxlee commented on a diff in the pull request: https://github.com/apache/spark/pull/14731#discussion_r75583436 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -241,16 +233,21 @@ class FileInputDStream[K, V, F

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14719 **[Test build #64155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64155/consoleFull)** for PR 14719 at commit

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14735 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14735 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64154/ Test FAILed. ---

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14735 **[Test build #64154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64154/consoleFull)** for PR 14735 at commit

[GitHub] spark issue #14155: [SPARK-16498][SQL] move hive hack for data source table ...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14155 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64149/ Test PASSed. ---

[GitHub] spark issue #14155: [SPARK-16498][SQL] move hive hack for data source table ...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14155 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14155: [SPARK-16498][SQL] move hive hack for data source table ...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14155 **[Test build #64149 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64149/consoleFull)** for PR 14155 at commit

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14734 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64153/ Test PASSed. ---

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14734 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14734 **[Test build #64153 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64153/consoleFull)** for PR 14734 at commit

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14734 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14734 **[Test build #64152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64152/consoleFull)** for PR 14734 at commit

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14734 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64152/ Test PASSed. ---

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14735 **[Test build #64151 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64151/consoleFull)** for PR 14735 at commit

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14735 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64151/ Test FAILed. ---

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14735 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14735: [MINOR][SPARKR] R MLlib refactor, cleanup, reformat, fix...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14735 **[Test build #64154 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64154/consoleFull)** for PR 14735 at commit

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14719 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14719 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64148/ Test FAILed. ---

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14719 **[Test build #64148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64148/consoleFull)** for PR 14719 at commit

[GitHub] spark issue #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14734 **[Test build #64153 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64153/consoleFull)** for PR 14734 at commit

[GitHub] spark pull request #14734: [SPARK-16508][SPARKR] small doc updates

2016-08-20 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/14734#discussion_r75581656 --- Diff: R/pkg/R/DataFrame.R --- @@ -2880,7 +2880,7 @@ setMethod("fillna", #' #' @param x a SparkDataFrame. #' @param row.names NULL or a

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14719 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64146/ Test FAILed. ---

[GitHub] spark issue #14719: [SPARK-17154][SQL] Wrong result can be returned or Analy...

2016-08-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14719 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   3   >