[GitHub] spark issue #19113: [SPARK-20978][SQL] Bump up Univocity version to 2.5.4

2017-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19113 Since the expected release of our next version Spark 2.3 is the end of this year, we still can revert it back to 2.2.1 if we realize this release 2.5.4 introduces new bugs or performance regressi

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19116 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81397/ Test FAILed. ---

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19116 **[Test build #81397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81397/testReport)** for PR 19116 at commit [`db0dee5`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #19127: [SPARK-21916][SQL] Set isolationOn=true when create hive...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19127 **[Test build #81396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81396/testReport)** for PR 19127 at commit [`2d13ab8`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #19127: [SPARK-21916][SQL] Set isolationOn=true when create hive...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19127 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19116 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19127: [SPARK-21916][SQL] Set isolationOn=true when create hive...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19127 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81396/ Test FAILed. ---

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19116 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #18576: [SPARK-21351][SQL] Update nullability based on children'...

2017-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18576 Changing the usage/semantics of `output` is risky. It could easily break the existing codes without notice, especially when you do not completely remove `output` from `QueryPlan`. Addin

[GitHub] spark issue #19119: [SPARK-21845] [SQL] [test-maven] Make codegen fallback o...

2017-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19119 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #19119: [SPARK-21845] [SQL] [test-maven] Make codegen fallback o...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19119 **[Test build #81398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81398/testReport)** for PR 19119 at commit [`b96da49`](https://github.com/apache/spark/commit/b9

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19116 **[Test build #81399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81399/testReport)** for PR 19116 at commit [`db0dee5`](https://github.com/apache/spark/commit/db

[GitHub] spark issue #18576: [SPARK-21351][SQL] Update nullability based on children'...

2017-09-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18576 ok, will look this around based on the suggestion. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addition

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136910147 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala --- @@ -169,6 +172,18 @@ class OrcFileFormat extends FileFormat with

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136910368 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala --- @@ -2000,4 +2000,15 @@ class SQLQuerySuite extends QueryTe

[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...

2017-09-05 Thread yaooqinn
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/19121 @jerryshao thanks for replying if a user explicitly set `SPARK_USER` but not wrapping the `SPARK_USER` UGI doAs for SparkContext initialization, there might be two different users commu

[GitHub] spark issue #19125: [SPARK-21913][SQL][TEST] `withDatabase` should drop data...

2017-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19125 LGTM Thanks! Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional com

[GitHub] spark issue #19127: [SPARK-21916][SQL] Set isolationOn=true when create hive...

2017-09-05 Thread yaooqinn
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/19127 hey i am working on this plz check https://github.com/apache/spark/pull/19068 --- - To unsubscribe, e-mail: reviews-unsubscr...

[GitHub] spark pull request #19125: [SPARK-21913][SQL][TEST] `withDatabase` should dr...

2017-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19125 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...

2017-09-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19060#discussion_r136912257 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...

2017-09-05 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19121 No, I don't agree with you. SPARK_USER is set in SparkContext with driver's current UGI and this env variable will be propagated to executors to create executor's UGI with the same user i

[GitHub] spark issue #19125: [SPARK-21913][SQL][TEST] `withDatabase` should drop data...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19125 Thank you for reviewing and merging, @gatorsmile ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For a

[GitHub] spark issue #19082: [SPARK-21870][SQL] Split aggregation code into small fun...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19082 **[Test build #81400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81400/testReport)** for PR 19082 at commit [`6c502a9`](https://github.com/apache/spark/commit/6c

[GitHub] spark pull request #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cache...

2017-09-05 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/19129 [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadata from SQLConf and docs ## What changes were proposed in this pull request? `spark.sql.parquet.cacheMetadata` is not used anym

[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81401/testReport)** for PR 19129 at commit [`3b305d0`](https://github.com/apache/spark/commit/3b

[GitHub] spark pull request #19122: [SPARK-21911][ML][PySpark] Parallel Model Evaluat...

2017-09-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19122#discussion_r136915493 --- Diff: python/pyspark/ml/tuning.py --- @@ -255,18 +257,23 @@ def _fit(self, dataset): randCol = self.uid + "_rand" df = dataset.s

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136915649 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala --- @@ -169,6 +172,18 @@ class OrcFileFormat extends FileFormat wi

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136916230 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala --- @@ -169,6 +172,18 @@ class OrcFileFormat extends FileFormat wi

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136916250 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala --- @@ -2000,4 +2000,15 @@ class SQLQuerySuite extends Quer

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136918576 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala --- @@ -2000,4 +2000,15 @@ class SQLQuerySuite extends Quer

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136919173 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala --- @@ -2000,4 +2000,15 @@ class SQLQuerySuite extends Quer

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136921664 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala --- @@ -2000,4 +2000,15 @@ class SQLQuerySuite extends Quer

[GitHub] spark pull request #19130: [SPARK-21917][CORE][YARN] Supporting Download htt...

2017-09-05 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/19130 [SPARK-21917][CORE][YARN] Supporting Download http(s) resources in yarn mode ## What changes were proposed in this pull request? In the current Spark, when submitting application on YARN with

[GitHub] spark issue #19130: [SPARK-21917][CORE][YARN] Supporting Download http(s) re...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19130 **[Test build #81402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81402/testReport)** for PR 19130 at commit [`42a79ab`](https://github.com/apache/spark/commit/42

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC datasource table ...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136923762 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala --- @@ -2000,4 +2000,15 @@ class SQLQuerySuite extends Quer

[GitHub] spark pull request #19122: [SPARK-21911][ML][PySpark] Parallel Model Evaluat...

2017-09-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19122#discussion_r136924008 --- Diff: python/pyspark/ml/tuning.py --- @@ -255,18 +257,24 @@ def _fit(self, dataset): randCol = self.uid + "_rand" df = dataset.s

[GitHub] spark pull request #19131: [MINOR][SQL]remove unuse import class

2017-09-05 Thread heary-cao
GitHub user heary-cao opened a pull request: https://github.com/apache/spark/pull/19131 [MINOR][SQL]remove unuse import class ## What changes were proposed in this pull request? this PR describe remove the import class that are unused. ## How was this patch tested?

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasourc...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136926003 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala --- @@ -85,6 +87,13 @@ case class CreateDataSou

[GitHub] spark pull request #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasourc...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19124#discussion_r136926193 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala --- @@ -0,0 +1,36 @@ +/* + * Licensed to

[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19131 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #16774: [SPARK-19357][ML] Adding parallel model evaluatio...

2017-09-05 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/16774#discussion_r136926433 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala --- @@ -100,31 +113,53 @@ class CrossValidator @Since("1.2.0") (@Since("1.4.0"

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19124 **[Test build #81403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81403/testReport)** for PR 19124 at commit [`cd539fe`](https://github.com/apache/spark/commit/cd

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19124 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19124 **[Test build #81403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81403/testReport)** for PR 19124 at commit [`cd539fe`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19124 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81403/ Test FAILed. ---

[GitHub] spark pull request #16774: [SPARK-19357][ML] Adding parallel model evaluatio...

2017-09-05 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/16774#discussion_r136928086 --- Diff: mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala --- @@ -120,6 +120,33 @@ class CrossValidatorSuite } }

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19124 **[Test build #81404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81404/testReport)** for PR 19124 at commit [`66aff54`](https://github.com/apache/spark/commit/66

[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class

2017-09-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19131 On the one hand, we don't normally just make a change to remove unused imports. But it's a good idea once in a while to clean up. Would you actually try to find all of them? at least, in several whol

[GitHub] spark pull request #16774: [SPARK-19357][ML] Adding parallel model evaluatio...

2017-09-05 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/16774#discussion_r136930082 --- Diff: mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala --- @@ -120,6 +120,33 @@ class CrossValidatorSuite } }

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19124 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81404/ Test FAILed. ---

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19124 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19124 **[Test build #81404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81404/testReport)** for PR 19124 at commit [`66aff54`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #19113: [SPARK-20978][SQL] Bump up Univocity version to 2.5.4

2017-09-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19113 If we need 2.5.x for the fix, then we need 2.5.x. It's worth picking up an update if it solves a real problem. And if we're going to update minor versions, it's generally good practice to pick the la

[GitHub] spark issue #18519: [SPARK-16742] Mesos Kerberos Support

2017-09-05 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18519 @ArtRand @vanzin , is it only worked in client deploy mode, am I understanding correctly? I don't see a code to ship tokens from local client to remote driver. --- -

[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19060#discussion_r136932032 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Soft

[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...

2017-09-05 Thread yaooqinn
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/19121 @jerryshao Yes, you are right. unlike yarn standalone don‘t switch the jvm process user. but you said that if a user explicitly set `SPARK_USER`, there is no mechanism in spark to set this user

[GitHub] spark pull request #19122: [SPARK-21911][ML][PySpark] Parallel Model Evaluat...

2017-09-05 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/19122#discussion_r136933807 --- Diff: python/pyspark/ml/tuning.py --- @@ -255,18 +257,23 @@ def _fit(self, dataset): randCol = self.uid + "_rand" df = dat

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19124 **[Test build #81405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81405/testReport)** for PR 19124 at commit [`aa78eaf`](https://github.com/apache/spark/commit/aa

[GitHub] spark pull request #19122: [SPARK-21911][ML][PySpark] Parallel Model Evaluat...

2017-09-05 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/19122#discussion_r136934638 --- Diff: python/pyspark/ml/tuning.py --- @@ -255,18 +257,24 @@ def _fit(self, dataset): randCol = self.uid + "_rand" df = dat

[GitHub] spark issue #19112: [SPARK-21901][SS] Define toString for StateOperatorProgr...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19112 **[Test build #81406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81406/testReport)** for PR 19112 at commit [`0e2b9c6`](https://github.com/apache/spark/commit/0e

[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...

2017-09-05 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19121 Sorry I didn't clearly say the problem. But IMO the changes you made is really not so necessary. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...

2017-09-05 Thread yaooqinn
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/19121 em..this may also not work for `DefaultContainerExecutor` which launch containers via admin. might close this pr --- - To unsub

[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class

2017-09-05 Thread heary-cao
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/19131 ok, i will do it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUs...

2017-09-05 Thread yaooqinn
Github user yaooqinn closed the pull request at: https://github.com/apache/spark/pull/19121 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19112: [SPARK-21901][SS] Define toString for StateOperatorProgr...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19112 **[Test build #81407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81407/testReport)** for PR 19112 at commit [`673e088`](https://github.com/apache/spark/commit/67

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19124 **[Test build #81405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81405/testReport)** for PR 19124 at commit [`aa78eaf`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19124 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81405/ Test FAILed. ---

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19124 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...

2017-09-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/19129 I roughly checked other options around parquet and I probably found `parquetOutputCommitterClass` in `SQLConf` also is not used now? --- ---

[GitHub] spark issue #19082: [SPARK-21870][SQL] Split aggregation code into small fun...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19082 **[Test build #81400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81400/testReport)** for PR 19082 at commit [`6c502a9`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #19082: [SPARK-21870][SQL] Split aggregation code into small fun...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19082 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81400/ Test PASSed. ---

[GitHub] spark issue #19082: [SPARK-21870][SQL] Split aggregation code into small fun...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19082 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81401/testReport)** for PR 19129 at commit [`3b305d0`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81401/ Test PASSed. ---

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19116 **[Test build #81399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81399/testReport)** for PR 19116 at commit [`db0dee5`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19127: [SPARK-21916][SQL] Set isolationOn=true when create hive...

2017-09-05 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19127 Sure, I will close this then. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark pull request #19127: [SPARK-21916][SQL] Set isolationOn=true when crea...

2017-09-05 Thread jinxing64
Github user jinxing64 closed the pull request at: https://github.com/apache/spark/pull/19127 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19116 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19116 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81399/ Test PASSed. ---

[GitHub] spark issue #19119: [SPARK-21845] [SQL] [test-maven] Make codegen fallback o...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19119 **[Test build #81398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81398/testReport)** for PR 19119 at commit [`b96da49`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19119: [SPARK-21845] [SQL] [test-maven] Make codegen fallback o...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81398/ Test PASSed. ---

[GitHub] spark issue #19119: [SPARK-21845] [SQL] [test-maven] Make codegen fallback o...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19119 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19116 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19116 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19132: [SPARK-21922] Fix duration always updating when t...

2017-09-05 Thread caneGuy
GitHub user caneGuy opened a pull request: https://github.com/apache/spark/pull/19132 [SPARK-21922] Fix duration always updating when task failed but status is still RUN… …NING ## What changes were proposed in this pull request? When executor failed and task metrics

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19124 Hi, @gatorsmile . I can fix it in most cases, but we have the [following test case](https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-tests/results/show_colum

[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19133: [SPARK-21902] Uniform calling for DiskBlockManage...

2017-09-05 Thread caneGuy
GitHub user caneGuy opened a pull request: https://github.com/apache/spark/pull/19133 [SPARK-21902] Uniform calling for DiskBlockManager.getFile and print root cause for doPut ## What changes were proposed in this pull request? As logging below, actually exception will be h

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19124 **[Test build #81408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81408/testReport)** for PR 19124 at commit [`0bf3b43`](https://github.com/apache/spark/commit/0b

[GitHub] spark issue #19124: [SPARK-21912][SQL] Creating ORC/Parquet datasource table...

2017-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19124 I just updated the output answer file to show the result for review only. --- - To unsubscribe, e-mail: reviews-unsubscr...

[GitHub] spark issue #19133: [SPARK-21902] Uniform calling for DiskBlockManager.getFi...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19133 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19116: [SPARK-21903][BUILD] Upgrade scalastyle to 1.0.0.

2017-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19116 Thank you @srowen and @viirya for reviewing this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addi

[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...

2017-09-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/19129 oh, yea. I got you. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: re

[GitHub] spark pull request #19112: [SPARK-21901][SS] Define toString for StateOperat...

2017-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19112#discussion_r136957034 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala --- @@ -200,7 +202,7 @@ class SourceProgress protected[sql]( */

[GitHub] spark pull request #19134: [SPARK-21893][BUILD][STREAMING] Put Kafka 0.8 beh...

2017-09-05 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/19134 [SPARK-21893][BUILD][STREAMING] Put Kafka 0.8 behind a profile ## What changes were proposed in this pull request? Put Kafka 0.8 support behind a kafka-0-8 profile. ## How was this

[GitHub] spark issue #19130: [SPARK-21917][CORE][YARN] Supporting adding http(s) reso...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19130 **[Test build #81402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81402/testReport)** for PR 19130 at commit [`42a79ab`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...

2017-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19134 **[Test build #81409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81409/testReport)** for PR 19134 at commit [`6980c6a`](https://github.com/apache/spark/commit/69

[GitHub] spark issue #19130: [SPARK-21917][CORE][YARN] Supporting adding http(s) reso...

2017-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19130 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81402/ Test FAILed. ---

[GitHub] spark pull request #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0....

2017-09-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/19134#discussion_r136960925 --- Diff: dev/create-release/release-build.sh --- @@ -80,8 +80,17 @@ NEXUS_PROFILE=d63f592e7eac0 # Profile for Spark staging uploads BASE_DIR=$(pwd)

  1   2   3   4   5   6   >