[GitHub] spark issue #19861: [SPARK-22387][SQL] Propagate session configs to data sou...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19861 **[Test build #84433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84433/testReport)** for PR 19861 at commit

[GitHub] spark issue #19865: [SPARK-22668][SQL] Do not pass global variables to argum...

2017-12-04 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19865 Thank you. I think so for this case. In general, to make global `ev.value` or to pass a global variable to `consume()` may potentially cause this problem. As @cloud-fan suggested, I

[GitHub] spark issue #19878: [SPARK-22682][SQL] HashExpression does not need to creat...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19878 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84431/ Test PASSed. ---

[GitHub] spark issue #19881: [SPARK-22683][CORE] Add tasksPerExecutorSlot parameter

2017-12-04 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19881 Please see JIRA. I don't think this is worth doing. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19878: [SPARK-22682][SQL] HashExpression does not need to creat...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19878 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19878: [SPARK-22682][SQL] HashExpression does not need to creat...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19878 **[Test build #84431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84431/testReport)** for PR 19878 at commit

[GitHub] spark issue #19881: [SPARK-22683][CORE] Add tasksPerExecutorSlot parameter

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19881 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19877: [SPARK-22681]Accumulator should only be updated o...

2017-12-04 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19877#discussion_r154729236 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -1832,6 +1832,27 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #19881: [SPARK-22683][CORE] Add tasksPerExecutorSlot para...

2017-12-04 Thread jcuquemelle
GitHub user jcuquemelle opened a pull request: https://github.com/apache/spark/pull/19881 [SPARK-22683][CORE] Add tasksPerExecutorSlot parameter ## What changes were proposed in this pull request? let's say an executor has spark.executor.cores / spark.task.cpus taskSlots

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19717 **[Test build #84438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84438/testReport)** for PR 19717 at commit

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19717 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19717 **[Test build #84437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84437/testReport)** for PR 19717 at commit

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19717 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84437/ Test FAILed. ---

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19717 **[Test build #84437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84437/testReport)** for PR 19717 at commit

[GitHub] spark issue #19880: [SPARK-22626][SQL][FOLLOWUP] improve documentation and s...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19880 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84434/ Test PASSed. ---

[GitHub] spark issue #19880: [SPARK-22626][SQL][FOLLOWUP] improve documentation and s...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19880 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19880: [SPARK-22626][SQL][FOLLOWUP] improve documentation and s...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19880 **[Test build #84434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84434/testReport)** for PR 19880 at commit

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154726920 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile --- @@ -0,0 +1,43 @@ +# +# Licensed to the Apache

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-04 Thread rezasafi
Github user rezasafi commented on the issue: https://github.com/apache/spark/pull/19848 Thank you very much @vanzin, @mridulm and @jiangxb1987. I really appreciate it. I will create PR for branch 2.2 ASAP. --- -

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154724747 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile --- @@ -0,0 +1,43 @@ +# +# Licensed to the Apache

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154721032 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala --- @@ -0,0 +1,182 @@

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on execut...

2017-12-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19840 I'm trying to understand what is https://github.com/apache/spark/blob/master/python/pyspark/context.py#L191 really achieving. It seems pretty broken to me and feels like the whole `pythonExec`

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154719522 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile --- @@ -0,0 +1,43 @@ +# +# Licensed to the Apache

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19870 To ensure no external behavior change, we should use RoundRobinPartitioning when the expression list is empty, instead of issuing an exception. It will be also consistent with the existing

[GitHub] spark pull request #19848: [SPARK-22162] Executors and the driver should use...

2017-12-04 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19848 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19848 (There was a conflict in 2.2, open a new PR if you want it there.) --- - To unsubscribe, e-mail:

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-04 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19848 Merging to master / 2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19880: [SPARK-22626][SQL][FOLLOWUP] improve documentation and s...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19880 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19873: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19873 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19873: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19873 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84430/ Test PASSed. ---

[GitHub] spark issue #19879: [SPARK-20706][SPARK-SHELL] Spark-shell not overriding me...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19879 **[Test build #4003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4003/testReport)** for PR 19879 at commit

[GitHub] spark issue #19873: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19873 **[Test build #84430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84430/testReport)** for PR 19873 at commit

[GitHub] spark issue #19879: [SPARK-20706][SPARK-SHELL] Spark-shell not overriding me...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19879 **[Test build #4003 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4003/testReport)** for PR 19879 at commit

[GitHub] spark issue #19879: [SPARK-20706][SPARK-SHELL] Spark-shell not overriding me...

2017-12-04 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19879 OK, so another back-port to fix the 2.11 REPL, which should be resolved in 2.12 anyway? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19875: [SPARK-22473][FOLLOWUP][TEST] Remove deprecated D...

2017-12-04 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19875 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19854: SPARK-22660:Use position() and limit() to fix ambiguity ...

2017-12-04 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19854 Ping @kellyzly --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19871: [SPARK-20728][SQL] Make OrcFileFormat configurable betwe...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19871 **[Test build #84436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84436/testReport)** for PR 19871 at commit

[GitHub] spark issue #19875: [SPARK-22473][FOLLOWUP][TEST] Remove deprecated Date fun...

2017-12-04 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19875 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19871: [SPARK-20728][SQL] Make OrcFileFormat configurable betwe...

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19871 @HyukjinKwon , for enabling the following test, I'm restructuring ORC tests now. I'll make a PR today for that. ``` ignore("LZO compression options for writing to an ORC file not

[GitHub] spark issue #10621: [SPARK-12617][PySpark]Move Py4jCallbackConnectionCleaner...

2017-12-04 Thread cpalomaressbd
Github user cpalomaressbd commented on the issue: https://github.com/apache/spark/pull/10621 Hi, One question, maybe is stupid question I dont know and I ask sorry in advance. We are workin with hortonworks, in the beginning of the project we were working with HDP 2.4.0

[GitHub] spark pull request #19871: [SPARK-20728][SQL] Make OrcFileFormat configurabl...

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19871#discussion_r154709256 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -568,8 +574,12 @@ object DataSource extends

[GitHub] spark issue #19869: [SPARK-22677][SQL] cleanup whole stage codegen for hash ...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19869 **[Test build #84435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84435/testReport)** for PR 19869 at commit

[GitHub] spark pull request #19871: [SPARK-20728][SQL] Make OrcFileFormat configurabl...

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19871#discussion_r154707683 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -85,7 +87,8 @@ case class DataSource(

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154707798 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to

[GitHub] spark pull request #19871: [SPARK-20728][SQL] Make OrcFileFormat configurabl...

2017-12-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19871#discussion_r154707557 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -363,6 +363,14 @@ object SQLConf {

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154707001 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -119,5 +139,60 @@ private[spark] object

[GitHub] spark pull request #19870: [SPARK-22665][SQL] Avoid repartitioning with empt...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19870#discussion_r154705369 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala --- @@ -838,6 +838,8 @@ case class

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19870 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19870 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84428/ Test PASSed. ---

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19870 **[Test build #84428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84428/testReport)** for PR 19870 at commit

[GitHub] spark issue #19869: [SPARK-22677][SQL] cleanup whole stage codegen for hash ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19869 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19869: [SPARK-22677][SQL] cleanup whole stage codegen for hash ...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19869 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84432/ Test FAILed. ---

[GitHub] spark issue #19869: [SPARK-22677][SQL] cleanup whole stage codegen for hash ...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19869 **[Test build #84432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84432/testReport)** for PR 19869 at commit

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

2017-12-04 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/19717#discussion_r154694277 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/DriverServiceBootstrapStep.scala --- @@ -0,0 +1,103

[GitHub] spark pull request #19861: [SPARK-22387][SQL] Propagate session configs to d...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19861#discussion_r154694199 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/ConfigSupport.java --- @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #19874: [SPARK-22675] [SQL] Refactoring PropagateTypes in TypeCo...

2017-12-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19874 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19861: [SPARK-22387][SQL] Propagate session configs to d...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19861#discussion_r154692737 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -184,9 +188,16 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154691937 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -389,13 +408,21 @@ abstract class HashExpression[E]

[GitHub] spark issue #19880: [SPARK-22626][SQL][FOLLOWUP] improve documentation and s...

2017-12-04 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19880 cc @cloud-fan @wangyum --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19880: [SPARK-22626][SQL][FOLLOWUP] improve documentation and s...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19880 **[Test build #84434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84434/testReport)** for PR 19880 at commit

[GitHub] spark pull request #19880: [SPARK-22626][SQL][FOLLOWUP] improve documentatio...

2017-12-04 Thread wzhfy
GitHub user wzhfy opened a pull request: https://github.com/apache/spark/pull/19880 [SPARK-22626][SQL][FOLLOWUP] improve documentation and simplify test case ## What changes were proposed in this pull request? The reason why some Hive tables have `numRows` statistics is

[GitHub] spark issue #19861: [SPARK-22387][SQL] Propagate session configs to data sou...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19861 **[Test build #84433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84433/testReport)** for PR 19861 at commit

[GitHub] spark issue #19877: [SPARK-22681]Accumulator should only be updated once for...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19877 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19877: [SPARK-22681]Accumulator should only be updated once for...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84427/ Test PASSed. ---

[GitHub] spark issue #19877: [SPARK-22681]Accumulator should only be updated once for...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19877 **[Test build #84427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84427/testReport)** for PR 19877 at commit

[GitHub] spark issue #19878: [SPARK-22682][SQL] HashExpression does not need to creat...

2017-12-04 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19878 left only one very minor comment, it LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19861: [SPARK-22387][SQL] Propagate session configs to d...

2017-12-04 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/19861#discussion_r154687413 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -184,9 +188,16 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154687417 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -389,13 +408,21 @@ abstract class HashExpression[E]

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154683889 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -730,23 +776,29 @@ case class HiveHash(children:

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154683716 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -270,17 +270,36 @@ abstract class HashExpression[E]

[GitHub] spark issue #19869: [SPARK-22677][SQL] cleanup whole stage codegen for hash ...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19869 **[Test build #84432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84432/testReport)** for PR 19869 at commit

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19878#discussion_r154682872 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -270,17 +270,36 @@ abstract class HashExpression[E]

[GitHub] spark pull request #19869: [SPARK-22677][SQL] cleanup whole stage codegen fo...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19869#discussion_r154682180 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala --- @@ -596,7 +596,7 @@ case class HashAggregateExec(

[GitHub] spark issue #19878: [SPARK-22682][SQL] HashExpression does not need to creat...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19878 **[Test build #84431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84431/testReport)** for PR 19878 at commit

[GitHub] spark issue #19879: [SPARK-20706][SPARK-SHELL] Spark-shell not overriding me...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19879 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19865: [SPARK-22668][SQL] Do not pass global variables to argum...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19865 hopefully https://github.com/apache/spark/pull/19878 can fix the problem. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19879: [SPARK-20706][SPARK-SHELL] Spark-shell not overri...

2017-12-04 Thread mpetruska
GitHub user mpetruska opened a pull request: https://github.com/apache/spark/pull/19879 [SPARK-20706][SPARK-SHELL] Spark-shell not overriding method/variable definition ## What changes were proposed in this pull request?

[GitHub] spark issue #19878: [SPARK-22682][SQL] HashExpression does not need to creat...

2017-12-04 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19878 cc @kiszk @mgaido91 @viirya @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19878: [SPARK-22682][SQL] HashExpression does not need t...

2017-12-04 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/19878 [SPARK-22682][SQL] HashExpression does not need to create global variables ## What changes were proposed in this pull request? It turns out that `HashExpression` can pass around some

[GitHub] spark issue #19875: [SPARK-22473][FOLLOWUP][TEST] Remove deprecated Date fun...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19875 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19875: [SPARK-22473][FOLLOWUP][TEST] Remove deprecated Date fun...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19875 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84425/ Test PASSed. ---

[GitHub] spark issue #19875: [SPARK-22473][FOLLOWUP][TEST] Remove deprecated Date fun...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19875 **[Test build #84425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84425/testReport)** for PR 19875 at commit

[GitHub] spark issue #19873: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19873 **[Test build #84430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84430/testReport)** for PR 19873 at commit

[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84429/ Test PASSed. ---

[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...

2017-12-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84429/testReport)** for PR 19792 at commit

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19789 I think we should wait for @zsxwing's feedback. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84429/testReport)** for PR 19792 at commit

[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...

2017-12-04 Thread gberger
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 Fixed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19870: [SPARK-22665][SQL] Avoid repartitioning with empty list ...

2017-12-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19870 **[Test build #84428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84428/testReport)** for PR 19870 at commit

[GitHub] spark issue #18424: [SPARK-17091] Add rule to convert IN predicate to equiva...

2017-12-04 Thread ptkool
Github user ptkool commented on the issue: https://github.com/apache/spark/pull/18424 @a10y Yes, I'm still tracking this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19875: [SPARK-22473][FOLLOWUP][TEST] Remove deprecated Date fun...

2017-12-04 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/19875 yes @HyukjinKwon , you are 100% right, sorry for this error. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r154644084 --- Diff: python/pyspark/sql/udf.py --- @@ -56,6 +56,10 @@ def _create_udf(f, returnType, evalType): return udf_obj._wrapped()

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r154642230 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AggregateInPandasExec.scala --- @@ -0,0 +1,135 @@ +/* + * Licensed to

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r154642902 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/pythonLogicalOperators.scala --- @@ -38,3 +38,13 @@ case class

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r154644620 --- Diff: python/pyspark/sql/tests.py --- @@ -4016,6 +4016,89 @@ def test_unsupported_types(self): with

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r154644235 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala --- @@ -437,6 +437,37 @@ class RelationalGroupedDataset

[GitHub] spark pull request #19872: WIP: [SPARK-22274][PySpark] User-defined aggregat...

2017-12-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r154644340 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala --- @@ -437,6 +437,37 @@ class RelationalGroupedDataset

[GitHub] spark issue #895: [SPARK-1940] Enabling rolling of executor logs, and automa...

2017-12-04 Thread wbowditch
Github user wbowditch commented on the issue: https://github.com/apache/spark/pull/895 Can these configuration additions be added to Spark Documentation (https://spark.apache.org/docs/latest/configuration.html) ? ---

[GitHub] spark pull request #19871: [SPARK-20728][SQL] Make OrcFileFormat configurabl...

2017-12-04 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/19871#discussion_r154640805 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -568,8 +574,12 @@ object DataSource extends

<    1   2   3   4   5   >