[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-03 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r214735437 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -754,6 +754,47 @@ class HiveDDLSuite }

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22320 **[Test build #95633 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95633/testReport)** for PR 22320 at commit

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread peter-toth
Github user peter-toth commented on the issue: https://github.com/apache/spark/pull/22318 Also added missing `if attr.resolved` which I think will fix the UT issues. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22318 **[Test build #95632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95632/testReport)** for PR 22318 at commit

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread peter-toth
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214732767 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -817,7 +819,7 @@ class Analyzer(

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread peter-toth
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214732751 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -805,10 +807,10 @@ class Analyzer( *

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread peter-toth
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214732731 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -921,12 +930,16 @@ class Analyzer(

[GitHub] spark issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22316 **[Test build #95631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95631/testReport)** for PR 22316 at commit

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22318 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95625/ Test FAILed. ---

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22318 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22318 **[Test build #95625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95625/testReport)** for PR 22318 at commit

[GitHub] spark pull request #22316: [SPARK-25048][SQL] Pivoting by multiple columns i...

2018-09-03 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/22316#discussion_r214722485 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFramePivotSuite.scala --- @@ -308,4 +308,27 @@ class DataFramePivotSuite extends QueryTest with

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22320 LGTM except some minor comments --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r214722461 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -754,6 +754,47 @@ class HiveDDLSuite } }

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r214722030 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala --- @@ -805,6 +805,81 @@ class DataFrameReaderWriterSuite

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r214721624 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -24,6 +24,7 @@ import java.util.concurrent.atomic.AtomicBoolean

[GitHub] spark issue #21669: [SPARK-23257][K8S][WIP] Kerberos Support for Spark on K8...

2018-09-03 Thread suryag10
Github user suryag10 commented on the issue: https://github.com/apache/spark/pull/21669 Hi Ilan, I was able to make work the Kerberos with one work around(which I am trying to do a full fix) and one fix. Fix is the one which i had commented earlier and is as follows:

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95630/testReport)** for PR 22319 at commit

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request #22288: [SPARK-22148][Scheduler] Acquire new executors to...

2018-09-03 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r214720097 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -414,9 +425,54 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request #22288: [SPARK-22148][Scheduler] Acquire new executors to...

2018-09-03 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r214719743 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -414,9 +425,54 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22317 **[Test build #95629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95629/testReport)** for PR 22317 at commit

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22317 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22317 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18906 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95624/ Test FAILed. ---

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18906 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18906 **[Test build #95624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95624/testReport)** for PR 18906 at commit

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22319 I prefer the current way as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-03 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r214697039 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala --- @@ -53,3 +57,21 @@ trait DataWritingCommand

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22320 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22263 **[Test build #95628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95628/testReport)** for PR 22263 at commit

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22320 **[Test build #95627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95627/testReport)** for PR 22320 at commit

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22263 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-03 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r214694881 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -69,7 +69,7 @@ case class

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22263 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95626 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95626/testReport)** for PR 22319 at commit

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95626/ Test FAILed. ---

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22179 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22179 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22306: [SPARK-25300][CORE]Unified the configuration parameter `...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22306 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #17745: [SPARK-17159][Streaming] optimise check for new files in...

2018-09-03 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17745 Patch is in the spark cloud integration module, you can take it and try to get into ASF spark provided you also add some credit to me in the patch. ---

[GitHub] spark issue #22306: [SPARK-25300][CORE]Unified the configuration parameter `...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22306 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22320 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22319 It's too hard to work around the case class compatibility issue. I'm leaving it unchanged, and add private mutable variables to store the nullable info. ---

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22320 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22320 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22314: [SPARK-25307][SQL] ArraySort function may return an erro...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22314 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22306: [SPARK-25300][CORE]Unified the configuration parameter `...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22306 **[Test build #95621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95621/testReport)** for PR 22306 at commit

[GitHub] spark issue #22314: [SPARK-25307][SQL] ArraySort function may return an erro...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22314 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22320 **[Test build #95620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95620/testReport)** for PR 22320 at commit

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95626/testReport)** for PR 22319 at commit

[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/6 **[Test build #95622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95622/testReport)** for PR 6 at commit

[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22179 **[Test build #95623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95623/testReport)** for PR 22179 at commit

[GitHub] spark issue #22218: [SPARK-25228][CORE]Add executor CPU time metric.

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22218 **[Test build #4330 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4330/testReport)** for PR 22218 at commit

[GitHub] spark issue #21860: [SPARK-24901][SQL]Merge the codegen of RegularHashMap an...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21860 **[Test build #95618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95618/testReport)** for PR 21860 at commit

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22320 **[Test build #95619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95619/testReport)** for PR 22320 at commit

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22318 **[Test build #95625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95625/testReport)** for PR 22318 at commit

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18906 **[Test build #95624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95624/testReport)** for PR 18906 at commit

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214692193 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -805,10 +807,10 @@ class Analyzer( *

[GitHub] spark pull request #21968: [SPARK-24999][SQL]Reduce unnecessary 'new' memory...

2018-09-03 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21968#discussion_r214691997 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/RowBasedHashMapGenerator.scala --- @@ -48,6 +48,8 @@ class

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214677137 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -805,10 +807,10 @@ class Analyzer( *

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214678100 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala --- @@ -26,6 +26,8 @@ object AttributeMap { def

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214690245 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -805,10 +807,10 @@ class Analyzer( *

[GitHub] spark issue #22232: [SPARK-25237][SQL]remove updateBytesReadWithFileSize bec...

2018-09-03 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22232 I think that's fine @maropu ; we can always apportion credit appropriately later. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214689283 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -817,7 +819,7 @@ class Analyzer( case

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214689183 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -805,10 +807,10 @@ class Analyzer( *

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214688902 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -921,12 +930,16 @@ class Analyzer(

[GitHub] spark issue #22232: [SPARK-25237][SQL]remove updateBytesReadWithFileSize bec...

2018-09-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22232 @srowen I could take this over or do follow-up if the author is still inactive. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22318 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22232: [SPARK-25237][SQL]remove updateBytesReadWithFileSize bec...

2018-09-03 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22232 @dujunling I personally think this can be merged, but only if the test is tightened up --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21638: [SPARK-22357][CORE] SparkContext.binaryFiles igno...

2018-09-03 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21638#discussion_r214685953 --- Diff: core/src/main/scala/org/apache/spark/input/PortableDataStream.scala --- @@ -47,7 +47,7 @@ private[spark] abstract class StreamFileInputFormat[T]

[GitHub] spark issue #21669: [SPARK-23257][K8S][WIP] Kerberos Support for Spark on K8...

2018-09-03 Thread rvesse
Github user rvesse commented on the issue: https://github.com/apache/spark/pull/21669 @ifilonenko I think the issue with the `UnixUsername` might possibly be avoided by exporting `HADOOP_USER_NAME` as an environment variable in the pod spec set to the same value as `SPARK_USER`.

[GitHub] spark pull request #22259: [SPARK-25044][SQL] (take 2) Address translation o...

2018-09-03 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22259#discussion_r214683934 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala --- @@ -47,7 +48,8 @@ case class ScalaUDF(

[GitHub] spark issue #22321: [DOC] Update the 'Specifying the Hadoop Version' link in...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22321 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95615/ Test PASSed. ---

[GitHub] spark issue #22321: [DOC] Update the 'Specifying the Hadoop Version' link in...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22321 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22314: [SPARK-25307][SQL] ArraySort function may return an erro...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22314 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22314: [SPARK-25307][SQL] ArraySort function may return an erro...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22314 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95617/ Test PASSed. ---

[GitHub] spark issue #22314: [SPARK-25307][SQL] ArraySort function may return an erro...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22314 **[Test build #95617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95617/testReport)** for PR 22314 at commit

[GitHub] spark issue #22321: [DOC] Update the 'Specifying the Hadoop Version' link in...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22321 **[Test build #95615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95615/testReport)** for PR 22321 at commit

[GitHub] spark issue #21743: [SPARK-24767][Launcher] Propagate MDC to spark-submit th...

2018-09-03 Thread yifeih
Github user yifeih commented on the issue: https://github.com/apache/spark/pull/21743 Ah ok i see, let's close this out then. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21743: [SPARK-24767][Launcher] Propagate MDC to spark-su...

2018-09-03 Thread yifeih
Github user yifeih closed the pull request at: https://github.com/apache/spark/pull/21743 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22310: [Spark-25298][Build] Improve build definition for...

2018-09-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22310 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22310: [Spark-25298][Build] Improve build definition for Scala ...

2018-09-03 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22310 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22317 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95614/ Test PASSed. ---

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22317 **[Test build #95614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95614/testReport)** for PR 22317 at commit

[GitHub] spark pull request #22218: [SPARK-25228][CORE]Add executor CPU time metric.

2018-09-03 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22218#discussion_r214675924 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorSource.scala --- @@ -73,6 +76,28 @@ class ExecutorSource(threadPool: ThreadPoolExecutor,

[GitHub] spark issue #19410: [SPARK-22184][CORE][GRAPHX] GraphX fails in case of insu...

2018-09-03 Thread szhem
Github user szhem commented on the issue: https://github.com/apache/spark/pull/19410 I've tested the mentioned checkpointers with `spark.cleaner.referenceTracking.cleanCheckpoints` set to `true` and without explicit checkpoint files removal. It seems that there are somewhere

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22112 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22112 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95607/ Test PASSed. ---

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22112 **[Test build #95607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95607/testReport)** for PR 22112 at commit

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22318 @ueshin @HyukjinKwon can you trigger tests? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r214671722 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -69,7 +69,7 @@ case class InsertIntoHiveTable(

<    1   2   3   4   5   >