[GitHub] spark issue #22343: [SPARK-25132][SQL][FOLLOW-UP] The behavior must be consi...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22343 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22343: [SPARK-25132][SQL][FOLLOW-UP] The behavior must be consi...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22343 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22343: [SPARK-25132][SQL][FOLLOW-UP] The behavior must b...

2018-09-05 Thread seancxmao
GitHub user seancxmao opened a pull request: https://github.com/apache/spark/pull/22343 [SPARK-25132][SQL][FOLLOW-UP] The behavior must be consistent to do the conversion ## What changes were proposed in this pull request? parquet data source tables and hive parquet tables have

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215310881 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala --- @@ -0,0 +1,241 @@ +/* + *

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215310403 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/FetchedDataPool.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed to

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215308701 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala --- @@ -0,0 +1,241 @@ +/* + *

[GitHub] spark pull request #22323: [SPARK-25262][K8S] Allow SPARK_LOCAL_DIRS to be t...

2018-09-05 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22323#discussion_r215303844 --- Diff: docs/running-on-kubernetes.md --- @@ -215,6 +215,19 @@ spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.clai

[GitHub] spark issue #22322: [SPARK-25312][Documentation, Spark Core] Add description...

2018-09-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22322 Thank you for adding a description for the conf. Would it be possible to add missing descriptions for other confs in `spark.network.crypto.*`? ---

[GitHub] spark pull request #22341: [SPARK-24889][Core] Update block info when unpers...

2018-09-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22341#discussion_r215300948 --- Diff: core/src/main/scala/org/apache/spark/storage/RDDInfo.scala --- @@ -55,7 +55,7 @@ class RDDInfo( } private[spark] object RDDInfo {

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22318 **[Test build #95723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95723/testReport)** for PR 22318 at commit

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22338 Yeah, it's interesting... Seems both `checkedCast` and `Platform.getByte` are changed and performance gets gain. Any single change doesn't work. ---

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22318 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22318 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95714/ Test FAILed. ---

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22318 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22318 **[Test build #95714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95714/testReport)** for PR 22318 at commit

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-05 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22112 yeah we should file a separate jira to look at the shuffle output.I'm running a few stress tests and will let you know how those go. could you file a jira for that and link to this

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22338 > While I say these performance differences, I do not understand why these difference occurs completely. That is why I said "I have not found the root cause". Yes, I am in the same

[GitHub] spark pull request #22341: [SPARK-24889][Core] Update block info when unpers...

2018-09-05 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/22341#discussion_r215285275 --- Diff: core/src/main/scala/org/apache/spark/storage/RDDInfo.scala --- @@ -55,7 +55,7 @@ class RDDInfo( } private[spark] object

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22342 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95719/ Test PASSed. ---

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22342 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22342 **[Test build #95719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95719/testReport)** for PR 22342 at commit

[GitHub] spark pull request #19773: [SPARK-22546][SQL] Supporting for changing column...

2018-09-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/19773#discussion_r215283130 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -350,16 +366,11 @@ case class AlterTableChangeColumnCommand(

[GitHub] spark pull request #19773: [SPARK-22546][SQL] Supporting for changing column...

2018-09-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/19773#discussion_r215279164 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -318,18 +318,34 @@ case class AlterTableChangeColumnCommand(

[GitHub] spark pull request #19773: [SPARK-22546][SQL] Supporting for changing column...

2018-09-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/19773#discussion_r215279507 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -318,18 +318,34 @@ case class AlterTableChangeColumnCommand(

[GitHub] spark pull request #19773: [SPARK-22546][SQL] Supporting for changing column...

2018-09-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/19773#discussion_r215280045 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -1697,6 +1697,16 @@ abstract class DDLSuite extends

[GitHub] spark pull request #22218: [SPARK-25228][CORE]Add executor CPU time metric.

2018-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22218 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22338 While I say these performance differences, I do not understand why these difference occurs completely. That is why I said "I have not found the root cause". Let us narrow down the problem and

[GitHub] spark issue #22218: [SPARK-25228][CORE]Add executor CPU time metric.

2018-09-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22218 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215281569 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/FetchedDataPool.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22336 **[Test build #95722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95722/testReport)** for PR 22336 at commit

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22338 **[Test build #95721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95721/testReport)** for PR 22338 at commit

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22336 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22336 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22338 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22338 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22165: [SPARK-25017][Core] Add test suite for BarrierCoordinato...

2018-09-05 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/22165 I think it should be fine to make `ContextBarrierState` private[spark] to test it, WDYT @mengxr ? --- - To unsubscribe,

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22336 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22338 you're right @kiszk. Let me update the PR accordingly in order to narrow down the change/problem, thanks. --- - To

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22336 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22336 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95710/ Test FAILed. ---

[GitHub] spark issue #22336: [SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` i...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22336 **[Test build #95710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95710/testReport)** for PR 22336 at commit

[GitHub] spark issue #22328: [SPARK-22666][ML][SQL] Spark datasource for image format

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22328 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95715/ Test FAILed. ---

[GitHub] spark issue #22328: [SPARK-22666][ML][SQL] Spark datasource for image format

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22328 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215277752 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/FetchedPoolSuite.scala --- @@ -0,0 +1,299 @@ +/* + * Licensed

[GitHub] spark issue #22277: [SPARK-25276] Redundant constrains when using alias

2018-09-05 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/22277 You can have `select * from (select a, a as c from table1 where a > 10) t where a > c` --- - To unsubscribe, e-mail:

[GitHub] spark issue #22328: [SPARK-22666][ML][SQL] Spark datasource for image format

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22328 **[Test build #95715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95715/testReport)** for PR 22328 at commit

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215275456 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala --- @@ -0,0 +1,241 @@ +/* + *

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22318 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22338 >This does fix the regression, will we also lose the perf speed up too? IIUC we did observe significant perf boost when introducing the memory block. @cloud-fan I checked the original PR

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread peter-toth
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215274203 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -754,11 +754,14 @@ class Analyzer( * a

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215274150 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -414,17 +468,37 @@ private[kafka010]

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22338 In addition to [your commit](https://github.com/apache/spark/pull/22338/commits/91adce590461dda885d88319a700a775e63f9ce6), I applied the following change, basically use `MemoryBlock` in

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22318 **[Test build #95720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95720/testReport)** for PR 22318 at commit

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread peter-toth
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215274137 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala --- @@ -23,12 +23,14 @@ package

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215274047 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala --- @@ -0,0 +1,241 @@ +/* + *

[GitHub] spark pull request #22326: [SPARK-25314][SQL] Fix Python UDF accessing attib...

2018-09-05 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/22326#discussion_r215271726 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1208,9 +1208,38 @@ object

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215270783 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala --- @@ -0,0 +1,241 @@ +/* + *

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22342 **[Test build #95719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95719/testReport)** for PR 22342 at commit

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22338 @kiszk yes, I know. Moving `checkedCast` makes a big difference, but if you just move it, without the other changes, there is no perf gain (at least this is what I found in my experiments). ---

[GitHub] spark issue #19773: [SPARK-22546][SQL] Supporting for changing column dataTy...

2018-09-05 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/19773 gentle ping @maropu, could you help to review this? I'll keep follow up this. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

2018-09-05 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/22138#discussion_r215268638 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/FetchedDataPool.scala --- @@ -0,0 +1,160 @@ +/* + * Licensed

[GitHub] spark pull request #22329: [SPARK-25328][PYTHON] Add an example for having t...

2018-09-05 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/22329#discussion_r215267320 --- Diff: python/pyspark/sql/functions.py --- @@ -2804,6 +2804,22 @@ def pandas_udf(f=None, returnType=None, functionType=None): | 1|1.5|

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22338 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95704/ Test PASSed. ---

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22338 @mgaido91 thanks, interestingly I did experiments with similar code in my box. While I am using the linux box, I can confirm the performance improvement (or performance recover).

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22338 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22338 **[Test build #95704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95704/testReport)** for PR 22338 at commit

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22326 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19616: [SPARK-22404][YARN] Provide an option to use unmanaged A...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19616 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22326 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22342 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21618: [SPARK-20408][SQL] Get the glob path in parallel to redu...

2018-09-05 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21618 gental ping @cloud-fan @gatorsmile @kiszk, we still meet this in internal folk, could you help to review? I'll resolve the conflict, great thanks. ---

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22342 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22342 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22179: [SPARK-25258][SPARK-23131][SPARK-25176][BUILD] Upgrade K...

2018-09-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22179 Do we have any compatibility issues here? Seems fine to me as we already shaded kryo. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22342 **[Test build #95717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95717/testReport)** for PR 22342 at commit

[GitHub] spark issue #22342: [MINOR][SS] Fix kafka-0-10-sql trivials

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22342 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95717/ Test FAILed. ---

[GitHub] spark pull request #22333: [SPARK-25335][BUILD] Skip Zinc downloading if it'...

2018-09-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22333#discussion_r215265080 --- Diff: build/mvn --- @@ -91,15 +92,23 @@ install_mvn() { # Install zinc under the build/ folder install_zinc() { - local

[GitHub] spark issue #22140: [SPARK-25072][PySpark] Forbid extra value for custom Row

2018-09-05 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22140 gental ping @HyukjinKwon @BryanCutler --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22342: Feature/fix kafka sql trivials

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22342 **[Test build #95717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95717/testReport)** for PR 22342 at commit

[GitHub] spark pull request #22319: [SPARK-25044][SQL][followup] add back UserDefined...

2018-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22319 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22326 **[Test build #95718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95718/testReport)** for PR 22326 at commit

[GitHub] spark pull request #22342: Feature/fix kafka sql trivials

2018-09-05 Thread dongjinleekr
GitHub user dongjinleekr opened a pull request: https://github.com/apache/spark/pull/22342 Feature/fix kafka sql trivials ## What changes were proposed in this pull request? Fix unused imports & outdated comments on `kafka-0-10-sql` module. (Found while I was working on

[GitHub] spark issue #22165: [SPARK-25017][Core] Add test suite for BarrierCoordinato...

2018-09-05 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22165 ``` Could I do the refactor of moving ContextBarrierState out of BarrierCoordinator? ``` gental ping @jiangxb1987, I still follow up this. :) ---

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22319 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22338: [SPARK-25317][CORE] Avoid perf regression in Murmur3 Has...

2018-09-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22338 This basically reverts the memory block in the hash computing, now the memory block is just a holder of the base object and base offset. This does fix the regression, will we also lose the perf

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22326 **[Test build #95716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95716/testReport)** for PR 22326 at commit

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22326 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95716/ Test FAILed. ---

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22326 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22326: [SPARK-25314][SQL] Fix Python UDF accessing attib...

2018-09-05 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/22326#discussion_r215261610 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1208,9 +1208,38 @@ object

[GitHub] spark issue #22328: [SPARK-22666][ML][SQL] Spark datasource for image format

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22328 **[Test build #95715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95715/testReport)** for PR 22328 at commit

[GitHub] spark issue #22328: [SPARK-22666][ML][SQL] Spark datasource for image format

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22328 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22326 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22326 **[Test build #95716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95716/testReport)** for PR 22326 at commit

[GitHub] spark issue #22326: [SPARK-25314][SQL] Fix Python UDF accessing attibutes fr...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22326 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22328: [SPARK-22666][ML][SQL] Spark datasource for image format

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22328 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...

2018-09-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22340 Yeah same question, but I see why that could cause a problem. Is the point here that while this is a test, the spark-submit run by the test should be run 'normally'? I am happy for a solution just

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215257096 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -754,11 +754,14 @@ class Analyzer( * a logical

[GitHub] spark issue #22318: [SPARK-25150][SQL] Rewrite condition when deduplicate Jo...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22318 **[Test build #95714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95714/testReport)** for PR 22318 at commit

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22112 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22112 Test PASSed. Refer to this link for build results (access rights to CI server needed):

<    1   2   3   4   5   6   >