[GitHub] spark issue #19311: [SPARK-22083][CORE] Release locks in MemoryStore.evictBl...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19311 **[Test build #82046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82046/testReport)** for PR 19311 at commit

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-21 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19311 [SPARK-22083][CORE] Release locks in MemoryStore.evictBlocksToFreeSpace ## What changes were proposed in this pull request? MemoryStore.evictBlocksToFreeSpace acquires write locks for all

[GitHub] spark pull request #19232: [SPARK-22009][ML] Using treeAggregate improve som...

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19232 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19288: [SPARK-22075][ML] GBTs unpersist datasets cached ...

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19288 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16578 **[Test build #82045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82045/testReport)** for PR 16578 at commit

[GitHub] spark issue #19232: [SPARK-22009][ML] Using treeAggregate improve some algs

2017-09-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19232 I'm gonna merge this as a non-trivial win. I think the benefit outweighs concerns. We can add to the change later. --- - To

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-21 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r140332294 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19250 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19250 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82037/ Test PASSed. ---

[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19250 **[Test build #82037 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82037/testReport)** for PR 19250 at commit

[GitHub] spark pull request #19305: [SPARK-22088][SQL] Incorrect scalastyle comment c...

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19305 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19305: [SPARK-22088][SQL] Incorrect scalastyle comment causes w...

2017-09-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19305 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19305: [SPARK-22088][SQL] Incorrect scalastyle comment causes w...

2017-09-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19305 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19222 **[Test build #82044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82044/testReport)** for PR 19222 at commit

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-21 Thread kalvinnchau
Github user kalvinnchau commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r140326376 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCredentialRenewer.scala --- @@ -0,0 +1,150 @@ +/*

[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82043/testReport)** for PR 18747 at commit

[GitHub] spark issue #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install ...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19310 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19310: [SPARK-18136] Fix SPARK_JARS_DIR for Python pip i...

2017-09-21 Thread jsnowacki
GitHub user jsnowacki opened a pull request: https://github.com/apache/spark/pull/19310 [SPARK-18136] Fix SPARK_JARS_DIR for Python pip install on Windows ## What changes were proposed in this pull request? Fix for setup of `SPARK_JARS_DIR` on Windows as it looks for

[GitHub] spark issue #18659: [SPARK-21190][PYSPARK][WIP] Python Vectorized UDFs

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18659 **[Test build #82042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82042/testReport)** for PR 18659 at commit

[GitHub] spark issue #18659: [SPARK-21190][PYSPARK][WIP] Python Vectorized UDFs

2017-09-21 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/18659 Thanks @ueshin , that works to allow the tests to pass. I do worry that it might cause some other issues and I would much prefer we upgrade Arrow to handle this, but I'll push this and we can

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-21 Thread dhruve
Github user dhruve commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r140321640 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark pull request #19308: [SPARK-22092] Reallocation in OffHeapColumnVector...

2017-09-21 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/19308#discussion_r140320919 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java --- @@ -517,6 +517,7 @@ public void

[GitHub] spark issue #13143: [SPARK-15359] [Mesos] Mesos dispatcher should handle DRI...

2017-09-21 Thread ArtRand
Github user ArtRand commented on the issue: https://github.com/apache/spark/pull/13143 @skonto @susanxhuynh @devaraj-kavali are people still interested in this? I was just playing around with this code to clean up ZK state.. Would be happy to try this when I have a few cycles. ---

[GitHub] spark issue #19309: [SPARK-19558][sql] Add config key to register QueryExecu...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19309 **[Test build #82041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82041/testReport)** for PR 19309 at commit

[GitHub] spark pull request #19309: [SPARK-19558][sql] Add config key to register Que...

2017-09-21 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/19309 [SPARK-19558][sql] Add config key to register QueryExecutionListeners automatically. This change adds a new SQL config key that is equivalent to SparkContext's "spark.extraListeners", allowing

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19307 Good to see this passes, as I expected, because it means we can also commit this chunk of progress towards 2.12 without any downside for 2.11. Right now this means that the 2.12 build does

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19204 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19204 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82040/ Test PASSed. ---

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19204 **[Test build #82040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82040/testReport)** for PR 19204 at commit

[GitHub] spark pull request #19291: Branch 2.1

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19291 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19296: Branch 2.2

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19296 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19307 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82036/ Test PASSed. ---

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19307 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19307 **[Test build #82036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82036/testReport)** for PR 19307 at commit

[GitHub] spark pull request #19280: [SPARK-21928][CORE] Set classloader on Serializer...

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19280 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19280: [SPARK-21928][CORE] Set classloader on SerializerManager...

2017-09-21 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19280 Didn't merge to 2.1, please open a PR against that branch if you want it there. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19280: [SPARK-21928][CORE] Set classloader on SerializerManager...

2017-09-21 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19280 LGTM, merging to master / 2.2 / 2.1. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19204: [SPARK-21981][PYTHON][ML] Added Python interface for Clu...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19204 **[Test build #82040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82040/testReport)** for PR 19204 at commit

[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...

2017-09-21 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18704 @michal-databricks do you have any thoughts? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18994 **[Test build #82039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82039/testReport)** for PR 18994 at commit

[GitHub] spark issue #18994: [SPARK-21784][SQL] Adds support for defining information...

2017-09-21 Thread sureshthalamati
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/18994 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19306: [SPARK-21977][HOTFIX] Adjust EnsureStatefulOpPart...

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19306 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19306: [SPARK-21977][HOTFIX] Adjust EnsureStatefulOpPartitionin...

2017-09-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19306 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19308: [SPARK-22092] Reallocation in OffHeapColumnVector.reserv...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19308 **[Test build #82038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82038/testReport)** for PR 19308 at commit

[GitHub] spark issue #19308: [SPARK-22092] Reallocation in OffHeapColumnVector.reserv...

2017-09-21 Thread ala
Github user ala commented on the issue: https://github.com/apache/spark/pull/19308 @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19308: [SPARK-22092] Reallocation in OffHeapColumnVector...

2017-09-21 Thread ala
GitHub user ala opened a pull request: https://github.com/apache/spark/pull/19308 [SPARK-22092] Reallocation in OffHeapColumnVector.reserveInternal corrupts array data ## What changes were proposed in this pull request? `OffHeapColumnVector.reserveInternal()` will only

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r140296863 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-21 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19290 isn't it the other way around? https://github.com/jimhester/lintr/compare/v1.0.1?expand=1 --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19194: [SPARK-20589] Allow limiting task concurrency per...

2017-09-21 Thread dhruve
Github user dhruve commented on a diff in the pull request: https://github.com/apache/spark/pull/19194#discussion_r140293577 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -619,6 +625,47 @@ private[spark] class ExecutorAllocationManager(

[GitHub] spark issue #19306: [SPARK-21977][HOTFIX] Adjust EnsureStatefulOpPartitionin...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19306 **[Test build #3928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3928/testReport)** for PR 19306 at commit

[GitHub] spark pull request #17902: [SPARK-20641][core] Add key-value store abstracti...

2017-09-21 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/17902#discussion_r140290617 --- Diff: common/kvstore/src/main/java/org/apache/spark/kvstore/KVStoreSerializer.java --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #19305: [SPARK-22088][SQL] Incorrect scalastyle comment causes w...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19305 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82035/ Test PASSed. ---

[GitHub] spark issue #19305: [SPARK-22088][SQL] Incorrect scalastyle comment causes w...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19305 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19305: [SPARK-22088][SQL] Incorrect scalastyle comment causes w...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19305 **[Test build #82035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82035/testReport)** for PR 19305 at commit

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82034/ Test FAILed. ---

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19290 **[Test build #82034 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82034/testReport)** for PR 19290 at commit

[GitHub] spark issue #19250: [SPARK-12297] Table timezone correction for Timestamps

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19250 **[Test build #82037 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82037/testReport)** for PR 19250 at commit

[GitHub] spark issue #19306: [SPARK-21977][HOTFIX] Adjust EnsureStatefulOpPartitionin...

2017-09-21 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/19306 Thanks @srowen . LGTM! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19234: [SPARK-22010][PySpark] Change fromInternal method of Tim...

2017-09-21 Thread maver1ck
Github user maver1ck commented on the issue: https://github.com/apache/spark/pull/19234 It was introduced with this PEP. https://www.python.org/dev/peps/pep-0495/ --- - To unsubscribe, e-mail:

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19307 **[Test build #82036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82036/testReport)** for PR 19307 at commit

[GitHub] spark issue #19145: [spark-21933][yarn] Spark Streaming request more executo...

2017-09-21 Thread klion26
Github user klion26 commented on the issue: https://github.com/apache/spark/pull/19145 @jerryshao thank you for your comment, I will try to find how MR/TEZ handle this --- - To unsubscribe, e-mail:

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19307 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should n...

2017-09-21 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/18924#discussion_r140249325 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala --- @@ -503,17 +518,15 @@ final class OnlineLDAOptimizer extends

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19290 To @shivaram: > does that also happen with lintr-1.0.1 ? I wonder if we are seeing some specific performance slowdown because of not using a release (it might be good to follow up with

[GitHub] spark pull request #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-09-21 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r140248789 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceV2Writer.java --- @@ -0,0 +1,71 @@ +/* + * Licensed to the

[GitHub] spark pull request #19269: [SPARK-22026][SQL][WIP] data source v2 write path

2017-09-21 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19269#discussion_r140248731 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataSourceV2Writer.java --- @@ -0,0 +1,71 @@ +/* + * Licensed to the

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19304 Thanks @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19304: [SPARK-22086][DOCS] Add expression description fo...

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19304 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19304 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #15544: [SPARK-17997] [SQL] Add an aggregation function f...

2017-09-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15544 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19306: [SPARK-21977][HOTFIX] Adjust EnsureStatefulOpPartitionin...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19306 **[Test build #3928 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3928/testReport)** for PR 19306 at commit

[GitHub] spark issue #15544: [SPARK-17997] [SQL] Add an aggregation function for coun...

2017-09-21 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15544 LGTM, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...

2017-09-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19290#discussion_r140244036 --- Diff: dev/lint-r.R --- @@ -28,6 +28,7 @@ if (! library(SparkR, lib.loc = LOCAL_LIB_LOC, logical.return = TRUE)) { # NOTE: The CRAN's version

[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...

2017-09-21 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19290#discussion_r140242203 --- Diff: dev/lint-r.R --- @@ -28,6 +28,7 @@ if (! library(SparkR, lib.loc = LOCAL_LIB_LOC, logical.return = TRUE)) { # NOTE: The CRAN's version is too

[GitHub] spark issue #19305: [SPARK-22088][SQL] Incorrect scalastyle comment causes w...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19305 **[Test build #82035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82035/testReport)** for PR 19305 at commit

[GitHub] spark pull request #19305: [SPARK-22088][SQL] Incorrect scalastyle comment c...

2017-09-21 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19305#discussion_r140239970 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -644,6 +652,7 @@ object StringTrimLeft {

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19290 **[Test build #82034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82034/testReport)** for PR 19290 at commit

[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...

2017-09-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19290#discussion_r140238274 --- Diff: dev/lint-r.R --- @@ -28,6 +28,7 @@ if (! library(SparkR, lib.loc = LOCAL_LIB_LOC, logical.return = TRUE)) { # NOTE: The CRAN's version

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19307 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82033/ Test FAILed. ---

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19307 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19307 **[Test build #82033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82033/testReport)** for PR 19307 at commit

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19304 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82031/ Test PASSed. ---

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19304 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19304 **[Test build #82031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82031/testReport)** for PR 19304 at commit

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16578 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82028/ Test PASSed. ---

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16578 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16578 **[Test build #82028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82028/testReport)** for PR 16578 at commit

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19304 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82026/ Test PASSed. ---

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19304 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19304: [SPARK-22086][DOCS] Add expression description for CASE ...

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19304 **[Test build #82026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82026/testReport)** for PR 19304 at commit

[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-09-21 Thread mallman
Github user mallman commented on a diff in the pull request: https://github.com/apache/spark/pull/16578#discussion_r140226905 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala --- @@ -0,0 +1,130 @@ +/* + *

[GitHub] spark issue #18015: [SAPRK-20785][WEB-UI][SQL]Spark should provide jump link...

2017-09-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18015 Hi @jerryshao, do you maybe have a preference? To me, I don't have a strong preference but I think I am okay with this. ---

[GitHub] spark issue #19301: [SPARK-22084][SQL] Fix performance regression in aggrega...

2017-09-21 Thread cenyuhai
Github user cenyuhai commented on the issue: https://github.com/apache/spark/pull/19301 should `sum(mt_cnt)` and `sum(ele_cnt)` be compute again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19301: [SPARK-22084][SQL] Fix performance regression in aggrega...

2017-09-21 Thread cenyuhai
Github user cenyuhai commented on the issue: https://github.com/apache/spark/pull/19301 I don't know wether my case can be optimized or not. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #18805: [SPARK-19112][CORE] Support for ZStandard codec

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18805 **[Test build #3927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3927/testReport)** for PR 18805 at commit

[GitHub] spark issue #19301: [SPARK-22084][SQL] Fix performance regression in aggrega...

2017-09-21 Thread cenyuhai
Github user cenyuhai commented on the issue: https://github.com/apache/spark/pull/19301 my case: ```sql select dt, geohash_of_latlng, sum(mt_cnt), sum(ele_cnt), round(sum(mt_cnt) * 1.0 * 100 / sum(mt_cnt_all), 2), round(sum(ele_cnt) * 1.0 * 100 /

[GitHub] spark issue #19296: Branch 2.2

2017-09-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19296 ping @rkp2916 close this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19281: [SPARK-21998][SQL] SortMergeJoinExec did not calc...

2017-09-21 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19281#discussion_r140215077 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala --- @@ -787,4 +789,64 @@ class JoinSuite extends QueryTest with SharedSQLContext {

[GitHub] spark pull request #19281: [SPARK-21998][SQL] SortMergeJoinExec did not calc...

2017-09-21 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/19281#discussion_r140191382 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -102,13 +102,22 @@ case class SortMergeJoinExec(

<    1   2   3   4   5   >