[GitHub] spark issue #19788: [SPARK-9853][Core] Optimize shuffle fetch of contiguous ...

2017-11-26 Thread gczsjdy
Github user gczsjdy commented on the issue: https://github.com/apache/spark/pull/19788 Can we just add the `ContinuousShuffleBlockId` without adding new conf `spark.shuffle.continuousFetch`? While in classes related to shuffle read like `ShuffleBlockFetcherIterator`, we also pattern

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153124078 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -158,111 +178,73 @@ abstract class

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153123637 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class

[GitHub] spark pull request #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit pr...

2017-11-26 Thread kiszk
Github user kiszk closed the pull request at: https://github.com/apache/spark/pull/18641 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...

2017-11-26 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18641 #19752 will cover this solution. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153123184 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -158,111 +178,73 @@ abstract class

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19813 **[Test build #84207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84207/testReport)** for PR 19813 at commit

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153120458 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class CodegenContext {

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153120160 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class CodegenContext {

[GitHub] spark issue #19764: [SPARK-22539][SQL] Add second order for rangepartitioner...

2017-11-26 Thread gczsjdy
Github user gczsjdy commented on the issue: https://github.com/apache/spark/pull/19764 @caneGuy Can you give a specific example to illustrate your change? Maybe former partition result & later partition result? ---

[GitHub] spark issue #19788: [SPARK-9853][Core] Optimize shuffle fetch of contiguous ...

2017-11-26 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19788 @yucai I'm thinking of the necessity to add this new configuration `spark.shuffle.continuousFetch` like you mentioned above. This PR you proposed is actually a superset of previous way, it is

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19752 **[Test build #84206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84206/testReport)** for PR 19752 at commit

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153118605 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -211,111 +231,73 @@ abstract class

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153118387 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -211,111 +231,73 @@ abstract class

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153118326 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -211,111 +231,73 @@ abstract class

[GitHub] spark issue #19788: [SPARK-9853][Core] Optimize shuffle fetch of contiguous ...

2017-11-26 Thread gczsjdy
Github user gczsjdy commented on the issue: https://github.com/apache/spark/pull/19788 What are ` external shuffle service` here? Can you explain a little bit? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19788: [SPARK-9853][Core] Optimize shuffle fetch of cont...

2017-11-26 Thread yucai
Github user yucai commented on a diff in the pull request: https://github.com/apache/spark/pull/19788#discussion_r153117548 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockId.scala --- @@ -116,8 +117,8 @@ object BlockId { def apply(name: String): BlockId = name

[GitHub] spark pull request #19788: [SPARK-9853][Core] Optimize shuffle fetch of cont...

2017-11-26 Thread gczsjdy
Github user gczsjdy commented on a diff in the pull request: https://github.com/apache/spark/pull/19788#discussion_r153117088 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockId.scala --- @@ -116,8 +117,8 @@ object BlockId { def apply(name: String): BlockId = name

[GitHub] spark pull request #11215: [SPARK-10969] [Streaming] [Kinesis] Allow specify...

2017-11-26 Thread kaklakariada
Github user kaklakariada closed the pull request at: https://github.com/apache/spark/pull/11215 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #11215: [SPARK-10969] [Streaming] [Kinesis] Allow specifying sep...

2017-11-26 Thread kaklakariada
Github user kaklakariada commented on the issue: https://github.com/apache/spark/pull/11215 Solved with https://issues.apache.org/jira/browse/SPARK-19911 / #17250, see [this

[GitHub] spark issue #19819: [SPARK-22606][Streaming]Add threadId to the CachedKafkaC...

2017-11-26 Thread lvdongr
Github user lvdongr commented on the issue: https://github.com/apache/spark/pull/19819 Will the cached consumer to the same partition increase , when different tasks consume the same partition and no place to remove? ---

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r15398 --- Diff: python/pyspark/sql/types.py --- @@ -1108,19 +1109,23 @@ def _has_nulltype(dt): return isinstance(dt, NullType) -def

[GitHub] spark pull request #19792: [SPARK-22566][PYTHON] Better error message for `_...

2017-11-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19792#discussion_r153110594 --- Diff: python/pyspark/sql/tests.py --- @@ -1722,6 +1723,83 @@ def test_infer_long_type(self): self.assertEqual(_infer_type(2**61),

[GitHub] spark pull request #19815: [SPARK-22602][SQL] remove ColumnVector#loadBytes

2017-11-26 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19815 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19815: [SPARK-22602][SQL] remove ColumnVector#loadBytes

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19815 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #11994: [SPARK-14151] Expose metrics Source and Sink inte...

2017-11-26 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/11994#discussion_r153110760 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala --- @@ -195,18 +196,26 @@ private[spark] class MetricsSystem private (

[GitHub] spark issue #19815: [SPARK-22602][SQL] remove ColumnVector#loadBytes

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19815 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19815: [SPARK-22602][SQL] remove ColumnVector#loadBytes

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19815 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84203/ Test PASSed. ---

[GitHub] spark issue #19815: [SPARK-22602][SQL] remove ColumnVector#loadBytes

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19815 **[Test build #84203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84203/testReport)** for PR 19815 at commit

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84202/ Test PASSed. ---

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11994 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11994 **[Test build #84202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84202/testReport)** for PR 11994 at commit

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2017-11-26 Thread sriramrajendiran
Github user sriramrajendiran commented on the issue: https://github.com/apache/spark/pull/16578 @felixcheung can you help ? we are hoping to see it in 2.3 release. Feature underneath a default disabled flag looks safe option. ---

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19607 **[Test build #84205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84205/testReport)** for PR 19607 at commit

[GitHub] spark issue #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp...

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19607 **[Test build #84204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84204/testReport)** for PR 19607 at commit

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153107748 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -997,6 +997,14 @@ object SQLConf { .intConf

[GitHub] spark pull request #19607: [SPARK-22395][SQL][PYTHON] Fix the behavior of ti...

2017-11-26 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/19607#discussion_r153107765 --- Diff: python/setup.py --- @@ -201,7 +201,7 @@ def _supports_symlinks(): extras_require={ 'ml': ['numpy>=1.7'],

[GitHub] spark issue #19815: [SPARK-22602][SQL] remove ColumnVector#loadBytes

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19815 LGTM pending Jenkins. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11994 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11994 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84201/ Test FAILed. ---

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11994 **[Test build #84201 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84201/testReport)** for PR 11994 at commit

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153101065 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -211,111 +231,73 @@ abstract class

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19752 LGTM except a few minor comments --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153100483 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -211,111 +231,73 @@ abstract class

[GitHub] spark pull request #19752: [SPARK-22520][SQL] Support code generation for la...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19752#discussion_r153100103 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -211,111 +231,73 @@ abstract class

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19752 LGTM cc @cloud-fan @kiszk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19714: [SPARK-22489][SQL] Shouldn't change broadcast join build...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19714 Just for curiosity, what if users ask to broadcast both the join side in the hint? Shall we throw exception or pick a smaller side to broadcast according to stats? BTW it's a behavior

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153099135 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153098738 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class

[GitHub] spark pull request #11994: [SPARK-14151] Expose metrics Source and Sink inte...

2017-11-26 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/11994#discussion_r153098194 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/ConsoleSink.scala --- @@ -25,27 +25,29 @@ import com.codahale.metrics.{ConsoleReporter,

[GitHub] spark pull request #11994: [SPARK-14151] Expose metrics Source and Sink inte...

2017-11-26 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/11994#discussion_r153097744 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala --- @@ -195,18 +196,26 @@ private[spark] class MetricsSystem private (

[GitHub] spark pull request #11994: [SPARK-14151] Expose metrics Source and Sink inte...

2017-11-26 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/11994#discussion_r153097850 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/ConsoleSink.scala --- @@ -25,27 +25,29 @@ import com.codahale.metrics.{ConsoleReporter,

[GitHub] spark pull request #11994: [SPARK-14151] Expose metrics Source and Sink inte...

2017-11-26 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/11994#discussion_r153097471 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala --- @@ -195,18 +196,26 @@ private[spark] class MetricsSystem private (

[GitHub] spark pull request #11994: [SPARK-14151] Expose metrics Source and Sink inte...

2017-11-26 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/11994#discussion_r153098545 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/Sink.scala --- @@ -17,8 +17,37 @@ package org.apache.spark.metrics.sink

[GitHub] spark issue #19813: [WIP][SPARK-22600][SQL] Fix 64kb limit for deeply nested...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19813 If we have a clear rule, I think it makes more sense to do this in `CodegenContext`, i.e. having a `def splitExpressions(expressions: Seq[String]): String`, which automatically extract the

[GitHub] spark issue #19714: [SPARK-22489][SQL] Shouldn't change broadcast join build...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19714 We also need to update the comment of `JoinSelection`. cc @liufengdb @cloud-fan @rxin --- - To unsubscribe, e-mail:

[GitHub] spark issue #19815: [SPARK-22602][SQL] remove ColumnVector#loadBytes

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19815 **[Test build #84203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84203/testReport)** for PR 19815 at commit

[GitHub] spark pull request #19714: [SPARK-22489][SQL] Shouldn't change broadcast joi...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19714#discussion_r153098067 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala --- @@ -223,4 +223,36 @@ class BroadcastJoinSuite

[GitHub] spark pull request #19818: [SPARK-22604][SQL] remove the get address methods...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19818#discussion_r153096313 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java --- @@ -62,13 +62,6 @@ */ public abstract

[GitHub] spark pull request #19818: [SPARK-22604][SQL] remove the get address methods...

2017-11-26 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19818#discussion_r153096296 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java --- @@ -73,12 +75,12 @@ public

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153095567 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class CodegenContext {

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11994 **[Test build #84202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84202/testReport)** for PR 11994 at commit

[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19814 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84200/ Test FAILed. ---

[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19814 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19814 **[Test build #84200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84200/testReport)** for PR 19814 at commit

[GitHub] spark issue #19764: [SPARK-22539][SQL] Add second order for rangepartitioner...

2017-11-26 Thread caneGuy
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19764 Ping any admin help review this?Thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11994 **[Test build #84201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84201/testReport)** for PR 11994 at commit

[GitHub] spark issue #17520: [WIP][SPARK-19712][SQL] Move PullupCorrelatedPredicates ...

2017-11-26 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/17520 @nsyca Can you resolve conflicts? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19717: [SPARK-18278] [Submission] Spark on Kubernetes - basic s...

2017-11-26 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19717 +CC @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153092301 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class CodegenContext {

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153092158 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class CodegenContext {

[GitHub] spark pull request #19823: [SPARK-22601][SQL] Data load is getting displayed...

2017-11-26 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19823#discussion_r153091293 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -341,6 +341,12 @@ case class LoadDataCommand( }

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153087493 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala --- @@ -0,0 +1,226 @@ +/*

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153088979 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala --- @@ -0,0 +1,70 @@

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153088232 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala --- @@ -0,0 +1,226 @@ +/*

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153091143 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -0,0

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153084210 --- Diff: resource-managers/kubernetes/core/pom.xml --- @@ -0,0 +1,94 @@ + + +http://maven.apache.org/POM/4.0.0;

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153087234 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala --- @@ -0,0 +1,226 @@ +/*

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153089366 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -0,0

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153089805 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -0,0

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153089664 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -0,0

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153087550 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala --- @@ -0,0 +1,229 @@ +/*

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153084631 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -0,0 +1,115 @@ +/* + * Licensed to the

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153089121 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -0,0

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153084513 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -0,0 +1,115 @@ +/* + * Licensed to the

[GitHub] spark pull request #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - B...

2017-11-26 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19468#discussion_r153090634 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodFactory.scala --- @@ -0,0 +1,226 @@ +/*

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-11-26 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/11994 @felixcheung thanks for your reviewing. I think there's no next step, current changes should be enough for user to externalize customized metrics source and sink. ---

[GitHub] spark pull request #19816: [SPARK-21693][R][FOLLOWUP] Reduce shuffle partiti...

2017-11-26 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19816 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19816: [SPARK-21693][R][FOLLOWUP] Reduce shuffle partitions run...

2017-11-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19816 Thanks @felixcheung. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19816: [SPARK-21693][R][FOLLOWUP] Reduce shuffle partitions run...

2017-11-26 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19816 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19788: [SPARK-9853][Core] Optimize shuffle fetch of cont...

2017-11-26 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19788#discussion_r153089584 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala --- @@ -196,12 +196,14 @@ private[spark] class

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153087888 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153087865 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84199/ Test PASSed. ---

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19821: [WIP][SPARK-22608][SQL] add new API to CodeGenera...

2017-11-26 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19821#discussion_r153087808 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -785,13 +785,36 @@ class

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19752 **[Test build #84199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84199/testReport)** for PR 19752 at commit

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84198/ Test PASSed. ---

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19752 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19752: [SPARK-22520][SQL] Support code generation for large Cas...

2017-11-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19752 **[Test build #84198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84198/testReport)** for PR 19752 at commit

  1   2   >