[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198683846 --- Diff: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala --- @@ -251,6 +261,217 @@ class EventLoggingListenerSuite

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-27 Thread edwinalu
Github user edwinalu commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r198684121 --- Diff: project/MimaExcludes.scala --- @@ -89,7 +89,13 @@ object MimaExcludes {

[GitHub] spark pull request #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to B...

2018-06-27 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/21557#discussion_r198684081 --- Diff: python/pyspark/ml/clustering.py --- @@ -622,10 +621,10 @@ def __init__(self, featuresCol="features", predictionCol="prediction", maxIter=2

[GitHub] spark issue #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to Bisectin...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21557 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to Bisectin...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21557 **[Test build #92404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92404/testReport)** for PR 21557 at commit

[GitHub] spark issue #21533: [SPARK-24195][Core] Bug fix for local:/ path in SparkCon...

2018-06-27 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21533 I think maybe we could: 1) either ignore the files with "local" scheme, and let user to decide how to fetch the files, like what current fix. 2) or copy the 'local' scheme files to

[GitHub] spark issue #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to Bisectin...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/529/

[GitHub] spark issue #21631: [SPARK-24645][SQL] Skip parsing when csvColumnPruning en...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21631 LGTM. @MaxGekk please take a following action. Will help and check if it's needed. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21651: [SPARK-18258] Sink need access to offset representation

2018-06-27 Thread tedyu
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21651 cc @tdas --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21625: [SPARK-24206][SQL][FOLLOW-UP] Update DataSourceReadBench...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21625 LGTM too Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21631: [SPARK-24645][SQL] Skip parsing when csvColumnPruning en...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21631 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21625: [SPARK-24206][SQL][FOLLOW-UP] Update DataSourceRe...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21625 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21631: [SPARK-24645][SQL] Skip parsing when csvColumnPru...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21631 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21597: [SPARK-24603][SQL] Fix findTightestCommonType reference ...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21597 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21451 **[Test build #92398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92398/testReport)** for PR 21451 at commit

[GitHub] spark issue #21597: [SPARK-24603][SQL] Fix findTightestCommonType reference ...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21597 **[Test build #92405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92405/testReport)** for PR 21597 at commit

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92398/ Test PASSed. ---

[GitHub] spark pull request #21596: [SPARK-24601] Bump Jackson version

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21596#discussion_r198686913 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala --- @@ -25,8 +25,13 @@ import

[GitHub] spark issue #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to Bisectin...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21557 **[Test build #92404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92404/testReport)** for PR 21557 at commit

[GitHub] spark issue #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to Bisectin...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21557 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to Bisectin...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92404/ Test PASSed. ---

[GitHub] spark issue #21597: [SPARK-24603][SQL] Fix findTightestCommonType reference ...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21597 Merged to master, branch-2.3 and branch-2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21597: [SPARK-24603][SQL] Fix findTightestCommonType ref...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21597 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20345: [SPARK-23172][SQL] Expand the ReorderJoin rule to handle...

2018-06-27 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20345 yea, sounds good to me. I'll add end-to-end tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21389 **[Test build #92382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92382/testReport)** for PR 21389 at commit

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198494546 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -309,6 +312,9 @@ private[spark] class

[GitHub] spark pull request #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to B...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/21557#discussion_r198675445 --- Diff: python/pyspark/ml/clustering.py --- @@ -622,10 +621,10 @@ def __init__(self, featuresCol="features", predictionCol="prediction", maxIter=2

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21589 **[Test build #92389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92389/testReport)** for PR 21589 at commit

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92389/ Test FAILed. ---

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21389 **[Test build #92390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92390/testReport)** for PR 21389 at commit

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21440 **[Test build #92399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92399/testReport)** for PR 21440 at commit

[GitHub] spark pull request #21511: [SPARK-24491][Kubernetes] Configuration support f...

2018-06-27 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21511#discussion_r198659423 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -104,6 +104,20 @@ private[spark] object Config

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21389 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/21546#discussion_r198660818 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala --- @@ -183,34 +182,111 @@ private[sql] object

[GitHub] spark issue #21600: [Spark-24553][WEB-UI] http 302 fixes for href redirect

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21600 Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92393/ Test FAILed. ---

[GitHub] spark pull request #21650: [SPARK-24624] Support mixture of Python UDF and S...

2018-06-27 Thread icexelloss
GitHub user icexelloss opened a pull request: https://github.com/apache/spark/pull/21650 [SPARK-24624] Support mixture of Python UDF and Scalar Pandas UDF ## What changes were proposed in this pull request? This PR add supports for using mixed Python UDF and Scalar Pandas

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21589 **[Test build #92393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92393/testReport)** for PR 21589 at commit

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21600: [Spark-24553][WEB-UI] http 302 fixes for href red...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21600 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13274: Glrm

2018-06-27 Thread Tagar
Github user Tagar commented on the issue: https://github.com/apache/spark/pull/13274 @rezazadeh is there is any plan to incorporate GLRM into core Spark? It seems https://github.com/rezazadeh/spark/tree/glrm/examples/src/main/scala/org/apache/spark/examples/glrm hasn't had

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21650 This PR takes me a while to get to because I am not very familiar with Catalyst rules. I think in the end the change is relative simple but I would appreciate some more careful review from

[GitHub] spark pull request #21650: [SPARK-24624] Support mixture of Python UDF and S...

2018-06-27 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21650#discussion_r198664314 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala --- @@ -94,36 +95,59 @@ object

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21589 **[Test build #92402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92402/testReport)** for PR 21589 at commit

[GitHub] spark pull request #21611: [SPARK-24569][SQL] Aggregator with output type Op...

2018-06-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21611#discussion_r198664098 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala --- @@ -333,4 +406,28 @@ class DatasetAggregatorSuite extends QueryTest

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21650 **[Test build #92401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92401/testReport)** for PR 21650 at commit

[GitHub] spark pull request #21611: [SPARK-24569][SQL] Aggregator with output type Op...

2018-06-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21611#discussion_r198664388 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala --- @@ -148,6 +148,79 @@ object VeryComplexResultAgg extends

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/528/

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21611 **[Test build #92403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92403/testReport)** for PR 21611 at commit

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21601: [SPARK-24610] fix reading small files via wholeTe...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21601#discussion_r198667795 --- Diff: core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala --- @@ -53,6 +53,19 @@ private[spark] class WholeTextFileInputFormat

[GitHub] spark pull request #21511: [SPARK-24491][Kubernetes] Configuration support f...

2018-06-27 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21511#discussion_r198653002 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -104,6 +104,20 @@ private[spark] object Config

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 ### Memory Improvements **toPandas()** The most significant improvement is reduction of the upper bound space complexity in the JVM driver. Before, the entire dataset was

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92390/ Test PASSed. ---

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198655159 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21440 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21440 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/525/

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198656665 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -659,6 +701,11 @@ private[spark] class BlockManager( * Get block

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198653427 --- Diff: core/src/main/scala/org/apache/spark/network/netty/NettyBlockRpcServer.scala --- @@ -73,10 +73,34 @@ class NettyBlockRpcServer( }

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198656910 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -723,7 +770,9 @@ private[spark] class BlockManager( }

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198652991 --- Diff: core/src/main/scala/org/apache/spark/network/BlockDataManager.scala --- @@ -43,6 +44,17 @@ trait BlockDataManager { level:

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198657121 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1341,12 +1390,16 @@ private[spark] class BlockManager( try {

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198652931 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/UploadBlockStream.java --- @@ -0,0 +1,89 @@ +/* + *

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21542 **[Test build #92391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92391/testReport)** for PR 21542 at commit

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92391/ Test FAILed. ---

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21613: [SPARK-24629][SQL]thrift server memory leaks when Beelin...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21613 cc @liufengdb --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #13274: Glrm

2018-06-27 Thread Tagar
Github user Tagar commented on a diff in the pull request: https://github.com/apache/spark/pull/13274#discussion_r198663163 --- Diff: examples/src/main/scala/org/apache/spark/examples/glrm/SparkGLRM.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21650 **[Test build #92400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92400/testReport)** for PR 21650 at commit

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/526/

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/527/

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21650 @icexelloss Can you also show the query plan of the examples in the PR description? Thanks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21650 nit: Also, can you put `[SQL][PYTHON]` in the title? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92392/ Test PASSed. ---

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198668704 --- Diff: core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala --- @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198668294 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92392/testReport)** for PR 21644 at commit

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198668278 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/512/

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92380/testReport)** for PR 21644 at commit

[GitHub] spark issue #21565: [SPARK-24558][Core]wrong Idle Timeout value is used in c...

2018-06-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21565 have you addressed all the comments? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21565: [SPARK-24558][Core]wrong Idle Timeout value is us...

2018-06-27 Thread sandeep-katta
Github user sandeep-katta commented on a diff in the pull request: https://github.com/apache/spark/pull/21565#discussion_r198447523 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -488,9 +488,16 @@ private[spark] class

[GitHub] spark issue #21565: [SPARK-24558][Core]wrong Idle Timeout value is used in c...

2018-06-27 Thread sandeep-katta
Github user sandeep-katta commented on the issue: https://github.com/apache/spark/pull/21565 yes all the review comments are addressed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21648: [SPARK-24665][PySpark] Add SQLConf in PySpark to ...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21648#discussion_r198375176 --- Diff: python/pyspark/sql/conf.py --- @@ -64,6 +64,97 @@ def _checkType(self, obj, identifier): (identifier, obj,

[GitHub] spark pull request #21648: [SPARK-24665][PySpark] Add SQLConf in PySpark to ...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21648#discussion_r198375388 --- Diff: python/pyspark/sql/conf.py --- @@ -64,6 +64,97 @@ def _checkType(self, obj, identifier): (identifier, obj,

[GitHub] spark pull request #21648: [SPARK-24665][PySpark] Add SQLConf in PySpark to ...

2018-06-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21648#discussion_r198375767 --- Diff: python/pyspark/sql/conf.py --- @@ -64,6 +64,97 @@ def _checkType(self, obj, identifier): (identifier, obj,

[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...

2018-06-27 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/21553 In the last commit I revert the changes of SQLConf and created a new PR of #21648. Could this follow up PR merged first? Thanks. ---

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18900 If hive outputs createTime for partitions, we should do it too, right? --- - To unsubscribe, e-mail:

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21389 **[Test build #92363 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92363/testReport)** for PR 21389 at commit

[GitHub] spark pull request #21648: [SPARK-24665][PySpark] Add SQLConf in PySpark to ...

2018-06-27 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21648#discussion_r198378065 --- Diff: python/pyspark/sql/dataframe.py --- @@ -358,22 +360,19 @@ def show(self, n=20, truncate=True, vertical=False): def

[GitHub] spark pull request #21560: [SPARK-24386][SS] coalesce(1) aggregates in conti...

2018-06-27 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/21560#discussion_r198380164 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousAggregationSuite.scala --- @@ -50,6 +51,42 @@ class

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18900 **[Test build #92374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92374/testReport)** for PR 18900 at commit

<    1   2   3   4   5   >