[GitHub] spark issue #21651: [SPARK-18258] Sink need access to offset representation

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21651 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21651: [SPARK-18258] Sink need access to offset represen...

2018-06-27 Thread ConcurrencyPractitioner
GitHub user ConcurrencyPractitioner opened a pull request: https://github.com/apache/spark/pull/21651 [SPARK-18258] Sink need access to offset representation ## What changes were proposed in this pull request? Currently, sinks only have access to the batchId and the data,

[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6

2018-06-27 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21495 > The only change with this PR is that the welcome message will be printed first, and then the Spark URL will be shown latter. It's a minor difference. I think we should create a JIRA to

[GitHub] spark pull request #21557: [SPARK-24439][ML][PYTHON]Add distanceMeasure to B...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/21557#discussion_r198675445 --- Diff: python/pyspark/ml/clustering.py --- @@ -622,10 +621,10 @@ def __init__(self, featuresCol="features", predictionCol="prediction", maxIter=2

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92392/ Test PASSed. ---

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92392/testReport)** for PR 21644 at commit

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198668278 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198668704 --- Diff: core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala --- @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198668294 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21601: [SPARK-24610] fix reading small files via wholeTe...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21601#discussion_r198667795 --- Diff: core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala --- @@ -53,6 +53,19 @@ private[spark] class WholeTextFileInputFormat

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21650 nit: Also, can you put `[SQL][PYTHON]` in the title? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21650 @icexelloss Can you also show the query plan of the examples in the PR description? Thanks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21611 **[Test build #92403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92403/testReport)** for PR 21611 at commit

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/528/

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21611: [SPARK-24569][SQL] Aggregator with output type Op...

2018-06-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21611#discussion_r198664388 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala --- @@ -148,6 +148,79 @@ object VeryComplexResultAgg extends

[GitHub] spark pull request #21650: [SPARK-24624] Support mixture of Python UDF and S...

2018-06-27 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21650#discussion_r198664314 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala --- @@ -94,36 +95,59 @@ object

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21589 **[Test build #92402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92402/testReport)** for PR 21589 at commit

[GitHub] spark pull request #21611: [SPARK-24569][SQL] Aggregator with output type Op...

2018-06-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21611#discussion_r198664098 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala --- @@ -333,4 +406,28 @@ class DatasetAggregatorSuite extends QueryTest

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21650 **[Test build #92401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92401/testReport)** for PR 21650 at commit

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/527/

[GitHub] spark issue #13274: Glrm

2018-06-27 Thread Tagar
Github user Tagar commented on the issue: https://github.com/apache/spark/pull/13274 @rezazadeh is there is any plan to incorporate GLRM into core Spark? It seems https://github.com/rezazadeh/spark/tree/glrm/examples/src/main/scala/org/apache/spark/examples/glrm hasn't had

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21650 This PR takes me a while to get to because I am not very familiar with Catalyst rules. I think in the end the change is relative simple but I would appreciate some more careful review from

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21650 **[Test build #92400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92400/testReport)** for PR 21650 at commit

[GitHub] spark pull request #13274: Glrm

2018-06-27 Thread Tagar
Github user Tagar commented on a diff in the pull request: https://github.com/apache/spark/pull/13274#discussion_r198663163 --- Diff: examples/src/main/scala/org/apache/spark/examples/glrm/SparkGLRM.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/526/

[GitHub] spark issue #21650: [SPARK-24624] Support mixture of Python UDF and Scalar P...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21650 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92393/ Test FAILed. ---

[GitHub] spark pull request #21600: [Spark-24553][WEB-UI] http 302 fixes for href red...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21600 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21589 **[Test build #92393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92393/testReport)** for PR 21589 at commit

[GitHub] spark pull request #21650: [SPARK-24624] Support mixture of Python UDF and S...

2018-06-27 Thread icexelloss
GitHub user icexelloss opened a pull request: https://github.com/apache/spark/pull/21650 [SPARK-24624] Support mixture of Python UDF and Scalar Pandas UDF ## What changes were proposed in this pull request? This PR add supports for using mixed Python UDF and Scalar Pandas

[GitHub] spark issue #21613: [SPARK-24629][SQL]thrift server memory leaks when Beelin...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21613 cc @liufengdb --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21600: [Spark-24553][WEB-UI] http 302 fixes for href redirect

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21600 Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/Pa...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21389 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/21546#discussion_r198660818 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala --- @@ -183,34 +182,111 @@ private[sql] object

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92391/ Test FAILed. ---

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21542 **[Test build #92391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92391/testReport)** for PR 21542 at commit

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21389 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21511: [SPARK-24491][Kubernetes] Configuration support f...

2018-06-27 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21511#discussion_r198659423 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -104,6 +104,20 @@ private[spark] object Config

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198652991 --- Diff: core/src/main/scala/org/apache/spark/network/BlockDataManager.scala --- @@ -43,6 +44,17 @@ trait BlockDataManager { level:

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198657121 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1341,12 +1390,16 @@ private[spark] class BlockManager( try {

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198652931 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/UploadBlockStream.java --- @@ -0,0 +1,89 @@ +/* + *

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198656665 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -659,6 +701,11 @@ private[spark] class BlockManager( * Get block

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198653427 --- Diff: core/src/main/scala/org/apache/spark/network/netty/NettyBlockRpcServer.scala --- @@ -73,10 +73,34 @@ class NettyBlockRpcServer( }

[GitHub] spark pull request #21451: [SPARK-24296][CORE][WIP] Replicate large blocks a...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21451#discussion_r198656910 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -723,7 +770,9 @@ private[spark] class BlockManager( }

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21440 **[Test build #92399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92399/testReport)** for PR 21440 at commit

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21440 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21440 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/525/

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198655159 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92390/ Test PASSed. ---

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 ### Memory Improvements **toPandas()** The most significant improvement is reduction of the upper bound space complexity in the JVM driver. Before, the entire dataset was

[GitHub] spark issue #21389: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFi...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21389 **[Test build #92390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92390/testReport)** for PR 21389 at commit

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92389/ Test FAILed. ---

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21589 **[Test build #92389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92389/testReport)** for PR 21589 at commit

[GitHub] spark pull request #21511: [SPARK-24491][Kubernetes] Configuration support f...

2018-06-27 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21511#discussion_r198653002 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -104,6 +104,20 @@ private[spark] object Config

[GitHub] spark issue #21075: [SPARK-23988][MESOS] Improve handling of appResource in ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21075 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198651582 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,38 @@ private[spark] class ChunkedByteBuffer(var chunks:

[GitHub] spark pull request #21636: [SPARK-24533] Typesafe rebranded to lightbend. Ch...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21636 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92397/ Test PASSed. ---

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92397/testReport)** for PR 21635 at commit

[GitHub] spark issue #20944: [SPARK-23831][SQL] Add org.apache.derby to IsolatedClien...

2018-06-27 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20944 So I had this come up while I was testing Spark 2.1.3 RC2 on a machine with an existing YARN cluster with `spark-testing-base`. Haven't had the chance to dig into it fully. ---

[GitHub] spark issue #21636: [SPARK-24533] Typesafe rebranded to lightbend. Changing ...

2018-06-27 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21636 Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21644: [SPARK-24660][SHS] Show correct error pages when ...

2018-06-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21644 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 ### Performance Tests - createDataFrame Tests run on a 4 node standalone cluster with 32 cores total, 14.04.1-Ubuntu and OpenJDK 8 measured wall clock time to execute

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21644 Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/524/

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 ### Performance Tests - toPandas Tests run on a 4 node standalone cluster with 32 cores total, 14.04.1-Ubuntu and OpenJDK 8 measured wall clock time to execute `toPandas()` and took

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198630017 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,38 @@ private[spark] class ChunkedByteBuffer(var chunks:

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198641717 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198632416 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-06-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r198636259 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92396/ Test PASSed. ---

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92396/testReport)** for PR 21635 at commit

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92397/testReport)** for PR 21635 at commit

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21451 **[Test build #92398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92398/testReport)** for PR 21451 at commit

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92386/ Test PASSed. ---

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21644 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21644: [SPARK-24660][SHS] Show correct error pages when downloa...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21644 **[Test build #92386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92386/testReport)** for PR 21644 at commit

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198641094 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnClusterSchedulerSource.scala --- @@ -0,0 +1,49 @@ +/* + *

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21560 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92387/ Test PASSed. ---

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21560 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21560 **[Test build #92387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92387/testReport)** for PR 21560 at commit

[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #92396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92396/testReport)** for PR 21635 at commit

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 _UPDATE_ I changed `toPandas` to write out of order partitions to python as they come in, followed by a list of indices to represent the correct batch order. In python, the batches are

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21546 **[Test build #92395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92395/testReport)** for PR 21546 at commit

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21546 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/523/

[GitHub] spark issue #21546: [WIP][SPARK-23030][SQL][PYTHON] Use Arrow stream format ...

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21546 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92394/ Test PASSed. ---

[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR

2018-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR

2018-06-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21649 **[Test build #92394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92394/testReport)** for PR 21649 at commit

[GitHub] spark pull request #21635: [SPARK-24594][YARN] Introducing metrics for YARN

2018-06-27 Thread attilapiros
Github user attilapiros commented on a diff in the pull request: https://github.com/apache/spark/pull/21635#discussion_r198630307 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -309,6 +312,9 @@ private[spark] class

<    1   2   3   4   5   >