[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21538 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21538 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21538 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91701/ Test PASSed. ---

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21538 **[Test build #91704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91704/testReport)** for PR 21538 at commit

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21538 add to whitelist --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21538 Yea, it's unfortunate .. we should fix and set up the Jenkins env too. --- - To unsubscribe, e-mail:

[GitHub] spark issue #16374: [SPARK-18925][STREAMING] Reduce memory usage of mapWithS...

2018-06-12 Thread vpchelko
Github user vpchelko commented on the issue: https://github.com/apache/spark/pull/16374 I don't use anymore approach above. To unpersist unnecessary RDD, I hacked MapWithStateDStream a little bit by calling unpersist for previously generated RDDs in internalMapWithStateStream.

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21538 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21538 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91704/ Test PASSed. ---

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21538 **[Test build #91704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91704/testReport)** for PR 21538 at commit

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91699/ Test FAILed. ---

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21537 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/44/

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3934/

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21537 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/43/

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21537 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3933/

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21537 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21537: [SPARK-24505][SQL] Convert strings in codegen to ...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21537#discussion_r194703775 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -1004,26 +1014,30 @@ case class Cast(child: Expression,

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/45/

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3935/

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21535 @viirya how much time this add to the overall testing time? If this is too much then we should perhaps just test interpreted encoders. WDYT? ---

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-12 Thread dongjinleekr
Github user dongjinleekr commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r194623958 --- Diff: python/pyspark/ml/feature.py --- @@ -2582,25 +2582,31 @@ class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol, JavaMLReadabl

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r194625679 --- Diff: python/pyspark/ml/feature.py --- @@ -2582,25 +2582,31 @@ class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol, JavaMLReadabl

[GitHub] spark issue #21536: [MINOR][CORE][TEST] Remove unnecessary sort in UnsafeInM...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21536 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/36/

[GitHub] spark issue #21498: [SPARK-24410][SQL][Core] Optimization for Union outputPa...

2018-06-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21498 @mgaido91 WDYT? Does the benchmark make sense to you? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3927/

[GitHub] spark pull request #20313: [SPARK-22974][ML] Attach attributes to output col...

2018-06-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20313#discussion_r194636521 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala --- @@ -264,7 +265,9 @@ class CountVectorizerModel(

[GitHub] spark issue #20313: [SPARK-22974][ML] Attach attributes to output column of ...

2018-06-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20313 cc @dbtsai too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21496 **[Test build #91694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91694/testReport)** for PR 21496 at commit

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

2018-06-12 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21370#discussion_r194641579 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -3209,6 +3222,19 @@ class Dataset[T] private[sql]( } }

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21537 cc @cloud-fan @hvanhovell @kiszk @mgaido91 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21537: [SPARK-24505][SQL] Convert strings in codegen to ...

2018-06-12 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/21537 [SPARK-24505][SQL] Convert strings in codegen to blocks: Cast and BoundAttribute ## What changes were proposed in this pull request? This is split from #21520. This includes changes of

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use level triggering and state...

2018-06-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r194664576 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSource.scala --- @@ -0,0 +1,65

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21535 **[Test build #91692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91692/testReport)** for PR 21535 at commit

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91692/ Test FAILed. ---

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21450 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91698/ Test FAILed. ---

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use level triggering and state...

2018-06-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r194670526 --- Diff: pom.xml --- @@ -760,6 +760,12 @@ 1.10.19 test + --- End diff -- > Does Akka streams

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21496 **[Test build #91694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91694/testReport)** for PR 21496 at commit

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds arrays_zip function to sparksql

2018-06-12 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21045 LGTM, (if we all agree and we go for it, then we have to update it also on the JIRA) --- - To unsubscribe, e-mail:

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21450 **[Test build #91700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91700/testReport)** for PR 21450 at commit

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21496 This is amazing to see many kind of test failures for a typo fix. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21496 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-06-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r194679672 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -122,8 +122,9 @@

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21496 **[Test build #91702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91702/testReport)** for PR 21496 at commit

[GitHub] spark pull request #21539: [SPARK-24500][SQL] Make sure streams are material...

2018-06-12 Thread hvanhovell
GitHub user hvanhovell opened a pull request: https://github.com/apache/spark/pull/21539 [SPARK-24500][SQL] Make sure streams are materialized during Tree transforms. ## What changes were proposed in this pull request? If you construct catalyst trees using

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21539 **[Test build #91703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91703/testReport)** for PR 21539 at commit

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3931/

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/41/

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-06-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r194687110 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSource.scala --- @@ -17,25 +17,170 @@

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21538 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21538: [SPARK-23754][PYTHON][FOLLOWUP][BACKPORT-2.3] Move UDF s...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21538 **[Test build #91701 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91701/testReport)** for PR 21538 at commit

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21537 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91697/ Test FAILed. ---

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21537 **[Test build #91697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91697/testReport)** for PR 21537 at commit

[GitHub] spark issue #21534: [SPARK-24526][build] Spaces in the build dir causes fail...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21534 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91696/ Test FAILed. ---

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3932/

[GitHub] spark issue #21498: [SPARK-24410][SQL][Core] Optimization for Union outputPa...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21498 @viirya sorry, I somehow lost your updated benchmark. Yes, it makes sense. In the case without any shuffle needed after the union we have about a 2% performance regression. I am not sure about

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21539 **[Test build #91708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91708/testReport)** for PR 21539 at commit

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-06-12 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21082 @icexelloss, mind resolving the conflict? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-06-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r194688618 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -796,6 +811,38 @@

[GitHub] spark pull request #21504: [SPARK-24479][SS] Added config for registering st...

2018-06-12 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21504#discussion_r194688537 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala --- @@ -96,6 +96,14 @@ object StaticSQLConf {

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/42/

[GitHub] spark issue #21537: [SPARK-24505][SQL] Convert strings in codegen to blocks:...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21537 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21529 **[Test build #91705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91705/testReport)** for PR 21529 at commit

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21529 thanks for your review @cloud-fan. Nice catch on the `transformUp`! I addressed all your comments. As far as the followup is regarded, we should decide if we want to support the

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21535 **[Test build #91699 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91699/testReport)** for PR 21535 at commit

[GitHub] spark pull request #21537: [SPARK-24505][SQL] Convert strings in codegen to ...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21537#discussion_r194704155 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -579,6 +579,22 @@ class CodegenContext

[GitHub] spark pull request #21537: [SPARK-24505][SQL] Convert strings in codegen to ...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21537#discussion_r194697467 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -625,25 +625,23 @@ case class Cast(child: Expression,

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21496 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21496 **[Test build #91702 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91702/testReport)** for PR 21496 at commit

[GitHub] spark issue #21496: docs: fix typo

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21496 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91702/ Test FAILed. ---

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21539 **[Test build #91703 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91703/testReport)** for PR 21539 at commit

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21539: [SPARK-24500][SQL] Make sure streams are materialized du...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21539 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91703/ Test FAILed. ---

[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...

2018-06-12 Thread rkarimi
Github user rkarimi commented on the issue: https://github.com/apache/spark/pull/17972 perhaps related: Big Random Forest Models (example: 100 or more trees with depth of around 20): Big models can be trained effectively even on machines with limited RAM (such

[GitHub] spark issue #21505: [SPARK-24457][SQL] Improving performance of stringToTime...

2018-06-12 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21505 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21523: [SPARK-24506][UI] Add UI filters also to thriftse...

2018-06-12 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21523#discussion_r194722157 --- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala --- @@ -91,7 +91,12 @@ private[spark] abstract class WebUI( /** Attach a handler to

[GitHub] spark issue #21501: [SPARK-15064][ML] Locale support in StopWordsRemover

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21501 **[Test build #91689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91689/testReport)** for PR 21501 at commit

[GitHub] spark issue #21276: [SPARK-24216][SQL] Spark TypedAggregateExpression uses g...

2018-06-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21276 I think this fixing is nice to have. cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...

2018-06-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21535 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-06-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21450 **[Test build #91698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91698/testReport)** for PR 21450 at commit

[GitHub] spark issue #21450: [SPARK-24319][SPARK SUBMIT] Fix spark-submit execution w...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21450 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-06-12 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r19467 --- Diff: core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala --- @@ -205,7 +208,7 @@ private[spark] class MetricsSystem private (

[GitHub] spark pull request #21538: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop ite...

2018-06-12 Thread e-dorigatti
GitHub user e-dorigatti opened a pull request: https://github.com/apache/spark/pull/21538 [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration wrapping from driver to executor SPARK-23754 was fixed in #21383 by changing the UDF code to wrap the user function, but this required

[GitHub] spark pull request #20260: [SPARK-23039][SQL] Finish TODO work in alter tabl...

2018-06-12 Thread xubo245
Github user xubo245 closed the pull request at: https://github.com/apache/spark/pull/20260 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...

2018-06-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21515#discussion_r194625766 --- Diff: dev/create-release/vote.tmpl --- @@ -0,0 +1,64 @@ +Please vote on releasing the following candidate as Apache Spark version {version}.

[GitHub] spark issue #19528: [SPARK-20393][WEBU UI][1.6] Strengthen Spark to prevent ...

2018-06-12 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19528 Jenkins test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...

2018-06-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21515#discussion_r194625981 --- Diff: dev/.rat-excludes --- @@ -106,3 +106,4 @@ spark-warehouse structured-streaming/* kafka-source-initial-offset-version-2.1.0.bin

[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...

2018-06-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21515#discussion_r194626614 --- Diff: dev/create-release/spark-rm/Dockerfile --- @@ -0,0 +1,89 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...

2018-06-12 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/21515#discussion_r194626379 --- Diff: dev/.rat-excludes --- @@ -106,3 +106,4 @@ spark-warehouse structured-streaming/* kafka-source-initial-offset-version-2.1.0.bin

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21319 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91685/ Test FAILed. ---

  1   2   3   4   5   6   7   8   >