[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91570/ Test FAILed. ---

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21504 **[Test build #91570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91570/testReport)** for PR 21504 at commit [`421e16b`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #21277: [SPARK-24217][ML]Power Iteration Clustering is not displ...

2018-06-08 Thread shahidki31
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/21277 Closing the PR due to the discussions in the JIRA, https://issues.apache.org/jira/browse/SPARK-15784 and the PR https://github.com/apache/spark/pull/21493 --- -

[GitHub] spark pull request #21277: [SPARK-24217][ML]Power Iteration Clustering is no...

2018-06-08 Thread shahidki31
Github user shahidki31 closed the pull request at: https://github.com/apache/spark/pull/21277 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20697 **[Test build #91576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91576/testReport)** for PR 20697 at commit [`f67d582`](https://github.com/apache/spark/commit/f6

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-08 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r194150705 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -0,0 +1,2

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-08 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r194150614 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala --- @@ -0,0 +1,88 @@ +

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-08 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r194150624 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala --- @@ -0,0 +1,88 @@ +

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91569/ Test PASSed. ---

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21508 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21508 **[Test build #91569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91569/testReport)** for PR 21508 at commit [`f174263`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #21468: [SPARK-22151] : PYTHONPATH not picked up from the...

2018-06-08 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/21468#discussion_r194148280 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -813,8 +813,14 @@ private[spark] class Client(

[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

2018-06-08 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21247#discussion_r194147472 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -138,3 +121,40 @@ private[sql] class JSONOptions(

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread yuchaoran2011
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 I'm not confused about anything. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 You said: "even when LOCATION is not specified, Sentry would still not allow the table to be created." I said that if that's the case, then your user doesn't have the needed permission. Beca

[GitHub] spark pull request #19498: [SPARK-17756][PYTHON][STREAMING] Workaround to av...

2018-06-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19498 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21360: [SPARK-24288] Enable preventing predicate pushdown

2018-06-08 Thread TomaszGaweda
Github user TomaszGaweda commented on the issue: https://github.com/apache/spark/pull/21360 I've tesed it with my application that had problem with predicate pushdowns to database. Looks good, performance is degradated a bit, but it was previously ran on Spark 2.3, not 2.4. However, m

[GitHub] spark issue #19498: [SPARK-17756][PYTHON][STREAMING] Workaround to avoid ret...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19498 Thanks @rxin, @tdas, @holdenk and @felixcheung. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additi

[GitHub] spark issue #19498: [SPARK-17756][PYTHON][STREAMING] Workaround to avoid ret...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19498 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #19498: [SPARK-17756][PYTHON][STREAMING] Workaround to avoid ret...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19498 Thanks for reviewing this @rxin. Will merge this only to master given the concerns here. --- - To unsubscribe, e-mail: revie

[GitHub] spark pull request #21509: [SPARK-24489]Check for invalid input type of weig...

2018-06-08 Thread shahidki31
Github user shahidki31 commented on a diff in the pull request: https://github.com/apache/spark/pull/21509#discussion_r194144115 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/PowerIterationClustering.scala --- @@ -166,6 +166,7 @@ class PowerIterationClustering priv

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21508 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread yuchaoran2011
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 You said "If that's the case then your user does not have permission to create the table." What I'm saying is the user did have the permission. --- --

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91568/ Test PASSed. ---

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21508 **[Test build #91568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91568/testReport)** for PR 21508 at commit [`46c4a55`](https://github.com/apache/spark/commit/4

[GitHub] spark pull request #21465: [SPARK-24333][ML][PYTHON]Add fit with validation ...

2018-06-08 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/21465#discussion_r194143768 --- Diff: python/pyspark/ml/classification.py --- @@ -1251,26 +1256,33 @@ class GBTClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-06-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21092 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 Yes, that's exactly what I said. I'm not sure what part is confusing you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread yuchaoran2011
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 I can double check. But I remember clearly that the user did have all the necessary privileges. Without this patch, a permission error was thrown. With this patch it worked fine. The CREATE TA

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-06-08 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 There's no design doc. I didn't feel the change was big enough to warrant one. 1. Currently there is no spill-over to disk. If the range is too big, users can switch this off and use the m

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21082 Yea, let's leave this open for few more days in case someone has more comments. --- - To unsubscribe, e-mail: reviews-unsubs

[GitHub] spark issue #21467: [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration ...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21467 Seems fine. will take a look before merginf this in. Thanks for working on this. --- - To unsubscribe, e-mail: reviews-unsub

[GitHub] spark issue #21482: [SPARK-24393][SQL] SQL builtin: isinf

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21482 **[Test build #91575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91575/testReport)** for PR 21482 at commit [`559900a`](https://github.com/apache/spark/commit/55

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21082 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3858/ Tes

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21082 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 If that's the case then your user does not have permission to create the table. The issue that my patch works around is a bug in Sentry where creating a table *with the location specified to the same

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-06-08 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21082 Thanks @HyukjinKwon for your review! @ueshin Do you want to take another look too? --- - To unsubscribe, e-mail: reviews-unsu

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread yuchaoran2011
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 @vanzin But in Spark SQL, even when LOCATION is not specified, Sentry would still not allow the table to be created. --- -

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-06-08 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r194133638 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -424,6 +424,21 @@ abstract class SparkStrategies extends Q

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-06-08 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r194133573 --- Diff: python/pyspark/worker.py --- @@ -128,6 +128,17 @@ def wrapped(*series): return lambda *a: (wrapped(*a), arrow_return_type)

[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...

2018-06-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 I'm kinda neutral on what should happen, except maybe for filing a Sentry bug. I'll let @cloud-fan decide whether this makes sense in Spark. For the Sentry bug, there's an easy way to reprodu

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21082 **[Test build #91574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91574/testReport)** for PR 21082 at commit [`6350408`](https://github.com/apache/spark/commit/63

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-08 Thread ssuchter
Github user ssuchter commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r194133234 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -0,0 +1,2

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

2018-06-08 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21511 > But I am concerned about it setting a precedent for having to change Spark's source code every time you want to request a new resource type. I am very familiar with the space of machine learning a

[GitHub] spark issue #21283: [SPARK-24224][ML-Examples]Java example code for Power It...

2018-06-08 Thread shahidki31
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/21283 Thanks @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-06-08 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r194130366 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala --- @@ -34,7 +34,12 @@ object PythonUDF { e.isI

[GitHub] spark issue #21248: [SPARK-24191][ML]Scala Example code for Power Iteration ...

2018-06-08 Thread shahidki31
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/21248 Thanks @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #21119: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

2018-06-08 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/21119 @mengxr @WeichenXu123 I will close this one and submit a new PR soon. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr..

[GitHub] spark pull request #21119: [SPARK-19826][ML][PYTHON]add spark.ml Python API ...

2018-06-08 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/21119 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19498: [SPARK-17756][PYTHON][STREAMING] Workaround to avoid ret...

2018-06-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/19498 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.o

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-06-08 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r194129021 --- Diff: python/pyspark/sql/tests.py --- @@ -5181,6 +5190,235 @@ def test_invalid_args(self): 'mixture.*aggregate function.*grou

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r194128800 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala --- @@ -0,0 +1,88 @@ +/

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r194128687 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala --- @@ -0,0 +1,88 @@ +/

[GitHub] spark pull request #21511: [SPARK-24491][Kubernetes] Configuration support f...

2018-06-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21511#discussion_r194128417 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala --- @@ -172,7 +184,7 @@ p

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 yea, I generally agree with that and I feel in the same way. I think I had a talk about this @gatorsmile and @cloud-fan multiple times. Here is my thought: we should not make a configuration onl

[GitHub] spark issue #21119: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

2018-06-08 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/21119 @huaxingao Create a new PR is better I think. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additio

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-08 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21427 @HyukjinKwon I agree with you 99% people will mostly certainly not use the config. I think @gatorsmile 's concern is that in the rare case that some people are actually depending on the existing

[GitHub] spark issue #21481: [SPARK-24452][SQL][Core] Avoid possible overflow in int ...

2018-06-08 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21481 Since it is Java bytecode analysis, it is available for Scala code, too. In my quick test, findBugs overlooked a possible overflow. On the other hand, findBugs found another redundant null check.

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21092 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3712/ ---

[GitHub] spark pull request #20697: [SPARK-23010][k8s] Initial checkin of k8s integra...

2018-06-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/20697#discussion_r194124004 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -0,0 +1

[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21247#discussion_r194123992 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -138,3 +121,40 @@ private[sql] class JSONOptions(

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21092 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3712/ --- --

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21092 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3857/ Tes

[GitHub] spark issue #21119: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

2018-06-08 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/21119 @mengxr Sorry for the delay. I will submit an update later today. Do you want me to close this PR and do a new one? or just update this PR? --- -

[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

2018-06-08 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21247#discussion_r194117346 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala --- @@ -138,3 +121,40 @@ private[sql] class JSONOptions(

[GitHub] spark pull request #21247: [SPARK-24190] Separating JSONOptions for read

2018-06-08 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21247#discussion_r194119938 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2427,4 +2427,14 @@ class JsonSuite extends QueryT

[GitHub] spark pull request #21482: [SPARK-24393][SQL] SQL builtin: isinf

2018-06-08 Thread NihalHarish
Github user NihalHarish commented on a diff in the pull request: https://github.com/apache/spark/pull/21482#discussion_r194119670 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullExpressionsSuite.scala --- @@ -56,6 +56,16 @@ class NullExpressions

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21483 Thank you @felixcheung, @mengxr and @BryanCutler for reviewing this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 I'm okay if that's only the way to get through here. but I must say I wonder who's going to intentionally switch this off though. This now sounds more like a bug or a design issue to be fixed to

[GitHub] spark issue #21481: [SPARK-24452][SQL][Core] Avoid possible overflow in int ...

2018-06-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21481 is findBugs available for scala code as well? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-06-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21109 do we have a design doc? I have a couple of high-level questions: 1. what if the range is big and the queue OOM? 2. can't we apply it with cartesian join if there is no equal condition? and

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21092 **[Test build #91573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91573/testReport)** for PR 21092 at commit [`a61d897`](https://github.com/apache/spark/commit/a6

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-06-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r194117385 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesConfSuite.scala --- @@ -88,15 +90,42 @@ class Kubernete

[GitHub] spark issue #21468: [SPARK-22151] : PYTHONPATH not picked up from the spark....

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21468 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21045 **[Test build #91572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91572/testReport)** for PR 21045 at commit [`8d3a838`](https://github.com/apache/spark/commit/8d

[GitHub] spark issue #21468: [SPARK-22151] : PYTHONPATH not picked up from the spark....

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21468 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91571/ Test PASSed. ---

[GitHub] spark issue #21468: [SPARK-22151] : PYTHONPATH not picked up from the spark....

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21468 **[Test build #91571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91571/testReport)** for PR 21468 at commit [`5e733ae`](https://github.com/apache/spark/commit/5

[GitHub] spark pull request #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports su...

2018-06-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21483 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-06-08 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r194109752 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesConfSuite.scala --- @@ -87,11 +89,37 @@ class KubernetesCo

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-06-08 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r194112328 --- Diff: docs/running-on-kubernetes.md --- @@ -624,4 +624,20 @@ specific to Spark on Kubernetes. spark.kubernetes.executor.secrets.ENV_VAR=spark-sec

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-06-08 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r194113403 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesConfSuite.scala --- @@ -88,15 +90,42 @@ class KubernetesCo

[GitHub] spark issue #21119: [SPARK-19826][ML][PYTHON]add spark.ml Python API for PIC

2018-06-08 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/21119 @huaxingao Any updates? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-08 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/21483 LGTM. Merged into master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-ma

[GitHub] spark issue #21468: [SPARK-22151] : PYTHONPATH not picked up from the spark....

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21468 **[Test build #91571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91571/testReport)** for PR 21468 at commit [`5e733ae`](https://github.com/apache/spark/commit/5e

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-08 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r194109519 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,173 @@ case class MapKeys(ch

[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-06-08 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r194107584 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -154,6 +176,24 @@ private[spark] object Config e

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21045 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21045 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91565/ Test FAILed. ---

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21045 **[Test build #91565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91565/testReport)** for PR 21045 at commit [`bbc20ee`](https://github.com/apache/spark/commit/b

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-08 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r194098947 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala --- @@ -84,7 +86,28 @@ class StopWordsRemover @Since("1.5.0") (@Since("1.5.

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-08 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r194099298 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala --- @@ -84,7 +86,28 @@ class StopWordsRemover @Since("1.5.0") (@Since("1.5.

[GitHub] spark pull request #21502: [SPARK-22575][SQL] Add destroy to Dataset

2018-06-08 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21502#discussion_r194101781 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala --- @@ -153,6 +154,23 @@ class BroadcastJoinSuite extends

[GitHub] spark pull request #21504: [SPARK-24479][SS] Added config for registering st...

2018-06-08 Thread arunmahadevan
Github user arunmahadevan commented on a diff in the pull request: https://github.com/apache/spark/pull/21504#discussion_r194101270 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala --- @@ -55,6 +57,19 @@ class StreamingQueryManager priva

[GitHub] spark issue #21504: [SPARK-24479][SS] Added config for registering streaming...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21504 **[Test build #91570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91570/testReport)** for PR 21504 at commit [`421e16b`](https://github.com/apache/spark/commit/42

[GitHub] spark issue #21512: [minor][WEB UI] Spark web ui auto refresh every x second...

2018-06-08 Thread tomhanetz
Github user tomhanetz commented on the issue: https://github.com/apache/spark/pull/21512 @abellina No, the page stays at the same view as it was before the refresh. It is the same functionality as hitting the browser refresh button manually. --- -

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-08 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21427 I ran into @ueshin and @gatorsmile at the Summit. It seems the preferable way to move forward is to having a configuration to fall back to the existing behavior and change the default behavior to

[GitHub] spark pull request #21504: [SPARK-24479][SS] Added config for registering st...

2018-06-08 Thread arunmahadevan
Github user arunmahadevan commented on a diff in the pull request: https://github.com/apache/spark/pull/21504#discussion_r194100709 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala --- @@ -55,6 +57,19 @@ class StreamingQueryManager priva

[GitHub] spark issue #21508: [SPARK-24488] [SQL] Fix issue when generator is aliased ...

2018-06-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21508 **[Test build #91569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91569/testReport)** for PR 21508 at commit [`f174263`](https://github.com/apache/spark/commit/f1

[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...

2018-06-08 Thread bkrieger
Github user bkrieger commented on a diff in the pull request: https://github.com/apache/spark/pull/21508#discussion_r194082850 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1568,11 +1568,13 @@ class Analyzer( expr.fi

<    1   2   3   4   5   6   7   8   >