[GitHub] spark issue #22653: [SPARK-25659][PYTHON][TEST] Test type inference specific...

2018-10-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22653 Let me open a JIRA after taking some more further looks. I checked some code places and looks it's not easy to support `None` for instance. ---

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22615 Hmmm... just want to raise a possible issue that maybe, just maybe, we should be hosting those jenkins configs in a repository that is owned by the ASF and writable by all Spark committers.

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223525420 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,75

[GitHub] spark pull request #22630: [SPARK-25497][SQL] Limit operation within whole s...

2018-10-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22630#discussion_r223525444 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -452,46 +452,73 @@ case class RangeExec(range:

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22614 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22614 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97126/ Test PASSed. ---

[GitHub] spark issue #22635: [SPARK-25591][PySpark][SQL] Avoid overwriting deserializ...

2018-10-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22635 @cloud-fan @gatorsmile @HyukjinKwon Thanks. Yes. As Pandas UDF has the same issue and it is fixed by this PR. --- - To

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223490770 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223497773 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,28 @@ trait TestPrematureExit { @volatile var

[GitHub] spark issue #22635: [SPARK-25591][PySpark][SQL] Avoid overwriting deserializ...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22635 How about pandas UDF? How about using RDD APIs? Do we face the same issues? --- - To unsubscribe, e-mail:

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22429 @zsxwing Please, have a look at the PR one more time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22656: [SPARK-25669][SQL] Check CSV header only when it exists

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22656 @HyukjinKwon Could you look at the PR, please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519103 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519430 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519277 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519608 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn...

2018-10-08 Thread ankuriitg
Github user ankuriitg commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r223519892 --- Diff: core/src/main/scala/org/apache/spark/internal/Logging.scala --- @@ -192,7 +211,15 @@ private[spark] object Logging {

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223518338 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223518841 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223518123 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223523341 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156

[GitHub] spark issue #22659: [SPARK-25623][SPARK-25624][SPARK-25625][TEST] Reduce tes...

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22659 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22653: [SPARK-25659][PYTHON][TEST] Test type inference s...

2018-10-08 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22653#discussion_r223534303 --- Diff: python/pyspark/sql/tests.py --- @@ -1149,6 +1149,75 @@ def test_infer_schema(self): result = self.spark.sql("SELECT l[0].a from test2

[GitHub] spark issue #22029: [SPARK-24395][SQL] IN operator should return NULL when c...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22029 @mgaido91 . The failure is relevant to this PR. Could you take a look? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-10-08 Thread aditanase
Github user aditanase commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r223491915 --- Diff: docs/running-on-kubernetes.md --- @@ -799,4 +815,168 @@ specific to Spark on Kubernetes. This sets the major Python version of the

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223491937 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,27 @@ trait TestPrematureExit { @volatile var

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22614 Also, https://github.com/apache/spark/pull/22614#discussion_r223172392 proposes a better and safer solution by introducing `exponential backoff with retries`. @kmanamcheri Could you update your

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Add base class for token rene...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r223493773 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/AbstractCredentialRenewer.scala --- @@ -0,0 +1,224 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-10-08 Thread aditanase
Github user aditanase commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r223493848 --- Diff: docs/running-on-kubernetes.md --- @@ -185,6 +185,22 @@ To use a secret through an environment variable use the following options to the

[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...

2018-10-08 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/22305 @BryanCutler Yes that was a typo :) Thanks! I am also +1 to support numpy data structure in addition to Pandas. So happy to discuss here or separately. ---

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223504756 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,28 @@ trait TestPrematureExit { @volatile

[GitHub] spark issue #22628: [SPARK-25641] Change the spark.shuffle.server.chunkFetch...

2018-10-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/22628 Is it supposed to the flaky ChunkFetchIntegrationSuite? http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.network.ChunkFetchIntegrationSuite_name=fetchFileChunk ---

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22614 **[Test build #97126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97126/testReport)** for PR 22614 at commit

[GitHub] spark issue #22630: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22630 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21669 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97128/ Test FAILed. ---

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 Yeah this does need to be in a public repo. apache/spark-jenkins-configurations or something. We can ask INFRA to create them. But, I'm not against just putting them in dev/ or something in the main

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21669 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 **[Test build #97128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97128/testReport)** for PR 21669 at commit

[GitHub] spark issue #22630: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22630 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22630: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22630 **[Test build #97136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97136/testReport)** for PR 22630 at commit

[GitHub] spark issue #22653: [SPARK-25659][PYTHON][TEST] Test type inference specific...

2018-10-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22653 Merged to master. Thank you @BryanCutler. Let me try to make sure I take further actions for related items. --- -

[GitHub] spark pull request #22659: [SPARK-25623][SPARK-25624][SPARK-25625][TEST] Red...

2018-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22659 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223535529 --- Diff: docs/running-on-kubernetes.md --- @@ -820,4 +820,37 @@ specific to Spark on Kubernetes. This sets the major Python version of the

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97129/ Test PASSed. ---

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread kmanamcheri
Github user kmanamcheri commented on the issue: https://github.com/apache/spark/pull/22614 @gatorsmile, @tejasapatil was reviewing the code before I added the new config option. I have asked him to review the new code. Lets see what his thoughts are on that. I have also asked him

[GitHub] spark issue #22654: [SPARK-25660][SQL] Fix for the backward slash as CSV fie...

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22654 @gatorsmile Please, take a look at this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97132/testReport)** for PR 22623 at commit

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97132/ Test FAILed. ---

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97134/testReport)** for PR 22623 at commit

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22615 @srowen sure, manually removing the failing jobs is one option... but since we auto-generate the job configs, any time we add a new branch they'll come back. i'd much rather do this

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97133/testReport)** for PR 22623 at commit

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223504565 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,27 @@ trait TestPrematureExit { @volatile

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223507242 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala --- @@ -63,7 +65,7 @@ private[spark] object PythonEvalType { */

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223507984 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -70,23 +70,25 @@ trait TestPrematureExit { mainObject:

[GitHub] spark pull request #22173: [SPARK-24355] Spark external shuffle server impro...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22173#discussion_r223513440 --- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java --- @@ -77,17 +82,54 @@ private static final

[GitHub] spark issue #22482: WIP - [SPARK-10816][SS] Support session window natively

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22482 **[Test build #97135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97135/testReport)** for PR 22482 at commit

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22615 > an unset var is just an empty string anyway. The issue is that if you call that function multiple times the variables might be set by a previous call. ---

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22615 consider me pinged. ;) i will need to do some refactoring in the jenkins job builder configs for this, so we'll definitely need to coordinate before this is merged. most likely

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21809 **[Test build #97130 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97130/testReport)** for PR 21809 at commit

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21809 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97130/ Test PASSed. ---

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21809 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22615 @srowen fair 'nuf... i'll create a jira for this tomorrow and we can hash out final design shite there (rather than overloading this PR). :) ---

[GitHub] spark issue #22482: WIP - [SPARK-10816][SS] Support session window natively

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22482 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97135/ Test PASSed. ---

[GitHub] spark issue #22482: WIP - [SPARK-10816][SS] Support session window natively

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22482 **[Test build #97135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97135/testReport)** for PR 22482 at commit

[GitHub] spark issue #22482: WIP - [SPARK-10816][SS] Support session window natively

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22482 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 **[Test build #97139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97139/testReport)** for PR 21669 at commit

[GitHub] spark pull request #22612: [SPARK-24958] Add executors' process tree total m...

2018-10-08 Thread rezasafi
Github user rezasafi commented on a diff in the pull request: https://github.com/apache/spark/pull/22612#discussion_r223486041 --- Diff: core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala --- @@ -0,0 +1,272 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3803/ ---

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223490214 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97132/testReport)** for PR 22623 at commit

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22627 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223506840 --- Diff: python/pyspark/worker.py --- @@ -154,6 +154,47 @@ def wrapped(*series): return lambda *a: (wrapped(*a), arrow_return_type)

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223505747 --- Diff: python/pyspark/worker.py --- @@ -154,6 +154,47 @@ def wrapped(*series): return lambda *a: (wrapped(*a), arrow_return_type)

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 I guess we've just pinged @shaneknapp ! But I figured the jobs would simply fail and could be removed at leisure. Yes, this mechanism is a little weird but may be the simplest thing here. I

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223524110 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,75

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97134/testReport)** for PR 22623 at commit

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 **[Test build #97138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97138/testReport)** for PR 21669 at commit

[GitHub] spark pull request #22029: [SPARK-24395][SQL] IN operator should return NULL...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22029#discussion_r223488428 --- Diff: sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql --- @@ -14,8 +14,25 @@ CREATE

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223491099 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223490811 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223491087 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223491101 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn...

2018-10-08 Thread NiharS
Github user NiharS commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r223512824 --- Diff: core/src/main/scala/org/apache/spark/internal/Logging.scala --- @@ -192,7 +211,15 @@ private[spark] object Logging {

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223525785 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,75 @@

[GitHub] spark issue #22653: [SPARK-25659][PYTHON][TEST] Test type inference specific...

2018-10-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22653 Sorry for late. @HyukjinKwon Yes, this LGTM. It is great we can follow up other issues like None and datetime.time in other JIRA. ---

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97134/ Test FAILed. ---

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97129/testReport)** for PR 22627 at commit

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21809 **[Test build #97130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97130/testReport)** for PR 21809 at commit

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/22627 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97131/testReport)** for PR 22627 at commit

[GitHub] spark pull request #22653: [SPARK-25659][PYTHON][TEST] Test type inference s...

2018-10-08 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22653#discussion_r223520528 --- Diff: python/pyspark/sql/tests.py --- @@ -1149,6 +1149,75 @@ def test_infer_schema(self): result = self.spark.sql("SELECT l[0].a from

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223524047 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156 @@

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22615 https://github.com/databricks/spark-jenkins-configurations/pull/47 --- - To unsubscribe, e-mail:

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97133/ Test FAILed. ---

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97133/testReport)** for PR 22623 at commit

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22614 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97119/ Test PASSed. ---

<    1   2   3   4   5   6   >