[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223523341 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223517953 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0 +1,238 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223513908 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala --- @@ -81,4 +83,35 @@ private[spark] object

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223519417 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBuilder.scala --- @@ -51,7 +67,25 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223513489 --- Diff: examples/src/main/scala/org/apache/spark/examples/HdfsTest.scala --- @@ -41,6 +41,8 @@ object HdfsTest { val end =

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223515094 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223516515 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0 +1,238 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223518860 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,75 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223515755 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223516199 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfExecutorFeatureStep.scala --- @@ -0,0 +1,58 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223514310 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,52 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223511748 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -646,8 +646,9 @@ private[spark] class SparkSubmit extends Logging {

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223515261 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223512267 --- Diff: docs/running-on-kubernetes.md --- @@ -820,4 +820,37 @@ specific to Spark on Kubernetes. This sets the major Python version of the docker

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223519651 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBuilderSuite.scala --- @@ -19,14 +19,18

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223517312 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0 +1,238 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223517875 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0 +1,238 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223518149 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0 +1,238 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223515400 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223517824 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0 +1,238 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223514752 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223512056 --- Diff: docs/running-on-kubernetes.md --- @@ -820,4 +820,37 @@ specific to Spark on Kubernetes. This sets the major Python version of the docker

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223518787 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,75 @@

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223512232 --- Diff: docs/running-on-kubernetes.md --- @@ -820,4 +820,37 @@ specific to Spark on Kubernetes. This sets the major Python version of the docker

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223520296 --- Diff: docs/security.md --- @@ -722,7 +722,85 @@ with encryption, at least. The Kerberos login will be periodically renewed using the provided

[GitHub] spark issue #22653: [SPARK-25659][PYTHON][TEST] Test type inference specific...

2018-10-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22653 Let me open a JIRA after taking some more further looks. I checked some code places and looks it's not easy to support `None` for instance. ---

[GitHub] spark pull request #22653: [SPARK-25659][PYTHON][TEST] Test type inference s...

2018-10-08 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22653#discussion_r223520528 --- Diff: python/pyspark/sql/tests.py --- @@ -1149,6 +1149,75 @@ def test_infer_schema(self): result = self.spark.sql("SELECT l[0].a from

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn...

2018-10-08 Thread ankuriitg
Github user ankuriitg commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r223519892 --- Diff: core/src/main/scala/org/apache/spark/internal/Logging.scala --- @@ -192,7 +211,15 @@ private[spark] object Logging {

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519608 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519103 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519430 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223519277 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223518841 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223518123 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-08 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r223518338 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala ---

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22615 @srowen sure, manually removing the failing jobs is one option... but since we auto-generate the job configs, any time we add a new branch they'll come back. i'd much rather do this

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22615 > an unset var is just an empty string anyway. The issue is that if you call that function multiple times the variables might be set by a previous call. ---

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/22615 consider me pinged. ;) i will need to do some refactoring in the jenkins job builder configs for this, so we'll definitely need to coordinate before this is merged. most likely

[GitHub] spark issue #22615: [SPARK-25016][BUILD][CORE] Remove support for Hadoop 2.6

2018-10-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22615 I guess we've just pinged @shaneknapp ! But I figured the jobs would simply fail and could be removed at leisure. Yes, this mechanism is a little weird but may be the simplest thing here. I

[GitHub] spark pull request #22173: [SPARK-24355] Spark external shuffle server impro...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22173#discussion_r223513440 --- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java --- @@ -77,17 +82,54 @@ private static final

[GitHub] spark issue #22482: WIP - [SPARK-10816][SS] Support session window natively

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22482 **[Test build #97135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97135/testReport)** for PR 22482 at commit

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn...

2018-10-08 Thread NiharS
Github user NiharS commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r223512824 --- Diff: core/src/main/scala/org/apache/spark/internal/Logging.scala --- @@ -192,7 +211,15 @@ private[spark] object Logging {

[GitHub] spark issue #22628: [SPARK-25641] Change the spark.shuffle.server.chunkFetch...

2018-10-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/22628 Is it supposed to the flaky ChunkFetchIntegrationSuite? http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.network.ChunkFetchIntegrationSuite_name=fetchFileChunk ---

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97134/testReport)** for PR 22623 at commit

[GitHub] spark pull request #22615: [SPARK-25016][BUILD][CORE] Remove support for Had...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22615#discussion_r223509836 --- Diff: dev/create-release/release-build.sh --- @@ -191,9 +191,17 @@ if [[ "$1" == "package" ]]; then make_binary_release() { NAME=$1

[GitHub] spark issue #22656: [SPARK-25669][SQL] Check CSV header only when it exists

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22656 @HyukjinKwon Could you look at the PR, please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22429 @zsxwing Please, have a look at the PR one more time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22654: [SPARK-25660][SQL] Fix for the backward slash as CSV fie...

2018-10-08 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/22654 @gatorsmile Please, take a look at this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97132/ Test FAILed. ---

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97132/testReport)** for PR 22623 at commit

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223507984 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -70,23 +70,25 @@ trait TestPrematureExit { mainObject:

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223507242 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala --- @@ -63,7 +65,7 @@ private[spark] object PythonEvalType { */

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223505747 --- Diff: python/pyspark/worker.py --- @@ -154,6 +154,47 @@ def wrapped(*series): return lambda *a: (wrapped(*a), arrow_return_type)

[GitHub] spark pull request #22305: [SPARK-24561][SQL][Python] User-defined window ag...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/22305#discussion_r223506840 --- Diff: python/pyspark/worker.py --- @@ -154,6 +154,47 @@ def wrapped(*series): return lambda *a: (wrapped(*a), arrow_return_type)

[GitHub] spark issue #22560: [SPARK-25547][Spark Core] Pluggable JDBC connection fact...

2018-10-08 Thread fsauer65
Github user fsauer65 commented on the issue: https://github.com/apache/spark/pull/22560 @gatorsmile did you get a chance to think about this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223504756 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,28 @@ trait TestPrematureExit { @volatile

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread devaraj-kavali
Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223504565 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,27 @@ trait TestPrematureExit { @volatile

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97133/testReport)** for PR 22623 at commit

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22627 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22635: [SPARK-25591][PySpark][SQL] Avoid overwriting deserializ...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22635 How about pandas UDF? How about using RDD APIs? Do we face the same issues? --- - To unsubscribe, e-mail:

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread kmanamcheri
Github user kmanamcheri commented on the issue: https://github.com/apache/spark/pull/22614 @gatorsmile, @tejasapatil was reviewing the code before I added the new config option. I have asked him to review the new code. Lets see what his thoughts are on that. I have also asked him

[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...

2018-10-08 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/22305 @BryanCutler Yes that was a typo :) Thanks! I am also +1 to support numpy data structure in addition to Pandas. So happy to discuss here or separately. ---

[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...

2018-10-08 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/22305 I think there is a typo in your example in the description ``` @pandas_udf('double', PandasUDFType.GROUPED_AGG) def avg(v): return v.mean() return avg ``` I think

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97131/testReport)** for PR 22627 at commit

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97131/ Test PASSed. ---

[GitHub] spark pull request #22614: [SPARK-25561][SQL] Implement a new config to cont...

2018-10-08 Thread kmanamcheri
Github user kmanamcheri commented on a diff in the pull request: https://github.com/apache/spark/pull/22614#discussion_r223498018 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala --- @@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223497773 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,28 @@ trait TestPrematureExit { @volatile var

[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97132/testReport)** for PR 22623 at commit

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22614 > a better and safer solution by introducing exponential backoff with retries I'm confused about that suggestion. What is being retried? The user query? The HMS call? What is changing in

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97131/testReport)** for PR 22627 at commit

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/22627 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Add base class for token rene...

2018-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r223493773 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/AbstractCredentialRenewer.scala --- @@ -0,0 +1,224 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-10-08 Thread aditanase
Github user aditanase commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r223493848 --- Diff: docs/running-on-kubernetes.md --- @@ -185,6 +185,22 @@ To use a secret through an environment variable use the following options to the

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223492379 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22614 Also, https://github.com/apache/spark/pull/22614#discussion_r223172392 proposes a better and safer solution by introducing `exponential backoff with retries`. @kmanamcheri Could you update your

[GitHub] spark pull request #22623: [SPARK-25636][CORE] spark-submit cuts off the fai...

2018-10-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22623#discussion_r223491937 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -74,20 +74,27 @@ trait TestPrematureExit { @volatile var

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-10-08 Thread aditanase
Github user aditanase commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r223491915 --- Diff: docs/running-on-kubernetes.md --- @@ -799,4 +815,168 @@ specific to Spark on Kubernetes. This sets the major Python version of the

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223490770 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223490811 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223491099 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopBootstrapUtil.scala --- @@ -0,0

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223491087 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223491101 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,214 @@ head(sql("select * from aggregates")) -#

[GitHub] spark issue #22614: [SPARK-25561][SQL] Implement a new config to control par...

2018-10-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22614 As @tejasapatil suggested above, this fallback is not suggested to be on in a prod setting. It could also impact the system health (e.g., the other concurrent queries that are querying the same

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread liyinan926
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223490214 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21669 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21669 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3803/ ---

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21669 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97129/ Test PASSed. ---

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22627 **[Test build #97129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97129/testReport)** for PR 22627 at commit

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21809 **[Test build #97130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97130/testReport)** for PR 21809 at commit

[GitHub] spark pull request #22029: [SPARK-24395][SQL] IN operator should return NULL...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22029#discussion_r223488428 --- Diff: sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql --- @@ -14,8 +14,25 @@ CREATE

[GitHub] spark issue #22029: [SPARK-24395][SQL] IN operator should return NULL when c...

2018-10-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22029 @mgaido91 . The failure is relevant to this PR. Could you take a look? --- - To unsubscribe, e-mail:

[GitHub] spark issue #21809: [SPARK-24851][UI] Map a Stage ID to it's Associated Job ...

2018-10-08 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21809 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

<    1   2   3   4   5   6   >