[spark] branch master updated: [SPARK-26625] Add oauthToken to spark.redaction.regex

This is an automated email from the ASF dual-hosted git repository. mcheah pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 01301d0 [SPARK-26625] Add oauthToken

[spark] branch master updated: [SPARK-25877][K8S] Move all feature logic to feature classes.

This is an automated email from the ASF dual-hosted git repository. mcheah pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a63e7b2 [SPARK-25877][K8S] Move all feature

spark git commit: [SPARK-26194][K8S] Auto generate auth secret for k8s apps.

red: Thu Dec 6 14:17:13 2018 -0800 Committer: mcheah Committed: Thu Dec 6 14:17:13 2018 -0800 -- .../org/apache/spark/SecurityManager.scala | 21 - .../org/apache/spark/SecurityManagerSuite.scala | 57 +++- d

[2/3] spark git commit: [SPARK-25876][K8S] Simplify kubernetes configuration types.

http://git-wip-us.apache.org/repos/asf/spark/blob/6be272b7/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala -- diff --git

[1/3] spark git commit: [SPARK-25876][K8S] Simplify kubernetes configuration types.

Repository: spark Updated Branches: refs/heads/master 8856e9f6a -> 6be272b75 http://git-wip-us.apache.org/repos/asf/spark/blob/6be272b7/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilderSuite.scala

[3/3] spark git commit: [SPARK-25876][K8S] Simplify kubernetes configuration types.

e19df193675cc4117763 Parents: 8856e9f Author: Marcelo Vanzin Authored: Fri Nov 30 16:23:37 2018 -0800 Committer: mcheah Committed: Fri Nov 30 16:23:37 2018 -0800 -- .../org/apache/spark/deploy/k8s/Config.scala| 17 +-

spark git commit: [SPARK-25957][K8S] Make building alternate language binding docker images optional

os/asf/spark/diff/9b48107f Branch: refs/heads/master Commit: 9b48107f9c84631e0ddaf0f2223296a3cbc16f83 Parents: 4aa9ccb Author: Nagaram Prasad Addepally Authored: Wed Nov 21 15:51:37 2018 -0800 Committer: mcheah Committed: Wed Nov 21 15:51:3

[2/2] spark git commit: [SPARK-25875][K8S] Merge code to set up driver command into a single step.

/asf/spark/diff/3404a73f Branch: refs/heads/master Commit: 3404a73f4cf7be37e574026d08ad5cf82cfac871 Parents: 7ea594e Author: Marcelo Vanzin Authored: Fri Nov 2 13:58:08 2018 -0700 Committer: mcheah Committed: Fri Nov 2 13:58:08

[1/2] spark git commit: [SPARK-25875][K8S] Merge code to set up driver command into a single step.

Repository: spark Updated Branches: refs/heads/master 7ea594e78 -> 3404a73f4 http://git-wip-us.apache.org/repos/asf/spark/blob/3404a73f/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/bindings/PythonDriverFeatureStepSuite.scala

spark git commit: [SPARK-25809][K8S][TEST] New K8S integration testing backends

sse Authored: Thu Nov 1 09:33:55 2018 -0700 Committer: mcheah Committed: Thu Nov 1 09:33:55 2018 -0700 -- .../k8s/SparkKubernetesClientFactory.scala | 5 + .../k8s/submit/LoggingPodStatusWatcher.scala|

[2/3] spark git commit: [SPARK-23429][CORE] Add executor memory metrics to heartbeat and expose in executors REST API

http://git-wip-us.apache.org/repos/asf/spark/blob/9241e1e7/core/src/test/resources/spark-events/application_1506645932520_24630151 -- diff --git a/core/src/test/resources/spark-events/application_1506645932520_24630151

[1/3] spark git commit: [SPARK-23429][CORE] Add executor memory metrics to heartbeat and expose in executors REST API

Repository: spark Updated Branches: refs/heads/master 458f5011b -> 9241e1e7e http://git-wip-us.apache.org/repos/asf/spark/blob/9241e1e7/core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala -- diff --git

[3/3] spark git commit: [SPARK-23429][CORE] Add executor memory metrics to heartbeat and expose in executors REST API

: http://git-wip-us.apache.org/repos/asf/spark/tree/9241e1e7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9241e1e7 Branch: refs/heads/master Commit: 9241e1e7e66574cfafa68791771959dfc39c9684 Parents: 458f501 Author: Edwina Lu Authored: Fri Sep 7 10:42:46 2018 -0700 Committer: mcheah

spark git commit: [SPARK-25262][K8S] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S

wip-us.apache.org/repos/asf/spark/diff/da6fa382 Branch: refs/heads/master Commit: da6fa3828bb824b65f50122a8a0a0d4741551257 Parents: 27d3b0a Author: Rob Vesse Authored: Thu Sep 6 16:18:59 2018 -0700 Committer: mcheah Committed: Thu Sep 6 16:18:59 2018 -0

spark git commit: [SPARK-25222][K8S] Improve container status logging

685 Parents: c84bc40 Author: Rob Vesse Authored: Thu Sep 6 16:15:11 2018 -0700 Committer: mcheah Committed: Thu Sep 6 16:15:11 2018 -0700 -- .../spark/deploy/k8s/KubernetesUtils.scala | 83 +++- .../k8s/sub

spark git commit: [SPARK-25264][K8S] Fix comma-delineated arguments passed into PythonRunner and RRunner

mit: e1d72f2c07ecd6f1880299e9373daa21cb032017 Parents: 32da87d Author: Ilan Filonenko Authored: Fri Aug 31 15:46:45 2018 -0700 Committer: mcheah Committed: Fri Aug 31 15:46:45 2018 -0700 -- .../deploy/k8s/features/bindings/PythonDriverFeatureStep.scala

spark git commit: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s

ark/tree/ba84bcb2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ba84bcb2 Branch: refs/heads/master Commit: ba84bcb2c4f73baf63782ff6fad5a607008c7cd2 Parents: da2dc69 Author: Ilan Filonenko Authored: Fri Aug 17 16:04:02 2018 -0700 Committer: mcheah Committed: Fri Aug 17 16:04:02 2018 -0

spark git commit: [SPARK-23984][K8S] Changed Python Version config to be camelCase

wip-us.apache.org/repos/asf/spark/tree/a791c29b Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a791c29b Branch: refs/heads/master Commit: a791c29bd824adadfb2d85594bc8dad4424df936 Parents: 717f58e Author: Ilan Filonenko Authored: Wed Aug 15 17:52:12 2018 -0700 Committer: mcheah Committed: Wed Aug 15

spark git commit: [SPARK-24960][K8S] explicitly expose ports on driver container

org/repos/asf/spark/tree/f5113ea8 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f5113ea8 Branch: refs/heads/master Commit: f5113ea8d79de724ec1579bc81a7abb61e44eeef Parents: 95a9d5e Author: Adelbert Chang Authored: Wed Aug 1 13:57:33 2018 -0700 Committer: mcheah Committed: Wed Aug 1 13:57

spark git commit: [SPARK-24963][K8S][TESTS] Add user-specified service account name for client mode test driver pod

pod in the client mode integration test ## How was this patch tested? Manual test against a custom Kubernetes cluster Author: mcheah Closes #21924 from mccheah/fix-service-account. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/com

spark git commit: [SPARK-24963][K8S][TESTS] Don't set service account name for client mode test

How was this patch tested? Test should continue running smoothly in Jenkins. Author: mcheah Closes #21900 from mccheah/fix-integration-test-service-account. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d6b7545b Tree: http://git-

spark git commit: [SPARK-23146][K8S][TESTS] Enable client mode integration test.

eck the integration test runs in the build. Author: mcheah Closes #21874 from mccheah/enable-client-mode-test. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0c83f718 Tree: http://git-wip-us.apache.org/repos/asf/spark/t

spark git commit: [SPARK-23146][K8S] Support client mode.

ing up their own connectivity. We also change the authentication configuration prefixes for client mode. ## How was this patch tested? Adding an integration test to exercise client mode support. Author: mcheah Closes #21748 from mccheah/k8s-client-mode. Project: http://git-wip-us.apache.org/re

spark git commit: [SPARK-24825][K8S][TEST] Kubernetes integration tests build the whole reactor

non-Kubernetes integration tests in the build, support specifying tags and tag all integration tests specifically with "k8s". Supply the k8s tag in the dev/dev-run-integration-tests.sh script. ## How was this patch tested? The build system will test this. Author: mcheah Closes #21800

spark git commit: [SPARK-23984][K8S][TEST] Added Integration Tests for PySpark on Kubernetes

mit: f1a99ad5825daf1b4cc275146ba8460cbcdf9701 Parents: a75571b Author: Ilan Filonenko Authored: Fri Jul 13 17:19:28 2018 -0700 Committer: mcheah Committed: Fri Jul 13 17:19:28 2018 -0700 -- .../k8s/integrationtest/KubernetesSuite.scala |

spark git commit: [SPARK-24683][K8S] Fix k8s no resource

tch tested? New integration test written to capture this case. Author: mcheah Closes #21660 from mccheah/fix-k8s-no-resource. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/42815548 Tree: http://git-wip-us.apache.org/re

spark git commit: Fix issue in 'docker-image-tool.sh'

b8dbfcc5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b8dbfcc5 Branch: refs/heads/branch-2.3 Commit: b8dbfcc572bf5f386d21ae0be7b3439d1abed860 Parents: 9d63e54 Author: Fabrizio Cucci Authored: Mon Jun 18 14:40:24 2018 -0700 Committer: mcheah Committed: Mon Jun 18 14:40:24 20

[1/2] spark git commit: [SPARK-24248][K8S] Use level triggering and state reconciliation in scheduling and lifecycle

Repository: spark Updated Branches: refs/heads/master 18cb0c079 -> 270a9a3ca http://git-wip-us.apache.org/repos/asf/spark/blob/270a9a3c/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocatorSuite.scala

[2/2] spark git commit: [SPARK-24248][K8S] Use level triggering and state reconciliation in scheduling and lifecycle

? Integration tests should test there's no regressions end to end. Unit tests to be updated, in particular focusing on different orderings of events, particularly accounting for when events come in unexpected ordering. Author: mcheah Closes #21366 from mccheah/event-queue-driven-scheduling

spark git commit: [SPARK-23010][K8S] Initial checkin of k8s integration tests.

f78 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f433ef78 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f433ef78 Branch: refs/heads/master Commit: f433ef786770e48e3594ad158ce9908f98ef0d9a Parents: b070ded Author: Sean Suchter Authored: Fri Jun 8 15:15:24 2018 -0700 Committer: mch

spark git commit: [SPARK-23984][K8S] Initial Python Bindings for PySpark on K8s

:34 2018 -0700 Committer: mcheah Committed: Fri Jun 8 11:18:34 2018 -0700 -- bin/docker-image-tool.sh| 23 ++-- .../org/apache/spark/deploy/SparkSubmit.scala | 14 ++- docs/running-on-kubernetes

spark git commit: [SPARK-23825][K8S] Requesting memory + memory overhead for pod memory

e Parents: 44a9f8e Author: David Vogelbacher <dvogelbac...@palantir.com> Authored: Mon Apr 2 12:00:37 2018 -0700 Committer: mcheah <mch...@palantir.com> Committed: Mon Apr 2 12:00:37 2018 -0700 -- .../de

[2/2] spark git commit: [SPARK-22839][K8S] Remove the use of init-container for downloading remote dependencies

Commit: f15906da153f139b698e192ec6f82f078f896f1e Parents: 4de638c Author: Ilan Filonenko <i...@cornell.edu> Authored: Mon Mar 19 11:29:56 2018 -0700 Committer: mcheah <mch...@palantir.com> Committed: Mon Mar 19 11:29:56 2018 -0700 -- bin/do

[1/2] spark git commit: [SPARK-22839][K8S] Remove the use of init-container for downloading remote dependencies

Repository: spark Updated Branches: refs/heads/master 4de638c19 -> f15906da1 http://git-wip-us.apache.org/repos/asf/spark/blob/f15906da/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/ClientSuite.scala