Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread Marcelo Vanzin
On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin wrote: >> >> Do the upgraded minikube/k8s versions break the current master client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: sin

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread Marcelo Vanzin
Do the upgraded minikube/k8s versions break the current master client version too? I'm not super concerned about 2.4 integration tests being broken for a little bit. It's very uncommon for new PRs to be open against branch-2.4 that would affect k8s. But I really don't want master to break. So if

[jira] [Resolved] (SPARK-26103) OutOfMemory error with large query plans

2019-03-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26103. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23169 [https

[jira] [Assigned] (SPARK-26103) OutOfMemory error with large query plans

2019-03-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26103: -- Assignee: Dave DeCaprio > OutOfMemory error with large query pl

[spark] branch master updated: [SPARK-26103][SQL] Limit the length of debug strings for query plans

2019-03-13 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 812ad55 [SPARK-26103][SQL] Limit the length

[jira] [Assigned] (LIVY-568) Upgrade jackson version from 2.9.5 to 2.9.8

2019-03-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-568: --- Assignee: Gyorgy Gal > Upgrade jackson version from 2.9.5 to 2.

[jira] [Assigned] (SPARK-26927) Race condition may cause dynamic allocation not working

2019-03-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26927: -- Assignee: liupengcheng > Race condition may cause dynamic allocation not work

[jira] [Resolved] (SPARK-26927) Race condition may cause dynamic allocation not working

2019-03-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26927. Resolution: Fixed Fix Version/s: 2.3.4 2.4.2

[spark] branch branch-2.3 updated: [SPARK-26927][CORE] Ensure executor is active when processing events in dynamic allocation manager.

2019-03-12 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new d0290ea [SPARK-26927][CORE] Ensure

[spark] branch branch-2.4 updated: [SPARK-26927][CORE] Ensure executor is active when processing events in dynamic allocation manager.

2019-03-12 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 432ea69 [SPARK-26927][CORE] Ensure

[spark] branch master updated: [SPARK-26927][CORE] Ensure executor is active when processing events in dynamic allocation manager.

2019-03-12 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d5cfe08 [SPARK-26927][CORE] Ensure executor

Re: Livy-0.6 release?

2019-03-12 Thread Marcelo Vanzin
shall we enable it by default or leave it disabled? > > Thanks > Saisai > > Jeff Zhang 于2019年3月12日周二 上午10:54写道: >> >> Thanks Marcelo, I can help to test it in zeppelin side which use livy as one >> interpreter. >> >> Marcelo Vanzin 于2019年3月12日周二 上

[jira] [Resolved] (LIVY-568) Upgrade jackson version from 2.9.5 to 2.9.8

2019-03-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-568. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 153 [https

[incubator-livy] branch master updated: [LIVY-568][BUILD] Upgrade jackson version to 2.9.8

2019-03-12 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-livy.git The following commit(s) were added to refs/heads/master by this push: new 5767789 [LIVY-568][BUILD] Upgrade

Re: Livy-0.6 release?

2019-03-11 Thread Marcelo Vanzin
celo. >> >> I see LIVY-551 is now part of the master. Hope to see Livy 0.6 perhaps soon. >> >> >> Thank you! >> Ruslan Dautkhanov >> >> >> On Tue, Feb 5, 2019 at 12:38 PM Marcelo Vanzin wrote: >>> >>> I think LIVY-551 is th

[jira] [Resolved] (SPARK-27073) Fix a race condition when handling of IdleStateEvent

2019-03-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27073. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23989 [https

[spark] branch master updated: [SPARK-27073][CORE] Fix a race condition when handling of IdleStateEvent

2019-03-11 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 064604a [SPARK-27073][CORE] Fix a race

[spark] branch master updated: [SPARK-27004][CORE] Remove stale HTTP auth code.

2019-03-11 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f1e223b [SPARK-27004][CORE] Remove stale HTTP

[jira] [Resolved] (SPARK-27004) Code for https uri authentication in Spark Submit needs to be removed

2019-03-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27004. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24033 [https

[jira] [Assigned] (SPARK-27004) Code for https uri authentication in Spark Submit needs to be removed

2019-03-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27004: -- Assignee: Marcelo Vanzin > Code for https uri authentication in Spark Submit ne

[jira] [Resolved] (SPARK-27129) Add JSON event serialization methods in JSONUtils for blacklisted events

2019-03-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27129. Resolution: Invalid Unless you can show why this (or in this case the lack

[jira] [Assigned] (SPARK-27061) Expose Driver UI port on driver service to access UI using service

2019-03-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27061: -- Assignee: Chandu Kavar > Expose Driver UI port on driver service to access UI us

[spark] branch master updated: [SPARK-27061][K8S] Expose Driver UI port on driver service to access …

2019-03-11 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d4542a8 [SPARK-27061][K8S] Expose Driver UI

[jira] [Resolved] (SPARK-27061) Expose Driver UI port on driver service to access UI using service

2019-03-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27061. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23990 [https

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789766#comment-16789766 ] Marcelo Vanzin commented on SPARK-26998: There are 3 ways to solve this: pipe, file, or env

[jira] [Resolved] (SPARK-27044) Maven dependency resolution does not support classifiers

2019-03-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27044. Resolution: Duplicate > Maven dependency resolution does not support classifi

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

2019-03-08 Thread Marcelo Vanzin
t; > Should we create a new rc7? > > DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, > Inc > > > On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin > > wrote: > > > > I personally find it a little weird to not have the commit in branch-2.

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

2019-03-08 Thread Marcelo Vanzin
I personally find it a little weird to not have the commit in branch-2.4. Not that this would happen, but if the v2.4.1-rc6 tag is overwritten (e.g. accidentally) then you lose the reference to that commit, and then the exact commit from which the rc was generated is lost. On Fri, Mar 8, 2019 at

[jira] [Updated] (SPARK-26604) Register channel for stream request

2019-03-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26604: --- Fix Version/s: 2.4.2 2.3.4 > Register channel for stream requ

[spark] branch branch-2.3 updated: [SPARK-26604][CORE][BACKPORT-2.4] Clean up channel registration for StreamManager

2019-03-07 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new c45f8da [SPARK-26604][CORE][BACKPORT

[spark] branch branch-2.4 updated: [SPARK-26604][CORE][BACKPORT-2.4] Clean up channel registration for StreamManager

2019-03-07 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 216eeec [SPARK-26604][CORE][BACKPORT

[jira] [Created] (SPARK-27094) Thread interrupt being swallowed while launching executors in YarnAllocator

2019-03-07 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-27094: -- Summary: Thread interrupt being swallowed while launching executors in YarnAllocator Key: SPARK-27094 URL: https://issues.apache.org/jira/browse/SPARK-27094

[jira] [Assigned] (SPARK-27075) Sorting table column in SQL WEBUI page throws 'IllegalArgumentException'

2019-03-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27075: -- Assignee: Shahid K I > Sorting table column in SQL WEBUI page thr

[jira] [Resolved] (SPARK-27075) Sorting table column in SQL WEBUI page throws 'IllegalArgumentException'

2019-03-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27075. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23994 [https

[spark] branch master updated: [SPARK-27075] Remove duplicate execution tag parameters from the url, when accessing the execution table in the SQL page

2019-03-07 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 713646d [SPARK-27075] Remove duplicate

[jira] [Resolved] (SPARK-27022) Kafka delegation token support

2019-03-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27022. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23929 [https

[spark] branch master updated: [SPARK-27022][DSTREAMS] Add kafka delegation token support.

2019-03-07 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 98a8725 [SPARK-27022][DSTREAMS] Add kafka

[jira] [Assigned] (SPARK-27022) Kafka delegation token support

2019-03-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27022: -- Assignee: Gabor Somogyi > Kafka delegation token supp

[jira] [Resolved] (SPARK-27019) Spark UI's SQL tab shows inconsistent values

2019-03-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27019. Resolution: Fixed Fix Version/s: 2.4.2 3.0.0 Issue resolved

[jira] [Assigned] (SPARK-27019) Spark UI's SQL tab shows inconsistent values

2019-03-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27019: -- Assignee: Shahid K I > Spark UI's SQL tab shows inconsistent val

[spark] branch master updated: [SPARK-27019][SQL][WEBUI] onJobStart happens after onExecutionEnd shouldn't overwrite kvstore

2019-03-06 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 62fd133 [SPARK-27019][SQL][WEBUI] onJobStart

[jira] [Commented] (SPARK-27072) Changing the parameter value of completedJob.sort to X prints stacktrace in sparkWebUI

2019-03-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785889#comment-16785889 ] Marcelo Vanzin commented on SPARK-27072: I'd expect an error 401 here, but you get a status 200

[jira] [Resolved] (SPARK-27015) spark-submit does not properly escape arguments sent to Mesos dispatcher

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27015. Resolution: Fixed Assignee: Martin Loncaric Fix Version/s: (was: 2.5.0

[spark] branch master updated: [SPARK-27015][MESOS] properly escape mesos scheduler arguments

2019-03-05 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0ba1954 [SPARK-27015][MESOS] properly escape

[jira] [Resolved] (SPARK-27021) Leaking Netty event loop group for shuffle chunk fetch requests

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27021. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23930 [https

[spark] branch master updated: [SPARK-27021][CORE] Cleanup of Netty event loop group for shuffle chunk fetch requests

2019-03-05 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 5668c42 [SPARK-27021][CORE] Cleanup of Netty

[jira] [Assigned] (SPARK-27021) Leaking Netty event loop group for shuffle chunk fetch requests

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27021: -- Assignee: Attila Zsolt Piros > Leaking Netty event loop group for shuffle ch

[jira] [Resolved] (SPARK-26928) Add driver CPU Time to the metrics system

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26928. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23838 [https

[spark] branch master updated: [SPARK-26928][CORE] Add driver CPU Time to the metrics system

2019-03-05 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 25d2850 [SPARK-26928][CORE] Add driver CPU Time

[jira] [Assigned] (SPARK-26928) Add driver CPU Time to the metrics system

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26928: -- Assignee: Luca Canali > Add driver CPU Time to the metrics sys

[jira] [Resolved] (SPARK-27012) Storage tab shows rdd details even after executor ended

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27012. Resolution: Fixed Assignee: Ajith S Fix Version/s: 3.0.0 > Storage

[spark] branch master updated: [SPARK-27012][CORE] Storage tab shows rdd details even after executor ended

2019-03-05 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6207360 [SPARK-27012][CORE] Storage tab shows

[jira] [Resolved] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27059. Resolution: Invalid > spark-submit on kubernetes cluster does not recognise k8s --mas

[jira] [Commented] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784762#comment-16784762 ] Marcelo Vanzin commented on SPARK-27059: Sounds like a problem with your system. Maybe your PATH

[spark] branch master updated: [SPARK-26592][SS][DOC] Add Kafka proxy user caveat to documentation

2019-03-05 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b99610e [SPARK-26592][SS][DOC] Add Kafka proxy

[jira] [Commented] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784674#comment-16784674 ] Marcelo Vanzin commented on SPARK-27059: You're most probably using a version of Spark that does

[spark] branch master updated: [SPARK-25689][FOLLOW-UP][CORE] Get proxy user's delegation tokens

2019-03-04 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 827d371 [SPARK-25689][FOLLOW-UP][CORE] Get

[jira] [Assigned] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24120: -- Assignee: Marcelo Vanzin > Show `Jobs` page when `jobId` is miss

[jira] [Assigned] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24120: -- Assignee: (was: Marcelo Vanzin) > Show `Jobs` page when `jobId` is miss

[jira] [Assigned] (SPARK-25564) Add output bytes metrics for each Executor

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25564: -- Assignee: Marcelo Vanzin > Add output bytes metrics for each Execu

[jira] [Assigned] (SPARK-25564) Add output bytes metrics for each Executor

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25564: -- Assignee: Marcelo Vanzin > Add output bytes metrics for each Execu

[jira] [Assigned] (SPARK-25564) Add output bytes metrics for each Executor

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25564: -- Assignee: (was: Marcelo Vanzin) > Add output bytes metrics for each Execu

[jira] [Assigned] (SPARK-25564) Add output bytes metrics for each Executor

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25564: -- Assignee: (was: Marcelo Vanzin) > Add output bytes metrics for each Execu

[jira] [Assigned] (SPARK-26792) Apply custom log URL to Spark UI

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26792: -- Assignee: Jungtaek Lim > Apply custom log URL to Spark

[jira] [Resolved] (SPARK-26792) Apply custom log URL to Spark UI

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26792. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23790 [https

[spark] branch master updated: [SPARK-26792][CORE] Apply custom log URL to Spark UI

2019-03-04 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d5bda2c [SPARK-26792][CORE] Apply custom log

[jira] [Assigned] (SPARK-25681) Delegation Tokens fetched twice upon HadoopFSDelegationTokenProvider creation

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25681: -- Assignee: (was: Marcelo Vanzin) > Delegation Tokens fetched twice u

[jira] [Resolved] (SPARK-26995) Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when using snappy

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26995. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23898 [https

[jira] [Assigned] (SPARK-25681) Delegation Tokens fetched twice upon HadoopFSDelegationTokenProvider creation

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25681: -- Assignee: Marcelo Vanzin > Delegation Tokens fetched twice u

[jira] [Assigned] (SPARK-25750) Integration Testing for Kerberos Support for Spark on Kubernetes

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25750: -- Assignee: Marcelo Vanzin > Integration Testing for Kerberos Support for Sp

[jira] [Assigned] (SPARK-25750) Integration Testing for Kerberos Support for Spark on Kubernetes

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25750: -- Assignee: (was: Marcelo Vanzin) > Integration Testing for Kerberos Supp

[jira] [Assigned] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24135: -- Assignee: Marcelo Vanzin > [K8s] Executors that fail to start up because of i

[jira] [Assigned] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24135: -- Assignee: (was: Marcelo Vanzin) > [K8s] Executors that fail to start up beca

[jira] [Assigned] (SPARK-26995) Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when using snappy

2019-03-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26995: -- Assignee: Luca Canali > Running Spark in Docker image with Alpine Linux 3.9.0 thr

[spark] branch master updated: [SPARK-26995][K8S] Make ld-linux-x86-64.so.2 visible to snappy native library under /lib in docker image with Alpine Linux

2019-03-04 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f13ea15 [SPARK-26995][K8S] Make ld-linux-x86-64

Re: [VOTE] Redirect github notifications to issues@

2019-03-04 Thread Marcelo Vanzin
FYI, it should all be done now, so if you want to get notifications, either sign up to issues@, or watch the repo on github. On Fri, Feb 22, 2019 at 9:49 AM Marcelo Vanzin wrote: > > Jira link: https://issues.apache.org/jira/browse/INFRA-17892 > > On Fri, Feb 22, 2019 at 9:44 AM Ma

[jira] [Resolved] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2

2019-03-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26997. Resolution: Fixed Target Version/s: 3.0.0 I reverted the client upgrade and re

[spark] branch master updated: [K8S][MINOR] Log minikube version when running integration tests.

2019-03-01 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9f16af6 [K8S][MINOR] Log minikube version when

[jira] [Resolved] (SPARK-26774) Document threading concerns in TaskSchedulerImpl

2019-02-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26774. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23874 [https

[jira] [Assigned] (SPARK-26774) Document threading concerns in TaskSchedulerImpl

2019-02-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26774: -- Assignee: Imran Rashid > Document threading concerns in TaskSchedulerI

[spark] branch master updated: [SPARK-26774][CORE] Update some docs on TaskSchedulerImpl.

2019-02-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c8e7eb1 [SPARK-26774][CORE] Update some docs

[spark] branch master updated: [SPARK-26895][CORE][FOLLOW-UP] Uninitializing log after `prepareSubmitEnvironment` in SparkSubmit

2019-02-27 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6e31ccf [SPARK-26895][CORE][FOLLOW-UP

[jira] [Assigned] (SPARK-27002) Latest kafka delegation token not always picked up

2019-02-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27002: -- Assignee: Gabor Somogyi > Latest kafka delegation token not always picked

[jira] [Resolved] (SPARK-27002) Latest kafka delegation token not always picked up

2019-02-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27002. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23906 [https

[spark] branch master updated: [SPARK-27002][SS] Get kafka delegation tokens right before consumer/producer created

2019-02-27 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 76e0b6b [SPARK-27002][SS] Get kafka delegation

[jira] [Assigned] (SPARK-24063) Control maximum epoch backlog

2019-02-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24063: -- Assignee: Gabor Somogyi > Control maximum epoch back

[jira] [Resolved] (SPARK-24063) Control maximum epoch backlog

2019-02-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24063. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23156 [https

[spark] branch master updated: [SPARK-24063][SS] Add maximum epoch queue threshold for ContinuousExecution

2019-02-27 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c4bbfd1 [SPARK-24063][SS] Add maximum epoch

[jira] [Assigned] (SPARK-24736) --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.

2019-02-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24736: -- Assignee: Marcelo Vanzin > --py-files not functional for non local URLs. It appe

[jira] [Resolved] (SPARK-24736) --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.

2019-02-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24736. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23793 [https

[spark] branch master updated: [SPARK-24736][K8S] Let spark-submit handle dependency resolution.

2019-02-27 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a6ddc9d [SPARK-24736][K8S] Let spark-submit

[jira] [Resolved] (SPARK-22860) Spark workers log ssl passwords passed to the executors

2019-02-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22860. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23820 [https

[jira] [Assigned] (SPARK-22860) Spark workers log ssl passwords passed to the executors

2019-02-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22860: -- Assignee: Jungtaek Lim > Spark workers log ssl passwords passed to the execut

[spark] branch master updated: [SPARK-22860][CORE][YARN] Redact command line arguments for running Driver and Executor before logging (standalone and YARN)

2019-02-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c17150a [SPARK-22860][CORE][YARN] Redact

[jira] [Reopened] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

2019-02-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-25590: Assignee: (was: Jiaxin Shan) Reopened since the patch was reverted. > kuberne

[jira] [Reopened] (SPARK-26742) Bump Kubernetes Client Version to 4.1.1

2019-02-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-26742: Assignee: (was: Jiaxin Shan) Reopened since we had to revert the patch. > B

[spark] branch master updated: Revert "[SPARK-26742][K8S] Update Kubernetes-Client version to 4.1.2"

2019-02-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new afbff64 Revert "[SPARK-26742][K8S] U

[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2

2019-02-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778376#comment-16778376 ] Marcelo Vanzin commented on SPARK-26997: It's linked in the description. I can revert the revert

[jira] [Commented] (SPARK-26997) k8s integration tests failing after client upgraded to 4.1.2

2019-02-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778240#comment-16778240 ] Marcelo Vanzin commented on SPARK-26997: I'm using virtualbox. > k8s integration tests fail

[commons-crypto] branch master updated: JNA version is updated to 4.4.0 (#93)

2019-02-26 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/commons-crypto.git The following commit(s) were added to refs/heads/master by this push: new 7b7a36d JNA version is updated

<    2   3   4   5   6   7   8   9   10   11   >