[spark] branch branch-2.4 updated: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new d9403e4 [SPARK-26726] Synchronize

[spark] branch master updated: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f4a17e9 [SPARK-26726] Synchronize the amount

[jira] [Commented] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756602#comment-16756602 ] Marcelo Vanzin commented on SPARK-23155: For those looking the patch is actually linked from

[jira] [Resolved] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23155. Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 3.0.0 > Y

[jira] [Resolved] (SPARK-26784) Allow running driver pod as provided user

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26784. Resolution: Won't Fix As far as I know you can do that with pod templates. So no point

[jira] [Created] (SPARK-26788) Remove SchedulerExtensionService

2019-01-30 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26788: -- Summary: Remove SchedulerExtensionService Key: SPARK-26788 URL: https://issues.apache.org/jira/browse/SPARK-26788 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-26737) Executor/Task STDERR & STDOUT log urls are not correct in Yarn deployment mode

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26737: --- Fix Version/s: 3.0.0 > Executor/Task STDERR & STDOUT log urls are not correct

[jira] [Resolved] (SPARK-26737) Executor/Task STDERR & STDOUT log urls are not correct in Yarn deployment mode

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26737. Resolution: Fixed Assignee: Jungtaek Lim The patch for SPARK-26311 ended up fixing

[jira] [Assigned] (SPARK-26311) [YARN] New feature: custom log URL for stdout/stderr

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26311: -- Assignee: Jungtaek Lim > [YARN] New feature: custom log URL for stdout/std

[spark] branch master updated: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS

2019-01-30 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ae5b2a6 [SPARK-26311][CORE] New feature: apply

[jira] [Resolved] (SPARK-26753) Log4j customization not working for spark-shell

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26753. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23675 [https

[jira] [Assigned] (SPARK-26753) Log4j customization not working for spark-shell

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26753: -- Assignee: Ankur Gupta > Log4j customization not working for spark-sh

[spark] branch master updated: [SPARK-26753][CORE] Fixed custom log levels for spark-shell by using Filter instead of Threshold

2019-01-30 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 25b97a4 [SPARK-26753][CORE] Fixed custom log

[jira] [Commented] (SPARK-26766) Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens

2019-01-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755501#comment-16755501 ] Marcelo Vanzin commented on SPARK-26766: The only thing YARN-specific about {{hadoopFSsToAccess

[spark-website] branch asf-site updated: CVE-2018-11760 (#175)

2019-01-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new d73a92d CVE-2018-11760 (#175

[jira] [Assigned] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26595: -- Assignee: Marcelo Vanzin > Allow delegation token renewal without a key

[jira] [Resolved] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26595. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23525 [https

[spark] branch master updated: [SPARK-26595][CORE] Allow credential renewal based on kerberos ticket cache.

2019-01-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2a67dbf [SPARK-26595][CORE] Allow credential

Re: Livy-0.6 release?

2019-01-28 Thread Marcelo Vanzin
There are a couple of patches under review that are currently blocking the release. Once those are done, we can work on releasing 0.6. On Mon, Jan 28, 2019 at 11:18 AM Roger Liu wrote: > > Hey there, > > > > I’m wondering if we have a timeline for releasing Livy-0.6? Its been a year > since

Re: JAPICMP Collections-4.3-RC2 report pdf (Was: Re: [VOTE][RC2] Commons collections 4.3)

2019-01-28 Thread Marcelo Vanzin
On Mon, Jan 28, 2019 at 11:36 AM sebb wrote: > > Haven't looked at the code, but if it's being compiled for java 8, and > > the new methods have a default implementation, then it's fine. clirr > > just complains because it's too old to know about default methods. > > I don't think so. > > I think

Re: JAPICMP Collections-4.3-RC2 report pdf (Was: Re: [VOTE][RC2] Commons collections 4.3)

2019-01-28 Thread Marcelo Vanzin
...Yes. That makes sense. Thanks. Seems like this > release should be a -1 then because we’re breaking BC without a major version > change. Right?? > > -Rob > > > > On Jan 28, 2019, at 2:07 PM, Marcelo Vanzin > > wrote: > > > > On Mon, Jan 28, 2019 at

Re: JAPICMP Collections-4.3-RC2 report pdf (Was: Re: [VOTE][RC2] Commons collections 4.3)

2019-01-28 Thread Marcelo Vanzin
On Mon, Jan 28, 2019 at 11:01 AM Rob Tompkins wrote: > Before I vote on the the thread, does adding a method to an interface cause > BC to break? I would think not. All of the clirr errors are merely additions. > Further the JAPICMP report confirms this. Existing classes that implement the

[jira] [Resolved] (SPARK-26432) Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26432. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23429 [https

[jira] [Assigned] (SPARK-26432) Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26432: -- Assignee: Sujith > Not able to connect Hbase 2.1 service Gett

[spark] branch master updated: [SPARK-26432][CORE] Obtain HBase delegation token operation compatible with HBase 2.x.x version API

2019-01-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new dfed439 [SPARK-26432][CORE] Obtain HBase

[jira] [Resolved] (SPARK-25825) Kerberos Support for Long Running Jobs in Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25825. Resolution: Won't Do I'm going to close this for now since there isn't a definition

[jira] [Assigned] (SPARK-25825) Kerberos Support for Long Running Jobs in Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25825: -- Assignee: (was: Marcelo Vanzin) > Kerberos Support for Long Running J

[jira] [Assigned] (SPARK-25825) Kerberos Support for Long Running Jobs in Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25825: -- Assignee: Marcelo Vanzin > Kerberos Support for Long Running Jobs in Kuberne

[jira] [Resolved] (SPARK-25751) Unit Testing for Kerberos Support for Spark on Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25751. Resolution: Won't Do Other changes in this area introduced unit tests already, so

[jira] [Resolved] (SPARK-23078) Allow Submitting Spark Thrift Server in Cluster Mode

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23078. Resolution: Won't Do No answer to my question, assuming use case is now covered. > Al

[jira] [Created] (SPARK-26733) Clean up entrypoint.sh

2019-01-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26733: -- Summary: Clean up entrypoint.sh Key: SPARK-26733 URL: https://issues.apache.org/jira/browse/SPARK-26733 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-26732) Flaky test: SparkContextInfoSuite.getRDDStorageInfo only reports on RDDs that actually persist data

2019-01-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26732: -- Summary: Flaky test: SparkContextInfoSuite.getRDDStorageInfo only reports on RDDs that actually persist data Key: SPARK-26732 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752734#comment-16752734 ] Marcelo Vanzin commented on SPARK-23411: Hmm, removing this method is a little tricky because

[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752661#comment-16752661 ] Marcelo Vanzin commented on SPARK-23411: I saw the test for this method fail (see e.g. https

[jira] [Assigned] (SPARK-23385) Allow SparkUITab to be customized adding in SparkConf and loaded when creating SparkUI

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23385: -- Assignee: Marcelo Vanzin > Allow SparkUITab to be customized adding in SparkC

[jira] [Assigned] (SPARK-22404) Provide an option to use unmanaged AM in yarn-client mode

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22404: -- Assignee: Devaraj K > Provide an option to use unmanaged AM in yarn-client m

[jira] [Assigned] (SPARK-23385) Allow SparkUITab to be customized adding in SparkConf and loaded when creating SparkUI

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23385: -- Assignee: (was: Marcelo Vanzin) > Allow SparkUITab to be customized add

[jira] [Resolved] (SPARK-22404) Provide an option to use unmanaged AM in yarn-client mode

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22404. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 19616 [https

[spark] branch master updated: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode

2019-01-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f06bc0c [SPARK-22404][YARN] Provide an option

[jira] [Resolved] (SPARK-26254) Move delegation token providers into a separate project

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26254. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23499 [https

[jira] [Assigned] (SPARK-26254) Move delegation token providers into a separate project

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26254: -- Assignee: Gabor Somogyi > Move delegation token providers into a separate proj

[spark] branch master updated: [SPARK-26254][CORE] Extract Hive + Kafka dependencies from Core.

2019-01-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 773efed [SPARK-26254][CORE] Extract Hive

[jira] [Resolved] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26694. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23618 [https

[spark] branch master updated: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell

2019-01-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b484490 [SPARK-26694][CORE] Progress bar should

[jira] [Assigned] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26694: -- Assignee: Ankur Gupta > Console progress bar not showing in

[jira] [Assigned] (SPARK-13587) Support virtualenv in PySpark

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-13587: -- Assignee: (was: Marcelo Vanzin) > Support virtualenv in PySp

[jira] [Assigned] (SPARK-13587) Support virtualenv in PySpark

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-13587: -- Assignee: Marcelo Vanzin (was: Jeff Zhang) > Support virtualenv in PySp

[jira] [Assigned] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26697: -- Assignee: Imran Rashid > ShuffleBlockFetcherIterator can log block sizes in addit

[spark] branch master updated: [SPARK-26697][CORE] Log local & remote block sizes.

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3699763 [SPARK-26697][CORE] Log local & re

[jira] [Resolved] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26697. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23621 [https

[jira] [Assigned] (SPARK-26530) Validate heartheat arguments in HeartbeatReceiver

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26530: -- Assignee: liupengcheng > Validate heartheat arguments in HeartbeatRecei

[jira] [Resolved] (SPARK-26530) Validate heartheat arguments in HeartbeatReceiver

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26530. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23445 [https

[spark] branch master updated: [SPARK-26530][CORE] Validate heartheat arguments in HeartbeatReceiver

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8d667c5 [SPARK-26530][CORE] Validate heartheat

[jira] [Updated] (SPARK-26682) Task attempt ID collision causes lost data

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26682: --- Fix Version/s: 2.3.3 > Task attempt ID collision causes lost d

[spark] branch branch-2.3 updated: [SPARK-26682][SQL] Use taskAttemptID instead of attemptNumber for Had…

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new ded902c [SPARK-26682][SQL] Use

[jira] [Commented] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751425#comment-16751425 ] Marcelo Vanzin commented on SPARK-25590: I filed https://github.com/fabric8io/kubernetes-client

Re: How to force-quit a Spark application?

2019-01-24 Thread Marcelo Vanzin
Hi, On Tue, Jan 22, 2019 at 11:30 AM Pola Yao wrote: > "Thread-1" #19 prio=5 os_prio=0 tid=0x7f9b6828e800 nid=0x77cb waiting on > condition [0x7f9a123e3000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for

[jira] [Resolved] (SPARK-26687) Building Spark Images has non-intuitive behaviour with paths to custom Dockerfiles

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26687. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23613 [https

[jira] [Assigned] (SPARK-26687) Building Spark Images has non-intuitive behaviour with paths to custom Dockerfiles

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26687: -- Assignee: Rob Vesse > Building Spark Images has non-intuitive behaviour with pa

[spark] branch master updated: [SPARK-26687][K8S] Fix handling of custom Dockerfile paths

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 69dab94 [SPARK-26687][K8S] Fix handling

[jira] [Resolved] (SPARK-26717) Support PodPriority for spark driver and executor on kubernetes

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26717. Resolution: Not A Problem Pretty sure this is covered by pod templates (SPARK-24434

Re: [VOTE] Release Apache Spark 2.3.3 (RC1)

2019-01-23 Thread Marcelo Vanzin
-1 too. I just upgraded https://issues.apache.org/jira/browse/SPARK-26682 to blocker. It's a small fix and we should make it in 2.3.3. On Thu, Jan 17, 2019 at 6:49 PM Takeshi Yamamuro wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.3.3. > > The vote is

[jira] [Updated] (SPARK-26682) Task attempt ID collision causes lost data

2019-01-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26682: --- Target Version/s: 2.3.3, 2.4.1 Labels: data-loss (was: ) Priority

[jira] [Commented] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750337#comment-16750337 ] Marcelo Vanzin commented on SPARK-26704: bq. they will be present and thus packaged

[jira] [Commented] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750249#comment-16750249 ] Marcelo Vanzin commented on SPARK-26704: You mentioned in the discussion that there is no need

[jira] [Assigned] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26605: -- Assignee: Marcelo Vanzin > New executors failing with expired tokens in client m

[jira] [Resolved] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26605. Resolution: Fixed Fix Version/s: 2.4.1 Issue resolved by pull request 23523 [https

[spark] branch branch-2.4 updated: [SPARK-26605][YARN] Update AM's credentials when creating tokens.

2019-01-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 10d7713 [SPARK-26605][YARN] Update AM's

[jira] [Created] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-22 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26694: -- Summary: Console progress bar not showing in 3.0 Key: SPARK-26694 URL: https://issues.apache.org/jira/browse/SPARK-26694 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26685. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23611 [https

[jira] [Assigned] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26685: -- Assignee: Rob Vesse > Building Spark Images with latest Docker does not hon

[spark] branch master updated: [SPARK-26685][K8S] Correct placement of ARG declaration

2019-01-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new dc2da72 [SPARK-26685][K8S] Correct placement

[jira] [Resolved] (SPARK-25887) Allow specifying Kubernetes context to use

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25887. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22904 [https

[jira] [Assigned] (SPARK-25887) Allow specifying Kubernetes context to use

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25887: -- Assignee: Rob Vesse > Allow specifying Kubernetes context to

[spark] branch master updated: [SPARK-25887][K8S] Configurable K8S context support

2019-01-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c542c24 [SPARK-25887][K8S] Configurable K8S

[jira] [Commented] (SPARK-25960) Support subpath mounting with Kubernetes

2019-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745353#comment-16745353 ] Marcelo Vanzin commented on SPARK-25960: I general we don't backport new features to previous

Re: How to force-quit a Spark application?

2019-01-16 Thread Marcelo Vanzin
.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > "VM Thread" os_p

Re: How to force-quit a Spark application?

2019-01-16 Thread Marcelo Vanzin
s to > exit the spark (e.g., System.exit()), but failed. Is there an explicit way to > shutdown all the alive threads in the spark application and then quit > afterwards? > > > On Tue, Jan 15, 2019 at 2:38 PM Marcelo Vanzin wrote: >> >> You should check the active thread

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

2019-01-15 Thread Marcelo Vanzin
+1 to that. HIVE-16391 by itself means we're giving up things like Hadoop 3, and we're also putting the burden on the Hive folks to fix a problem that we created. The current PR is basically a Spark-side fix for that bug. It does mean also upgrading Hive (which gives us Hadoop 3, yay!), but I

Re: How to force-quit a Spark application?

2019-01-15 Thread Marcelo Vanzin
You should check the active threads in your app. Since your pool uses non-daemon threads, that will prevent the app from exiting. spark.stop() should have stopped the Spark jobs in other threads, at least. But if something is blocking one of those threads, or if something is creating a non-daemon

[jira] [Resolved] (SPARK-25857) Document delegation token code in Spark

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25857. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23348 [https

[jira] [Assigned] (SPARK-25857) Document delegation token code in Spark

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25857: -- Assignee: Marcelo Vanzin > Document delegation token code in Sp

[spark] branch master updated: [SPARK-25857][CORE] Add developer documentation regarding delegation tokens.

2019-01-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8a54492 [SPARK-25857][CORE] Add developer

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

2019-01-15 Thread Marcelo Vanzin
The metastore interactions in Spark are currently based on APIs that are in the Hive exec jar; so that makes it not possible to have Spark work with Hadoop 3 until the exec jar is upgraded. It could be possible to re-implement those interactions based solely on the metastore client Hive

[jira] [Assigned] (SPARK-26592) Kafka delegation token doesn't support proxy user

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26592: -- Assignee: Gabor Somogyi > Kafka delegation token doesn't support proxy u

[jira] [Resolved] (SPARK-26592) Kafka delegation token doesn't support proxy user

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26592. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23511 [https

[spark] branch master updated: [SPARK-26592][SS] Throw exception when kafka delegation token tried to obtain with proxy user

2019-01-15 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 5ca45e8 [SPARK-26592][SS] Throw exception when

[jira] [Commented] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740719#comment-16740719 ] Marcelo Vanzin commented on SPARK-26606: So you're using standalone cluster mode. It might

[jira] [Commented] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740692#comment-16740692 ] Marcelo Vanzin commented on SPARK-26606: What master are you using? > parameters pas

[jira] [Commented] (SPARK-26606) extraJavaOptions does not work

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740675#comment-16740675 ] Marcelo Vanzin commented on SPARK-26606: Can you define what "does not work" means

[jira] [Commented] (SPARK-26606) extraJavaOptions does not work

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740668#comment-16740668 ] Marcelo Vanzin commented on SPARK-26606: Define not working? e.g. I see you have both single

[jira] [Created] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-11 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26605: -- Summary: New executors failing with expired tokens in client mode Key: SPARK-26605 URL: https://issues.apache.org/jira/browse/SPARK-26605 Project: Spark

[jira] [Resolved] (SPARK-26482) Use ConfigEntry for hardcoded configs for ui categories.

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26482. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23423 [https

[jira] [Assigned] (SPARK-26482) Use ConfigEntry for hardcoded configs for ui categories.

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26482: -- Assignee: Jungtaek Lim > Use ConfigEntry for hardcoded configs for ui categor

[spark] branch master updated: [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui categories

2019-01-11 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d9e4cf6 [SPARK-26482][CORE] Use ConfigEntry

[jira] [Created] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-10 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26595: -- Summary: Allow delegation token renewal without a keytab Key: SPARK-26595 URL: https://issues.apache.org/jira/browse/SPARK-26595 Project: Spark Issue

[jira] [Resolved] (SPARK-24493) Kerberos Ticket Renewal is failing in Hadoop 2.8+ and Hadoop 3

2019-01-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24493. Resolution: Later Given that Spark does not publish Hadoop 2.8-based binaries, and I

[jira] [Resolved] (SPARK-26414) Race between SparkContext and YARN AM can cause NPE in UI setup code

2019-01-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26414. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 3.0.0 > R

[jira] [Commented] (SPARK-26414) Race between SparkContext and YARN AM can cause NPE in UI setup code

2019-01-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738580#comment-16738580 ] Marcelo Vanzin commented on SPARK-26414: I actually ended up fixing this in the change for SPARK

[jira] [Resolved] (SPARK-23386) Enable direct application links before replay

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23386. Resolution: Won't Fix I'm closing this for now as I believe SPARK-6951 handles a lot

<    5   6   7   8   9   10   11   12   13   14   >