[jira] [Assigned] (SPARK-25887) Allow specifying Kubernetes context to use

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25887: -- Assignee: Rob Vesse > Allow specifying Kubernetes context to

[jira] [Commented] (SPARK-25960) Support subpath mounting with Kubernetes

2019-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745353#comment-16745353 ] Marcelo Vanzin commented on SPARK-25960: I general we don't bac

Re: How to force-quit a Spark application?

2019-01-16 Thread Marcelo Vanzin
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > "

Re: How to force-quit a Spark application?

2019-01-16 Thread Marcelo Vanzin
ed many ways to > exit the spark (e.g., System.exit()), but failed. Is there an explicit way to > shutdown all the alive threads in the spark application and then quit > afterwards? > > > On Tue, Jan 15, 2019 at 2:38 PM Marcelo Vanzin wrote: >> >> You should check th

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

2019-01-15 Thread Marcelo Vanzin
+1 to that. HIVE-16391 by itself means we're giving up things like Hadoop 3, and we're also putting the burden on the Hive folks to fix a problem that we created. The current PR is basically a Spark-side fix for that bug. It does mean also upgrading Hive (which gives us Hadoop 3, yay!), but I thin

Re: How to force-quit a Spark application?

2019-01-15 Thread Marcelo Vanzin
You should check the active threads in your app. Since your pool uses non-daemon threads, that will prevent the app from exiting. spark.stop() should have stopped the Spark jobs in other threads, at least. But if something is blocking one of those threads, or if something is creating a non-daemon

[jira] [Assigned] (SPARK-25857) Document delegation token code in Spark

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25857: -- Assignee: Marcelo Vanzin > Document delegation token code in Sp

[jira] [Resolved] (SPARK-25857) Document delegation token code in Spark

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25857. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23348 [https

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

2019-01-15 Thread Marcelo Vanzin
The metastore interactions in Spark are currently based on APIs that are in the Hive exec jar; so that makes it not possible to have Spark work with Hadoop 3 until the exec jar is upgraded. It could be possible to re-implement those interactions based solely on the metastore client Hive publishes;

[jira] [Assigned] (SPARK-26592) Kafka delegation token doesn't support proxy user

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26592: -- Assignee: Gabor Somogyi > Kafka delegation token doesn't support pr

[jira] [Resolved] (SPARK-26592) Kafka delegation token doesn't support proxy user

2019-01-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26592. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23511 [https

[jira] [Commented] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740719#comment-16740719 ] Marcelo Vanzin commented on SPARK-26606: So you're using standalon

[jira] [Commented] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740692#comment-16740692 ] Marcelo Vanzin commented on SPARK-26606: What master are you u

[jira] [Commented] (SPARK-26606) extraJavaOptions does not work

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740675#comment-16740675 ] Marcelo Vanzin commented on SPARK-26606: Can you define what "does

[jira] [Commented] (SPARK-26606) extraJavaOptions does not work

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740668#comment-16740668 ] Marcelo Vanzin commented on SPARK-26606: Define not working? e.g. I see

[jira] [Created] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-11 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26605: -- Summary: New executors failing with expired tokens in client mode Key: SPARK-26605 URL: https://issues.apache.org/jira/browse/SPARK-26605 Project: Spark

[jira] [Resolved] (SPARK-26482) Use ConfigEntry for hardcoded configs for ui categories.

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26482. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23423 [https

[jira] [Assigned] (SPARK-26482) Use ConfigEntry for hardcoded configs for ui categories.

2019-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26482: -- Assignee: Jungtaek Lim > Use ConfigEntry for hardcoded configs for ui categor

[jira] [Created] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-10 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26595: -- Summary: Allow delegation token renewal without a keytab Key: SPARK-26595 URL: https://issues.apache.org/jira/browse/SPARK-26595 Project: Spark Issue

[jira] [Resolved] (SPARK-24493) Kerberos Ticket Renewal is failing in Hadoop 2.8+ and Hadoop 3

2019-01-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24493. Resolution: Later Given that Spark does not publish Hadoop 2.8-based binaries, and I

[jira] [Resolved] (SPARK-26414) Race between SparkContext and YARN AM can cause NPE in UI setup code

2019-01-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26414. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 3.0.0 > R

[jira] [Commented] (SPARK-26414) Race between SparkContext and YARN AM can cause NPE in UI setup code

2019-01-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738580#comment-16738580 ] Marcelo Vanzin commented on SPARK-26414: I actually ended up fixing this in

[jira] [Resolved] (SPARK-23386) Enable direct application links before replay

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23386. Resolution: Won't Fix I'm closing this for now as I believe SPARK-6951 handle

[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736493#comment-16736493 ] Marcelo Vanzin commented on SPARK-26565: Stepping back a little, what is

[jira] [Assigned] (SPARK-26491) Use ConfigEntry for hardcoded configs for test categories.

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26491: -- Assignee: Marco Gaido > Use ConfigEntry for hardcoded configs for test categor

[jira] [Resolved] (SPARK-26491) Use ConfigEntry for hardcoded configs for test categories.

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26491. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23413 [https

Livy is now on gitbox

2019-01-07 Thread Marcelo Vanzin
If you're using Github, this shouldn't change anything. If you're a committer, you should update your Apache remote to point at: https://gitbox.apache.org/repos/asf/incubator-livy.git Same thing for the website repo: https://gitbox.apache.org/repos/asf?p=incubator-livy-website.git -- Marcelo

Re: [VOTE] Move source repo to gitbox

2019-01-07 Thread Marcelo Vanzin
Vote passes. I opened an INFRA ticket to handle the move: https://issues.apache.org/jira/browse/INFRA-17575 On Thu, Jan 3, 2019 at 9:57 AM Marcelo Vanzin wrote: > > Creating a formal vote for this. I don't think we have a choice but > they seem to request a vote anyway. Thi

[jira] [Resolved] (SPARK-26561) Symbol 'type org.apache.spark.Logging' is missing from the classpath. This symbol is required by 'class org.apache.hadoop.hbase.spark.HBaseContext'

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26561. Resolution: Not A Bug External libraries should not use internal Spark classes like

Re: Spark History UI + Keycloak Integration

2019-01-04 Thread Marcelo Vanzin
On Fri, Jan 4, 2019 at 3:25 AM G, Ajay (Nokia - IN/Bangalore) wrote: ... > Added session handler for all context - > contextHandler.setSessionHandler(new SessionHandler()) ... > Keycloak authentication seems to work, Is this the right approach ? If it is > fine I can submit a PR. I don't reme

[jira] [Assigned] (SPARK-26489) Use ConfigEntry for hardcoded configs for python/r categories.

2019-01-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26489: -- Assignee: Jungtaek Lim > Use ConfigEntry for hardcoded configs for pytho

[jira] [Resolved] (SPARK-26489) Use ConfigEntry for hardcoded configs for python/r categories.

2019-01-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26489. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23428 [https

[jira] [Resolved] (LIVY-538) Add integration tests for the thrift server

2019-01-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-538. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 131 [https

Re: How to reissue a delegated token after max lifetime passes for a spark streaming application on a Kerberized cluster

2019-01-03 Thread Marcelo Vanzin
ing with “kms-dt”. > > > > Anyone knows why this is happening ? Any suggestion to make it working > with KMS ? > > > > Thanks > > > > > > > > [image: cid:image001.jpg@01D41D15.E01B6F00] > > *Paolo Platter* > > *CTO* > > E-mail: p

[jira] [Resolved] (SPARK-26522) Auth secret error in RBackendAuthHandler

2019-01-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26522. Resolution: Not A Problem Yes, this is an issue with Livy that is fixed in the master

Re: How to reissue a delegated token after max lifetime passes for a spark streaming application on a Kerberized cluster

2019-01-03 Thread Marcelo Vanzin
If you are using the principal / keytab params, Spark should create tokens as needed. If it's not, something else is going wrong, and only looking at full logs for the app would help. On Wed, Jan 2, 2019 at 5:09 PM Ali Nazemian wrote: > > Hi, > > We are using a headless keytab to run our long-runn

[VOTE] Move source repo to gitbox

2019-01-03 Thread Marcelo Vanzin
Creating a formal vote for this. I don't think we have a choice but they seem to request a vote anyway. This will be lazy consensus, I plan to create an infra ticket for the migration on Monday if the vote passes. Starting with my +1. -- Marcelo

[jira] [Assigned] (SPARK-23980) Resilient Spark driver on Kubernetes

2019-01-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23980: -- Assignee: (was: Marcelo Vanzin) > Resilient Spark driver on Kuberne

[jira] [Assigned] (SPARK-23980) Resilient Spark driver on Kubernetes

2019-01-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23980: -- Assignee: Marcelo Vanzin > Resilient Spark driver on Kuberne

[jira] [Resolved] (SPARK-26441) Add kind configuration of driver pod

2019-01-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26441. Resolution: Duplicate > Add kind configuration of driver

Re: Incubator Podling Report (Due 2nd January)

2019-01-02 Thread Marcelo Vanzin
bozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > [image: Inactive hide details for Marcelo Vanzin ---01/02/2019 09:57:30 > AM---I drafted this report based on last month's; if I don't he]Marcelo > Vanzin ---01/02/2019 09:5

Incubator wiki write access

2019-01-02 Thread Marcelo Vanzin
Could I get write access so I can update the podling report page? User name is MarceloVanzin. Thanks! -- Marcelo - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@

Re: Incubator Podling Report (Due 2nd January)

2019-01-02 Thread Marcelo Vanzin
I drafted this report based on last month's; if I don't hear back before EOD I'll post it to the wiki, assuming I get the needed permissions in time. == Livy is web service that exposes a REST interface for managing long running Apache Spark contexts in your cluster. With Livy, new applic

[jira] [Resolved] (SPARK-25220) [K8S] Split out node selector config between driver and executors.

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25220. Resolution: Duplicate Seems like this is covered by SPARK-24434 (pod templates). > [

[jira] [Resolved] (SPARK-24491) Support for requesting GPU resources on K8S

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24491. Resolution: Duplicate Consensus seems to be that pod templates (SPARK-24434) already

[jira] [Resolved] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25642. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22498 [https

[jira] [Assigned] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25642: -- Assignee: Parth Gandhi > Add new Metrics in External Shuffle Service to h

[jira] [Resolved] (SPARK-25442) Support STS to run in K8S deployment with spark deployment mode as cluster

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25442. Resolution: Duplicate > Support STS to run in K8S deployment with spark deployment m

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: (was: Marcelo Vanzin) > Add support for dynamic resource allocat

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: Marcelo Vanzin > Add support for dynamic resource allocat

[jira] [Commented] (LIVY-544) Statements in a Session must execute one by one which is time-consuming

2018-12-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726948#comment-16726948 ] Marcelo Vanzin commented on LIVY-544: - bq. when user's main usage scenar

[jira] [Resolved] (SPARK-26311) [YARN] New feature: custom log URL for stdout/stderr

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26311. Resolution: Duplicate Duping to the existing bug; no need to track this in two different

[jira] [Assigned] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23155: -- Assignee: Marcelo Vanzin > YARN-aggregated executor/driver logs appear unavaila

[jira] [Assigned] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23155: -- Assignee: Marcelo Vanzin > YARN-aggregated executor/driver logs appear unavaila

[jira] [Assigned] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23155: -- Assignee: (was: Marcelo Vanzin) > YARN-aggregated executor/driver logs app

[jira] [Assigned] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23155: -- Assignee: (was: Marcelo Vanzin) > YARN-aggregated executor/driver logs app

[jira] [Assigned] (SPARK-25064) Total Tasks in WebUI does not match Active+Failed+Complete Tasks

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25064: -- Assignee: Marcelo Vanzin > Total Tasks in WebUI does not match Active+Fai

[jira] [Assigned] (SPARK-25064) Total Tasks in WebUI does not match Active+Failed+Complete Tasks

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25064: -- Assignee: (was: Marcelo Vanzin) > Total Tasks in WebUI does not match Act

Re: Custom Metric Sink on Executor Always ClassNotFound

2018-12-20 Thread Marcelo Vanzin
First, it's really weird to use "org.apache.spark" for a class that is not in Spark. For executors, the jar file of the sink needs to be in the system classpath; the application jar is not in the system classpath, so that does not work. There are different ways for you to get it there, most of the

[jira] [Assigned] (SPARK-26392) Cancel pending allocate requests by taking locality preference into account

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26392: -- Assignee: wuyi > Cancel pending allocate requests by taking locality preference i

[jira] [Resolved] (SPARK-26392) Cancel pending allocate requests by taking locality preference into account

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26392. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23344 [https

[jira] [Commented] (LIVY-372) Livy Doesn't work with Spark on Mesos in cluster mode

2018-12-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726059#comment-16726059 ] Marcelo Vanzin commented on LIVY-372: - Livy is definitely not tested on anything

[jira] [Assigned] (LIVY-479) Enable livy.rsc.launcher.address configuration

2018-12-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-479: --- Assignee: Tao Li > Enable livy.rsc.launcher.address configurat

[jira] [Resolved] (LIVY-479) Enable livy.rsc.launcher.address configuration

2018-12-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-479. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 101 [https

[jira] [Created] (SPARK-26414) Race between SparkContext and YARN AM can cause NPE in UI setup code

2018-12-19 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26414: -- Summary: Race between SparkContext and YARN AM can cause NPE in UI setup code Key: SPARK-26414 URL: https://issues.apache.org/jira/browse/SPARK-26414 Project

[jira] [Assigned] (LIVY-543) Fix inconsistent Jetty dependencies

2018-12-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-543: --- Assignee: Marco Gaido > Fix inconsistent Jetty dependenc

[jira] [Resolved] (LIVY-543) Fix inconsistent Jetty dependencies

2018-12-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-543. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 134 [https

[jira] [Resolved] (SPARK-25815) Kerberos Support in Kubernetes resource manager (Client Mode)

2018-12-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25815. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22911 [https

[jira] [Assigned] (SPARK-25815) Kerberos Support in Kubernetes resource manager (Client Mode)

2018-12-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25815: -- Assignee: Marcelo Vanzin > Kerberos Support in Kubernetes resource manager (Cli

[jira] [Assigned] (SPARK-25869) Spark on YARN: the original diagnostics is missing when job failed maxAppAttempts times

2018-12-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25869: -- Assignee: (was: Marcelo Vanzin) > Spark on YARN: the original diagnostics

[jira] [Assigned] (SPARK-25869) Spark on YARN: the original diagnostics is missing when job failed maxAppAttempts times

2018-12-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25869: -- Assignee: Marcelo Vanzin > Spark on YARN: the original diagnostics is missing w

[jira] [Commented] (SPARK-26386) [K8s] Add configuration for executor OwnerReference when running in client mode

2018-12-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723283#comment-16723283 ] Marcelo Vanzin commented on SPARK-26386: Doe

[jira] [Assigned] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch

2018-12-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26255: -- Assignee: Chakravarthi > Custom error/exception is not thrown for the SQL tab w

[jira] [Resolved] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch

2018-12-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26255. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23312 [https

[jira] [Reopened] (SPARK-20130) Flaky test: BlockManagerProactiveReplicationSuite

2018-12-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-20130: Reopening since I've seen this a few times recently. e.g.: https://amplab.cs.berkele

[jira] [Resolved] (SPARK-21239) Support WAL recover in windows

2018-12-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21239. Resolution: Duplicate As far as my understanding goes the fix for SPARK-25778 should also

[jira] [Resolved] (SPARK-23886) update query.status

2018-12-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23886. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23095 [https

[jira] [Assigned] (SPARK-23886) update query.status

2018-12-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23886: -- Assignee: Gabor Somogyi > update query.sta

[jira] [Resolved] (SPARK-23504) Flaky test: RateSourceV2Suite.basic microbatch execution

2018-12-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23504. Resolution: Cannot Reproduce Closing this for now based on the above comment. > Fl

[jira] [Resolved] (CRYPTO-140) adding suuport for openssl 1.1.0g in common crypto

2018-12-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/CRYPTO-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved CRYPTO-140. --- Resolution: Duplicate Let's use a single bug to track 1.1 compatibility. > adding

[jira] [Resolved] (CRYPTO-142) Native library does not load if ENGINE_load_rdrand is not available

2018-12-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/CRYPTO-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved CRYPTO-142. --- Resolution: Duplicate > Native library does not load if ENGINE_load_rdrand is not availa

[jira] [Commented] (CRYPTO-142) Native library does not load if ENGINE_load_rdrand is not available

2018-12-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/CRYPTO-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720474#comment-16720474 ] Marcelo Vanzin commented on CRYPTO-142: --- This is probably because of OpenSSL

[jira] [Resolved] (SPARK-26322) Simplify kafka delegation token sasl.mechanism configuration

2018-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26322. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23274 [https

[jira] [Assigned] (SPARK-26322) Simplify kafka delegation token sasl.mechanism configuration

2018-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26322: -- Assignee: Gabor Somogyi > Simplify kafka delegation token sasl.mechan

[jira] [Resolved] (SPARK-23464) MesosClusterScheduler double-escapes parameters to bash command

2018-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23464. Resolution: Duplicate > MesosClusterScheduler double-escapes parameters to bash comm

[jira] [Assigned] (SPARK-25277) YARN applicationMaster metrics should not register static and JVM metrics

2018-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25277: -- Assignee: Luca Canali > YARN applicationMaster metrics should not register sta

[jira] [Resolved] (SPARK-25277) YARN applicationMaster metrics should not register static and JVM metrics

2018-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25277. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22279 [https

[jira] [Commented] (SPARK-26254) Move delegation token providers into a separate project

2018-12-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719574#comment-16719574 ] Marcelo Vanzin commented on SPARK-26254: bq. loaded the providers

[jira] [Resolved] (SPARK-26333) FsHistoryProviderSuite failed because setReadable doesn't work in RedHat

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26333. Resolution: Not A Bug > FsHistoryProviderSuite failed because setReadable doesn't

[jira] [Commented] (SPARK-26333) FsHistoryProviderSuite failed because setReadable doesn't work in RedHat

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718397#comment-16718397 ] Marcelo Vanzin commented on SPARK-26333: {{setReadable}} works as root.

[jira] [Commented] (SPARK-26343) Speed up running the kubernetes integration tests locally

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718396#comment-16718396 ] Marcelo Vanzin commented on SPARK-26343: You can do that in sbt ("k

[jira] [Assigned] (SPARK-22640) Can not switch python exec in executor side

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22640: -- Assignee: Marcelo Vanzin > Can not switch python exec in executor s

[jira] [Assigned] (SPARK-22640) Can not switch python exec in executor side

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22640: -- Assignee: (was: Marcelo Vanzin) > Can not switch python exec in executor s

[jira] [Resolved] (SPARK-11085) Add support for HTTP proxy

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11085. Resolution: Duplicate I believe SPARK-17568 should take care of this, since that allows

[jira] [Assigned] (SPARK-26239) Add configurable auth secret source in k8s backend

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26239: -- Assignee: Matt Cheah > Add configurable auth secret source in k8s back

[jira] [Resolved] (SPARK-26239) Add configurable auth secret source in k8s backend

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26239. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23252 [https

[jira] [Assigned] (SPARK-25200) Allow setting HADOOP_CONF_DIR as a spark property

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25200: -- Assignee: Marcelo Vanzin > Allow setting HADOOP_CONF_DIR as a spark prope

[jira] [Assigned] (SPARK-25200) Allow setting HADOOP_CONF_DIR as a spark property

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25200: -- Assignee: (was: Marcelo Vanzin) > Allow setting HADOOP_CONF_DIR as a sp

[jira] [Commented] (SPARK-26333) FsHistoryProviderSuite failed because setReadable doesn't work in RedHat

2018-12-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717612#comment-16717612 ] Marcelo Vanzin commented on SPARK-26333: Are you running the test as root?

[jira] [Assigned] (SPARK-23472) Add config properties for administrator JVM options

2018-12-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23472: -- Assignee: Marcelo Vanzin > Add config properties for administrator JVM opti

<    2   3   4   5   6   7   8   9   10   11   >