Help needed optimize spark history server performance

Dear Spark Community, I'm writing to seek your expertise in optimizing the performance of our Spark History Server (SHS) deployed on Amazon EKS. We're encountering timeouts (HTTP 504) when loading large event logs exceeding 5 GB. *Our Setup:* - Deployment: SHS on EKS with Nginx ingress (idle

Spark History Server in GCP

Hi There, I have been able to start the Spark History Server in GKE Kubernetes Cluster. And I have created a Service Account in my google project with permissions as storage admin,storage object admin and owner. Now when I have tried to submit the job using Spark Submit, using below options

spark 3.1.1 history server fails to boot with scala/MatchError

we have spark 2.4.x clusters running fine; however when running spark 3.1.1 the spark history server fails to boot note: we build spark 3.1.1 from source for hadoop 3.2.1 we use supervisord to start services so our start command is: $SPARK_HOME/bin/spark-class

Spark History Server to S3 doesn't show up incomplete jobs

/ spark.history.fs.update.interval = 5s Once my application is completed, I can see it shows up on the spark history server. However, running applications doesn't show up on "incomplete applications". I have also checked the log, whenever my application end, I can see this message: 21/0

Re: Spark History Server log files questions

eck it right away unless it is the weekend, but depending on the use case you could store more days.. On Mon, 22 Mar 2021 at 23:52, Hung Vu wrote: > Hi, > > I have couple questions to ask regarding the Spark history server: > > 1. Is there a way for a cluster to selectively clean old fi

Spark History Server log files questions

Hi, I have couple questions to ask regarding the Spark history server: 1. Is there a way for a cluster to selectively clean old files? For example, if we want to keep some logs from 3 days ago but also cleaned some logs from 2 days ago, is there a filter or config to do that? 2. We have over

Running Spark history Server at Context localhost:18080/sparkhistory

Hi, I want to run Running Spark history Server at Context localhost:18080/sparkhistory instead at port localhost:18080 The end goal is to access Spark History Server with a domain name i.e, domainname/sparkhistory is there any hacks or spark config options? -- Thanks, Regards, SandishKumar

Re: Spark UI History server on Kubernetes

> Hi Lakshman, > > > > We’ve set these 2 properties to bringup spark history server > > > > spark.history.fs.logDirectory > > spark.history.ui.port > > > > We’re writing the logs to HDFS. In order to write logs, we’re setting &g

RE: Spark UI History server on Kubernetes

Hi Lakshman, We’ve set these 2 properties to bringup spark history server spark.history.fs.logDirectory spark.history.ui.port We’re writing the logs to HDFS. In order to write logs, we’re setting following properties while submitting the spark job spark.eventLog.enabled true

RE: Spark UI History server on Kubernetes

, 2019 6:02 PM To: user@spark.apache.org Subject: Spark UI History server on Kubernetes Hello, We are running Spark 2.4 on Kubernetes cluster, able to access the Spark UI using "kubectl port-forward". However, this spark UI contains currently running Spark application logs, we

Spark UI History server on Kubernetes

s to setup 'Spark History server' on Kubernetes. Thanks! Best Regards, Lakshman Battini.

[Spark cluster standalone v2.4.0] - problems with reverse proxy functionnality regarding submitted applications in cluster mode and the spark history server ui

on the same worker from which the submit command is executed, the access to the application through the master works fine and others uris too (I mean the problem disappear). 2 - When the reverse proxy is setted, I can't access anymore to the history server UI (many js and css errors) in both

[Spark cluster standalone v2.4.0] - problems with reverse proxy functionnality regarding submitted applications in cluster mode and the spark history server ui

on the same worker from which the submit command is executed, the access to the application through the master works fine and others uris too (I mean the problem disappear). 2 - When the reverse proxy is setted, I can't access anymore to the history server UI (many js and css errors) in both

Re: Upgrading spark history server, no logs showing.

I finally circled back and tinkered with this, eventually finding the solution. It turned out to be HDFS permissions on the history files. For whatever reason, our HDFS perms worked fine with spark 2.0.2 and 2.1.2, but when we ran spark 2.3.0 it wouldn't load any history in the UI. I found out

Upgrading spark history server, no logs showing.

Hi, We have multiple installations of spark installed on our clusters. They reside in different directories which the jobs point to when they run. For a couple of years now, we've run our history server off spark 2.0.2. We have 2.1.2, 2.2.1 and 2.3.0 installed as well. I've tried upgrading

History server and non-HDFS filesystems

to the history server - even though we are using the same service principal that was used to write the logs. I tracked it down to this call in "FSHistoryProvider" (which was added for v2.2.0): SparkHadoopUtil.checkAccessPermission() >From what I can tell, it is preemptively checking th

Heap Settings for History Server

Hi folks, I couldn't find much literature on this so I figured I could ask here. Does anyone have experience in tuning the memory settings and interval times of the Spark History Server? Let's say I have 500 applications at 0.5 G each with a *spark.history.fs.update.interval* of 400s

Re: Spark history server running on Mongo

Yes, you are absolutely right, though UI does not change often, and it potentially allows to iterate faster, IMHO, which is why started working on this. For me, it felt like this functionality could easily be outsourced to a separate project. And, as you pointed out, I did add some small fixes to

Re: Spark history server running on Mongo

On Tue, Jul 18, 2017 at 7:21 PM, Ivan Sadikov wrote: > Repository that I linked to does not require rebuilding Spark and could be > used with current distribution, which is preferable in my case. Fair enough, although that means that you're re-implementing the Spark UI,

Re: Spark history server running on Mongo

t; just create a MongoDB implementation if you want. >> >> On Tue, Jul 18, 2017 at 12:56 AM, Ivan Sadikov <ivan.sadi...@gmail.com> >> wrote: >> > Hello everyone! >> > >> > I have been working on Spark history server that uses MongoDB as a >>

Re: Spark history server running on Mongo

; > > I have been working on Spark history server that uses MongoDB as a > datastore > > for processed events to iterate on idea that Spree project uses for Spark > > UI. Project was originally designed to improve on standalone history > server > > with reduced memory foo

Re: Spark history server running on Mongo

gt; > I have been working on Spark history server that uses MongoDB as a datastore > for processed events to iterate on idea that Spree project uses for Spark > UI. Project was originally designed to improve on standalone history server > with reduced memory footprint. > > Project

Spark history server running on Mongo

Hello everyone! I have been working on Spark history server that uses MongoDB as a datastore for processed events to iterate on idea that Spree project uses for Spark UI. Project was originally designed to improve on standalone history server with reduced memory footprint. Project lives here

Re: Why spark history server does not show RDD even if it is persisted?

om> wrote: > The REST APIs are not just for Spark history server. When an application > is running, you can use the REST APIs to talk to Spark UI HTTP server as > well. > > On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari <paragp...@gmail.com> > wrote: > >> ping

Re: Why spark history server does not show RDD even if it is persisted?

The REST APIs are not just for Spark history server. When an application is running, you can use the REST APIs to talk to Spark UI HTTP server as well. On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari <paragp...@gmail.com> wrote: > ping... > > > > *Thanks,Parag Chaudhari

Re: Why spark history server does not show RDD even if it is persisted?

om> wrote: > Thanks! > > If spark does not log these events in event log then why spark history > server provides an API to get RDD information? > > From the documentation, > > /applications/[app-id]/storage/rdd A list of stored RDDs for the given > application. > &g

Re: Why spark history server does not show RDD even if it is persisted?

Thanks! If spark does not log these events in event log then why spark history server provides an API to get RDD information? >From the documentation, /applications/[app-id]/storage/rdd A list of stored RDDs for the given application. /applications/[app-id]/storage/rdd/[rdd-id] Deta

Re: Why spark history server does not show RDD even if it is persisted?

t;> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it >> will not be written into event-log, I think that's why you cannot get such >> info in history server. >> >> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <paragp...@gmail.com> >

Re: Why spark history server does not show RDD even if it is persisted?

t be written into event-log, I think that's why you cannot get such info > in history server. > > On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <paragp...@gmail.com> > wrote: > >> Hi, >> >> I am running spark shell in spark version 2.0.2. Here is my program,

Re: Why spark history server does not show RDD even if it is persisted?

AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will not be written into event-log, I think that's why you cannot get such info in history server. On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <paragp...@gmail.com> wrote: > Hi, > > I am running spark

Why spark history server does not show RDD even if it is persisted?

Hi, I am running spark shell in spark version 2.0.2. Here is my program, var myrdd = sc.parallelize(Array.range(1, 10)) myrdd.setName("test") myrdd.cache myrdd.collect But I am not able to see any RDD info in "storage" tab in spark history server. I l

Re: how can I set the log configuration file for spark history server ?

(-dev) Just configure your log4j.properties in $SPARK_HOME/conf (or set a custom $SPARK_CONF_DIR for the history server). On Thu, Dec 8, 2016 at 7:20 PM, John Fang <xiaojian@alibaba-inc.com> wrote: > ./start-history-server.sh > starting org.apache.spark.deploy.history.HistorySer

Re: how can I set the log configuration file for spark history server ?

) to the history server, you need to enable/configure it. Hope that helps. -Don On Thu, Dec 8, 2016 at 9:20 PM, John Fang <xiaojian@alibaba-inc.com> wrote: > ./start-history-server.sh > starting org.apache.spark.deploy.history.HistoryServer, > logging to /home/admin/koala/data/version

how can I set the log configuration file for spark history server ?

./start-history-server.sh starting org.apache.spark.deploy.history.HistoryServer, logging to  /home/admin/koala/data/versions/0/SPARK/2.0.2/spark-2.0.2-bin-hadoop2.6/logs/spark-admin-org.apache.spark.deploy.history.HistoryServer-1-v069166214.sqa.zmf.out Then the history will print all log to the

Re: Passing Custom App Id for consumption in History Server

ication which >> schedules spark jobs, and records the metadata for that job (say job >> params, cores, etc). In this system application, I want to link every job >> with its corresponding UI (history server). The only way I can do this is >> if I have the app Id o

Re: Passing Custom App Id for consumption in History Server

provided app id instead of generating one using time? > > Lets take the following scenario : I have a system application which > schedules spark jobs, and records the metadata for that job (say job > params, cores, etc). In this system application, I want to link every job > with its co

Passing Custom App Id for consumption in History Server

, and records the metadata for that job (say job params, cores, etc). In this system application, I want to link every job with its corresponding UI (history server). The only way I can do this is if I have the app Id of that job stored in this system application. And the only way one can get the app Id

Re: Spark 2.0 History Server Storage

1. SPARK-16859 submitted On Tue, Aug 2, 2016 at 9:07 PM, Andrei Ivanov wrote: > OK, answering myself - this is broken since 1.6.2 by SPARK-13845 > > > On Tue, Aug

Re: Spark 2.0 History Server Storage

OK, answering myself - this is broken since 1.6.2 by SPARK-13845 On Tue, Aug 2, 2016 at 12:10 AM, Andrei Ivanov wrote: > Hi all, > > I've just tried upgrading Spark to 2.0 and so far it looks generally good. > > But there

Spark 2.0 History Server Storage

Hi all, I've just tried upgrading Spark to 2.0 and so far it looks generally good. But there is at least one issue I see right away - jon histories are missing storage information (persisted RRDs). This info is also missing from pre upgrade jobs. Does anyone have a clue what can be wrong?

Re: Redirect from yarn to spark history server

. I can see spark job in > yarn but I am able to go from any "log history" link from yarn to spark > history server. How would I keep track of yarn log and its corresponding log > in spark history server? Is there any setting in yarn/spark that let me > redirect to spark h

Redirect from yarn to spark history server

Hello, I am running pyspark job using yarn-cluster mode. I can see spark job in yarn but I am able to go from any "log history" link from yarn to spark history server. How would I keep track of yarn log and its corresponding log in spark history server? Is there any setting in yarn/spar

Problem with History Server

Hello: I'm using the history server to keep track of the applications I run in my cluster. I'm using Spark with YARN. When I run on application it finishes correctly even YARN says that it finished. This is the result of the YARN Resource Manager API {u'app': [{u'runningContainers': -1

Re: History Server Refresh?

park applications set up to write their event logs > into S3 - this is super useful for ephemeral clusters, I can have > persistent history even though my hosts go away. > > A history server is set up to view this s3 location, and that works fine > too - at least on startup. > > T

Re: History Server Refresh?

osts go away. A history server is set up to view this s3 location, and that works fine too - at least on startup. The problem is that the history server doesn't seem to notice new logs arriving into the S3 bucket. Any idea how I can get it to scan the folder for new files? Thanks, -miles s3 isn't a rea

Check if spark master/history server is running via Java

Hi, How to check if spark master /history server is running on node? is there any command for it? I would like to accomplish it with java if possible. Thanks, Mihir Monani

History Server Refresh?

Hey there. I have my spark applications set up to write their event logs into S3 - this is super useful for ephemeral clusters, I can have persistent history even though my hosts go away. A history server is set up to view this s3 location, and that works fine too - at least on startup

Documentation for "hidden" RESTful API for submitting jobs (not history server)

Hi all, While googling Spark, I accidentally found a RESTful API existing in Spark for submitting jobs. The link is here, http://arturmkrtchyan.com/apache-spark-hidden-rest-api As Josh said, I can see the history of this RESTful API, https://issues.apache.org/jira/browse/SPARK-5388 and also

Re: Spark History Server NOT showing Jobs with Hortonworks

nworks (HDP 2.3.2_Spark 1.4.1) – The Spark History Server is NOT showing the Spark Running Jobs in Local Mode The local-host:4040/app/v1 is ALSO not working How can I look at my local Spark job? # Generated by Apache Ambari. Fri Feb 5 00:37:06 2016 spark.history.kerberos

Re: Spark History Server NOT showing Jobs with Hortonworks

nology Ltd, its subsidiaries nor their > employees accept any responsibility. > > > > > > *From:* Sutanu Das [mailto:sd2...@att.com] > *Sent:* 18 February 2016 22:58 > *To:* Mich Talebzadeh <m...@peridale.co.uk>; user@spark.apache.org > > *Subject:* RE: Spark Histor

RE: Spark History Server NOT showing Jobs with Hortonworks

[mailto:sd2...@att.com] Sent: 18 February 2016 22:58 To: Mich Talebzadeh <m...@peridale.co.uk>; user@spark.apache.org Subject: RE: Spark History Server NOT showing Jobs with Hortonworks Hi Mich, Community - Do I need to specify it in the properties file in my spark-submit ? From: Mich Ta

RE: Spark History Server NOT showing Jobs with Hortonworks

Hi Mich, Community - Do I need to specify it in the properties file in my spark-submit ? From: Mich Talebzadeh [mailto:m...@peridale.co.uk] Sent: Thursday, February 18, 2016 4:28 PM To: Sutanu Das; user@spark.apache.org Subject: RE: Spark History Server NOT showing Jobs with Hortonworks

RE: Spark History Server NOT showing Jobs with Hortonworks

idiaries nor their employees accept any responsibility. From: Sutanu Das [mailto:sd2...@att.com] Sent: 18 February 2016 22:22 To: user@spark.apache.org Subject: Spark History Server NOT showing Jobs with Hortonworks Hi Community, Challenged with Spark issues with Hortonworks (HDP 2.3.2

Spark History Server NOT showing Jobs with Hortonworks

Hi Community, Challenged with Spark issues with Hortonworks (HDP 2.3.2_Spark 1.4.1) - The Spark History Server is NOT showing the Spark Running Jobs in Local Mode The local-host:4040/app/v1 is ALSO not working How can I look at my local Spark job? # Generated by Apache Ambari. Fri Feb 5 00

Re: pyspark - spark history server

person. Thanks! On Fri, Feb 5, 2016 at 2:08 PM, cs user <acldstk...@gmail.com> wrote: > Hi All, > > I'm having trouble getting a job to use the spark history server. We have > a cluster configured with Ambari, if I run the job from one of the nodes > within the Amba

pyspark - spark history server

Hi All, I'm having trouble getting a job to use the spark history server. We have a cluster configured with Ambari, if I run the job from one of the nodes within the Ambari configured cluster, everything works fine, the job appears in the spark history server. If I configure a client external

DAG visualization: no visualization information available with history server

Hello All, I am running the history server for a completed application. This application was run with the following parameters bin/spark-submit --class --master local[2] --conf spark.local.dir=/mnt/ --conf spark.eventLog.dir=/mnt/sparklog/ --conf spark.eventLog.enabled=true --conf

Re: Is Spark History Server supported for Mesos?

On 9 Dec 2015, at 22:01, Kelvin Chu <2dot7kel...@gmail.com<mailto:2dot7kel...@gmail.com>> wrote: Spark on YARN can use History Server by setting the configuration spark.yarn.historyServer.address. That's the stuff in SPARK-1537 which isn' actually built in yet. But, I can't

Is Spark History Server supported for Mesos?

Spark on YARN can use History Server by setting the configuration spark.yarn.historyServer.address. But, I can't find similar config for Mesos. Is History Server supported by Spark on Mesos? Thanks. Kelvin

spark history server + yarn log aggregation issue

Hi, I am running Spark-on-YARN on a secure cluster with yarn log aggregation set up. Once a job completes, when viewing stdout/stderr executor logs in the Spark history server UI it redirects me to the local nodemanager where a page appears for a second saying ‘Redirecting to log server

Re: History server is not receiving any event

Are you starting your history server? ./sbin/start-history-server.sh You can read more here http://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact Thanks Best Regards On Tue, Aug 25, 2015 at 1:07 AM, b.bhavesh b.borisan...@gmail.com wrote: Hi, I am working

History server is not receiving any event

Hi, I am working on streaming application. I tried to configure history server to persist the events of application in hadoop file system (hdfs). However, it is not logging any events. I am running Apache Spark 1.4.1 (pyspark) under Ubuntu 14.04 with three nodes. Here is my configuration: File

RE: Web UI vs History Server Bugs

Probably your application has crashed or was terminated without invoking the stop method of spark context - in such cases it doesn't create the empty flag file which apparently tells the history server that it can safely show the log data - simpy go to some of the other dirs of the history server

Re: Web UI vs History Server Bugs

No, what I'm seeing is that while the cluster is running, I can't see the app info after the app is completed. That is to say, when I click on the application name on master:8080, no info is shown. However, when I examine the same file on the History Server, the application information opens fine

Re: Spark 1.4 History Server - HDP 2.2

On 20 Jun 2015, at 17:37, Ashish Soni asoni.le...@gmail.com wrote: Can any one help i am getting below error when i try to start the History Server I do not see any org.apache.spark.deploy.yarn.history.pakage inside the assembly jar not sure how to get

Re: Web UI vs History Server Bugs

On 17 Jun 2015, at 19:10, jcai jonathon@yale.edu wrote: Hi, I am running this on Spark stand-alone mode. I find that when I examine the web UI, a couple bugs arise: 1. There is a discrepancy between the number denoting the duration of the application when I run the history server

Spark 1.4 History Server - HDP 2.2

Can any one help i am getting below error when i try to start the History Server I do not see any org.apache.spark.deploy.yarn.history.pakage inside the assembly jar not sure how to get that java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.history.YarnHistoryProvider Thanks

Re: Web UI vs History Server Bugs

. There is a discrepancy between the number denoting the duration of the application when I run the history server and the number given by the web UI (default address is master:8080). I checked more specific details, including task and stage durations (when clicking on the application), and these appear

Web UI vs History Server Bugs

Hi, I am running this on Spark stand-alone mode. I find that when I examine the web UI, a couple bugs arise: 1. There is a discrepancy between the number denoting the duration of the application when I run the history server and the number given by the web UI (default address is master:8080). I

Re: Spark History Server pointing to S3

Not quiet sure, but try pointing the spark.history.fs.logDirectory to your s3 Thanks Best Regards On Tue, Jun 16, 2015 at 6:26 PM, Gianluca Privitera gianluca.privite...@studio.unibo.it wrote: In Spark website it’s stated in the View After the Fact section (

Spark History Server pointing to S3

In Spark website it’s stated in the View After the Fact section (https://spark.apache.org/docs/latest/monitoring.html) that you can point the start-history-server.sh script to a directory in order do view the Web UI using the logs as data source. Is it possible to point that script to S3?

Re: Spark History Server pointing to S3

It gives me an exception with org.apache.spark.deploy.history.FsHistoryProvider , a problem with the file system. I can reproduce the exception if you want. It perfectly works if I give a local path, I tested it in 1.3.0 version. Gianluca On 16 Jun 2015, at 15:08, Akhil Das

Re: spark eventLog and history server

, then you basically missed the opportunity. Does it mean I don't need to start history server if I only use spark in standalone mode? - Yes, you don't need to start the history server. On Mon, Jun 8, 2015 at 7:57 PM, Du Li l...@yahoo-inc.com.invalid wrote: Event log is enabled in my spark

spark eventLog and history server

not show the event log. The event log files have suffix .inprogress. Removing the suffix does not solve the problem. Do I need to do anything here in order to view the event logs of finished jobs? Or do I need to stop ssc differently? In addition, the documentation seems to suggest history server

Re: View all user's application logs in history server

No one using History server? :) Am I the only one need to see all user's logs? Jianshi On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang jianshi.hu...@gmail.com wrote: Hi, I'm using Spark 1.4.0-rc1 and I'm using default settings for history server. But I can only see my own logs

Re: View all user's application logs in history server

You may be the only one not seeing all the logs. Are you sure all the users are writing to the same log directory? The HS can only read from a single log directory. On Wed, May 27, 2015 at 5:33 AM, Jianshi Huang jianshi.hu...@gmail.com wrote: No one using History server? :) Am I the only one

Re: View all user's application logs in history server

directory. On Wed, May 27, 2015 at 5:33 AM, Jianshi Huang jianshi.hu...@gmail.com wrote: No one using History server? :) Am I the only one need to see all user's logs? Jianshi On Thu, May 21, 2015 at 1:29 PM, Jianshi Huang jianshi.hu...@gmail.com wrote: Hi, I'm using Spark 1.4.0-rc1

Re: View all user's application logs in history server

Then: - Are all files readable by the user running the history server? - Did all applications call sc.stop() correctly (i.e. files do not have the .inprogress suffix)? Other than that, always look at the logs first, looking for any errors that may be thrown. On Wed, May 27, 2015 at 9:10 AM

View all user's application logs in history server

Hi, I'm using Spark 1.4.0-rc1 and I'm using default settings for history server. But I can only see my own logs. Is it possible to view all user's logs? The permission is fine for the user group. -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/

history server

i am trying to launch the spark 1.3.1 history server on a secure cluster. i can see in the logs that it successfully logs into kerberos, and it is replaying all the logs, but i never see the log message that indicate the web server is started (i should see something like Successfully started

Re: history server

The history server may need several hours to start if you have a lot of event logs. Is it stuck, or still replaying logs? Best Regards, Shixiong Zhu 2015-05-07 11:03 GMT-07:00 Marcelo Vanzin van...@cloudera.com: Can you get a jstack for the process? Maybe it's stuck somewhere. On Thu, May 7

Re: history server

SPARK-5522 is really cool. Didn't notice it. Best Regards, Shixiong Zhu 2015-05-07 11:36 GMT-07:00 Marcelo Vanzin van...@cloudera.com: That shouldn't be true in 1.3 (see SPARK-5522). On Thu, May 7, 2015 at 11:33 AM, Shixiong Zhu zsxw...@gmail.com wrote: The history server may need several

Re: history server

That shouldn't be true in 1.3 (see SPARK-5522). On Thu, May 7, 2015 at 11:33 AM, Shixiong Zhu zsxw...@gmail.com wrote: The history server may need several hours to start if you have a lot of event logs. Is it stuck, or still replaying logs? Best Regards, Shixiong Zhu 2015-05-07 11:03 GMT

Re: history server

:00 AM, Koert Kuipers ko...@tresata.com wrote: i am trying to launch the spark 1.3.1 history server on a secure cluster. i can see in the logs that it successfully logs into kerberos, and it is replaying all the logs, but i never see the log message that indicate the web server is started (i

Re: history server

, Marcelo Vanzin van...@cloudera.com wrote: Can you get a jstack for the process? Maybe it's stuck somewhere. On Thu, May 7, 2015 at 11:00 AM, Koert Kuipers ko...@tresata.com wrote: i am trying to launch the spark 1.3.1 history server on a secure cluster. i can see in the logs

Re: history server

Hi, Sorry this may be a little off topic but I tried searching for docs on history server but couldn't really find much. Can someone point me to a doc or give me a point of reference for the use and intent of a history server? -- Ankur On 7 May 2015, at 12:06, Koert Kuipers ko

Re: history server

somewhere. On Thu, May 7, 2015 at 11:00 AM, Koert Kuipers ko...@tresata.com wrote: i am trying to launch the spark 1.3.1 history server on a secure cluster. i can see in the logs that it successfully logs into kerberos, and it is replaying all the logs, but i never see the log message

Re: history server

Can you get a jstack for the process? Maybe it's stuck somewhere. On Thu, May 7, 2015 at 11:00 AM, Koert Kuipers ko...@tresata.com wrote: i am trying to launch the spark 1.3.1 history server on a secure cluster. i can see in the logs that it successfully logs into kerberos

Re: history-server does't read logs which are on FS

Thanks, it helped. We can't use Spark 1.3 because Cassandra DSE doesn't support it. 2015-04-17 21:48 GMT+02:00 Imran Rashid iras...@cloudera.com: are you calling sc.stop() at the end of your applications? The history server only displays completed applications, but if you don't call sc.stop

Can not get executor's Log from Spark's History Server

Hi, Experts I run my Spark Cluster on Yarn. I used to get executors' Logs from Spark's History Server. But after I started my Hadoop jobhistory server and made configuration to aggregate logs of hadoop jobs to a HDFS directory, I found that I could not get spark's executors' Logs any more

Re: Can not get executor's Log from Spark's History Server

The Spark history server does not have the ability to serve executor logs currently. You need to use the yarn logs command for that. On Tue, Apr 7, 2015 at 2:51 AM, donhoff_h 165612...@qq.com wrote: Hi, Experts I run my Spark Cluster on Yarn. I used to get executors' Logs from Spark's History

Re: Spark History Server : jobs link doesn't open

on spark History server doesn't open and shows following message : HTTP ERROR: 500 Problem accessing /history/application_1425934191900_87572. Reason: Server Error -- *Powered by Jetty://*

Spark History Server : jobs link doesn't open

We have Spark on YARN, with Cloudera Manager 5.3.2 and CDH 5.3.2 Jobs link on spark History server doesn't open and shows following message : HTTP ERROR: 500 Problem accessing /history/application_1425934191900_87572. Reason: Server Error -- *Powered by Jetty://*

Re: Spark History Server : jobs link doesn't open

, Mar 26, 2015 at 4:27 PM, , Roy rp...@njit.edu wrote: We have Spark on YARN, with Cloudera Manager 5.3.2 and CDH 5.3.2 Jobs link on spark History server doesn't open and shows following message : HTTP ERROR: 500 Problem accessing /history/application_1425934191900_87572. Reason: Server

Re: Spark Job History Server

Hortonworks directly if they're suggesting you use those. On Wed, Mar 18, 2015 at 3:30 AM, patcharee patcharee.thong...@uni.no wrote: Hi, I am using spark 1.3. I would like to use Spark Job History Server. I added the following line into conf/spark-defaults.conf spark.yarn.services

Re: Spark Job History Server

Hi Patcharee, It is an alpha feature in HDP distribution, integrating ATS with Spark history server. If you are using upstream, you can configure spark as regular without these configuration. But other related configuration are still mandatory, such as hdp.version related. Thanks. Zhan Zhang

Re: Spark Job History Server

History Server. I added the following line into conf/spark-defaults.conf spark.yarn.services org.apache.spark.deploy.yarn. history.YarnHistoryService spark.history.provider org.apache.spark.deploy.yarn. history.YarnHistoryProvider spark.yarn.historyServer.address sandbox.hortonworks.com:19888

Re: Spark Job History Server

Spark Job History Server. I added the following line into conf/spark-defaults.conf spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider spark.yarn.historyServer.address

Re: Spark Job History Server

: ./sbin/start-history-server.sh ​Read more here http://spark.apache.org/docs/1.3.0/monitoring.html.​ Thanks Best Regards On Wed, Mar 18, 2015 at 4:00 PM, patcharee patcharee.thong...@uni.no wrote: Hi, I am using spark 1.3. I would like to use Spark Job History Server. I added

Spark Job History Server

Hi, I am using spark 1.3. I would like to use Spark Job History Server. I added the following line into conf/spark-defaults.conf spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider

  1   2   >