Re: Spark UI History server on Kubernetes
In addition to what Rao mentioned, if you are using cloud blob storage such as AWS S3, you can specify your history location to be an S3 location such as: `s3://mybucket/path/to/history` On Wed, Jan 23, 2019 at 12:55 AM Rao, Abhishek (Nokia - IN/Bangalore) < abhishek@nokia.com> wrote: > Hi Lakshman, > > > > We’ve set these 2 properties to bringup spark history server > > > > spark.history.fs.logDirectory > > spark.history.ui.port > > > > We’re writing the logs to HDFS. In order to write logs, we’re setting > following properties while submitting the spark job > > spark.eventLog.enabled true > > spark.eventLog.dir > > > > Thanks and Regards, > > Abhishek > > > > *From:* Battini Lakshman > *Sent:* Wednesday, January 23, 2019 1:55 PM > *To:* Rao, Abhishek (Nokia - IN/Bangalore) > *Subject:* Re: Spark UI History server on Kubernetes > > > > HI Abhishek, > > > > Thank you for your response. Could you please let me know the properties > you configured for bringing up History Server and its UI. > > > > Also, are you writing the logs to any directory on persistent storage, if > yes, could you let me know the changes you did in Spark to write logs to > that directory. Thanks! > > > > Best Regards, > > Lakshman Battini. > > > > On Tue, Jan 22, 2019 at 10:53 PM Rao, Abhishek (Nokia - IN/Bangalore) < > abhishek@nokia.com> wrote: > > Hi, > > > > We’ve setup spark-history service (based on spark 2.4) on K8S. UI works > perfectly fine when running on NodePort. We’re facing some issues when on > ingress. > > Please let us know what kind of inputs do you need? > > > > Thanks and Regards, > > Abhishek > > > > *From:* Battini Lakshman > *Sent:* Tuesday, January 22, 2019 6:02 PM > *To:* user@spark.apache.org > *Subject:* Spark UI History server on Kubernetes > > > > Hello, > > > > We are running Spark 2.4 on Kubernetes cluster, able to access the Spark > UI using "kubectl port-forward". > > > > However, this spark UI contains currently running Spark application logs, > we would like to maintain the 'completed' spark application logs as well. > Could someone help us to setup 'Spark History server' on Kubernetes. Thanks! > > > > Best Regards, > > Lakshman Battini. > >
RE: Spark UI History server on Kubernetes
Hi Lakshman, We’ve set these 2 properties to bringup spark history server spark.history.fs.logDirectory spark.history.ui.port We’re writing the logs to HDFS. In order to write logs, we’re setting following properties while submitting the spark job spark.eventLog.enabled true spark.eventLog.dir Thanks and Regards, Abhishek From: Battini Lakshman Sent: Wednesday, January 23, 2019 1:55 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Subject: Re: Spark UI History server on Kubernetes HI Abhishek, Thank you for your response. Could you please let me know the properties you configured for bringing up History Server and its UI. Also, are you writing the logs to any directory on persistent storage, if yes, could you let me know the changes you did in Spark to write logs to that directory. Thanks! Best Regards, Lakshman Battini. On Tue, Jan 22, 2019 at 10:53 PM Rao, Abhishek (Nokia - IN/Bangalore) mailto:abhishek@nokia.com>> wrote: Hi, We’ve setup spark-history service (based on spark 2.4) on K8S. UI works perfectly fine when running on NodePort. We’re facing some issues when on ingress. Please let us know what kind of inputs do you need? Thanks and Regards, Abhishek From: Battini Lakshman mailto:battini.laksh...@gmail.com>> Sent: Tuesday, January 22, 2019 6:02 PM To: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Spark UI History server on Kubernetes Hello, We are running Spark 2.4 on Kubernetes cluster, able to access the Spark UI using "kubectl port-forward". However, this spark UI contains currently running Spark application logs, we would like to maintain the 'completed' spark application logs as well. Could someone help us to setup 'Spark History server' on Kubernetes. Thanks! Best Regards, Lakshman Battini.
RE: Spark UI History server on Kubernetes
Hi, We’ve setup spark-history service (based on spark 2.4) on K8S. UI works perfectly fine when running on NodePort. We’re facing some issues when on ingress. Please let us know what kind of inputs do you need? Thanks and Regards, Abhishek From: Battini Lakshman Sent: Tuesday, January 22, 2019 6:02 PM To: user@spark.apache.org Subject: Spark UI History server on Kubernetes Hello, We are running Spark 2.4 on Kubernetes cluster, able to access the Spark UI using "kubectl port-forward". However, this spark UI contains currently running Spark application logs, we would like to maintain the 'completed' spark application logs as well. Could someone help us to setup 'Spark History server' on Kubernetes. Thanks! Best Regards, Lakshman Battini.
