In addition to what Rao mentioned, if you are using cloud blob storage such as AWS S3, you can specify your history location to be an S3 location such as: `s3://mybucket/path/to/history`
On Wed, Jan 23, 2019 at 12:55 AM Rao, Abhishek (Nokia - IN/Bangalore) < abhishek....@nokia.com> wrote: > Hi Lakshman, > > > > We’ve set these 2 properties to bringup spark history server > > > > spark.history.fs.logDirectory <location where the logs are stored> > > spark.history.ui.port <Port on which history server should come up> > > > > We’re writing the logs to HDFS. In order to write logs, we’re setting > following properties while submitting the spark job > > spark.eventLog.enabled true > > spark.eventLog.dir <Directory where logs should be written> > > > > Thanks and Regards, > > Abhishek > > > > *From:* Battini Lakshman <battini.laksh...@gmail.com> > *Sent:* Wednesday, January 23, 2019 1:55 PM > *To:* Rao, Abhishek (Nokia - IN/Bangalore) <abhishek....@nokia.com> > *Subject:* Re: Spark UI History server on Kubernetes > > > > HI Abhishek, > > > > Thank you for your response. Could you please let me know the properties > you configured for bringing up History Server and its UI. > > > > Also, are you writing the logs to any directory on persistent storage, if > yes, could you let me know the changes you did in Spark to write logs to > that directory. Thanks! > > > > Best Regards, > > Lakshman Battini. > > > > On Tue, Jan 22, 2019 at 10:53 PM Rao, Abhishek (Nokia - IN/Bangalore) < > abhishek....@nokia.com> wrote: > > Hi, > > > > We’ve setup spark-history service (based on spark 2.4) on K8S. UI works > perfectly fine when running on NodePort. We’re facing some issues when on > ingress. > > Please let us know what kind of inputs do you need? > > > > Thanks and Regards, > > Abhishek > > > > *From:* Battini Lakshman <battini.laksh...@gmail.com> > *Sent:* Tuesday, January 22, 2019 6:02 PM > *To:* user@spark.apache.org > *Subject:* Spark UI History server on Kubernetes > > > > Hello, > > > > We are running Spark 2.4 on Kubernetes cluster, able to access the Spark > UI using "kubectl port-forward". > > > > However, this spark UI contains currently running Spark application logs, > we would like to maintain the 'completed' spark application logs as well. > Could someone help us to setup 'Spark History server' on Kubernetes. Thanks! > > > > Best Regards, > > Lakshman Battini. > >
