I reported the issue as SPARK-22783. Hope this will be resolved soon.
Regards Omkar On Fri, Dec 1, 2017 at 10:55 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > There's really no current solution to this. There's a brief discussion > about it on SPARK-12140. > > Here we recommend people disable event logs for streaming apps, as > sub-optimal as that might be... > > On Fri, Dec 1, 2017 at 3:27 AM, kankalapti omkar naidu > <omkarnaid...@gmail.com> wrote: > > Dear spark developers, > > > > I did not find rolling policy for spark event logs,for long living > streaming > > applications the event log is becoming huge with in few hours. > > The only way I found is fixing the existing event logging spark > > listener(EventLoggingListener), > > but I believe there should be already a solution tackling this problem > > because it is a frequently occurring problem when ever somebody runs > > streaming applications for long time. > > > > What would be the right way to fix this,whether any config settings are > > available for this or writing our own scripts to cleanup the log > directory > > regularly based on time/size or we need to implement rolling policy > feature? > > > > Regards > > Omkar > > > > -- > Marcelo >
