Hi,
With the current design, eventlogs are not ideal for long running streaming
applications. So, it is better then to disable the eventlogs. There was a
proposal for splitting the eventlogs based on size/Job/query for long
running applications, not sure about the followup for that.
Regards,
Hi.
There is a workaround for that.
You can disable event logs for Spark Streaming applications.
On Tue, Jul 16, 2019 at 1:08 PM raman gugnani
wrote:
> HI ,
>
> I have long running spark streaming jobs.
> Event log directories are getting filled with .inprogress files.
> Is there fix or work
HI ,
I have long running spark streaming jobs.
Event log directories are getting filled with .inprogress files.
Is there fix or work around for spark streaming.
There is also one jira raised for the same by one reporter.
https://issues.apache.org/jira/browse/SPARK-22783
--
Raman Gugnani
