Re: structured streaming- checkpoint metadata growing indefinetely

2022-05-04 Thread Wojciech Indyk
For posterity: the problem was FileStreamSourceLog class. I needed to overwrite method shouldRetain, that by default returns true and its doc say: Default implementation retains all log entries. Implementations should override the method to change the behavior. -- Kind regards/ Pozdrawiam,

Re: Spark error with jupyter

2022-05-04 Thread Gourav Sengupta
Hi, looks like spark listener is not working? Is your session still running? Try to see the SPARK UI to find out whether the session is still active or not Regards, Gourav On Tue, May 3, 2022 at 7:37 PM Bjørn Jørgensen wrote: > I use jupyterlab and spark and I have not seen this before. > >