RE: about blob.storage.dir and .buffer files

2016-01-29 Thread Gwenhael Pasquiers
:35 To: user@flink.apache.org Subject: Re: about blob.storage.dir and .buffer files Hi Gwenhael! Let's look into this and fix anything we find. Can you briefly tell us: - How much data is in the blob-store directory, versus in the buffer files? - How many buffer files do you have an

Re: about blob.storage.dir and .buffer files

2016-01-28 Thread Stephan Ewen
Hi Gwenhael! Let's look into this and fix anything we find. Can you briefly tell us: - How much data is in the blob-store directory, versus in the buffer files? - How many buffer files do you have and how large are they in average? Greetings, Stephan On Thu, Jan 28, 2016 at 10:18 AM, Ti

Re: about blob.storage.dir and .buffer files

2016-01-28 Thread Till Rohrmann
Hi Gwenhael, in theory the blob storage files can be any binary data. At the moment, this is however only used to distribute the user code jars. The jars are kept around as long as the job is running. Every library-cache-manager.cleanup.interval interval the files are checked and those which are n

about blob.storage.dir and .buffer files

2016-01-27 Thread Gwenhael Pasquiers
Hello, We got a question about blob.storage.dir and it’s .buffer files : What are they ? And are they cleaned or is there a way to limit their size and to evaluate the necessary space ? We got a node root volume disk filled by those files (~20GB) and it crashed. Well, the root was filled becaus