Hi ! Here are the answers : - How much data is in the blob-store directory, versus in the buffer files? : around 20MB per application versus 20gb ( one specific windowing app may be).
- How many buffer files do you have and how large are they in average? : standard apps ( no buffer files or really small ones) , the one app : around 10 files of 2gb each in (ex /tmp_flink/flink-io-*/b3177f8e1910a686ccb48c1da533581d06305acccc2a370734b493327b117757.1441.buffer) From: ewenstep...@gmail.com [mailto:ewenstep...@gmail.com] On Behalf Of Stephan Ewen Sent: jeudi 28 janvier 2016 15:35 To: user@flink.apache.org Subject: Re: about blob.storage.dir and .buffer files Hi Gwenhael! Let's look into this and fix anything we find. Can you briefly tell us: - How much data is in the blob-store directory, versus in the buffer files? - How many buffer files do you have and how large are they in average? Greetings, Stephan On Thu, Jan 28, 2016 at 10:18 AM, Till Rohrmann <trohrm...@apache.org<mailto:trohrm...@apache.org>> wrote: Hi Gwenhael, in theory the blob storage files can be any binary data. At the moment, this is however only used to distribute the user code jars. The jars are kept around as long as the job is running. Every library-cache-manager.cleanup.interval interval the files are checked and those which are no longer referenced are deleted. In the case of a termination of Flink all files should be purged. If this is not the case, then we have to check what the problem is. Could you check the size of your user jars? Cheers, Till On Wed, Jan 27, 2016 at 4:38 PM, Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com<mailto:gwenhael.pasqui...@ericsson.com>> wrote: Hello, We got a question about blob.storage.dir and it’s .buffer files : What are they ? And are they cleaned or is there a way to limit their size and to evaluate the necessary space ? We got a node root volume disk filled by those files (~20GB) and it crashed. Well, the root was filled because we changed the path to /tmp_flink and it was on the root filesystem, our bad. We changed it in an urge because the default path (/tmp) was being periodically cleaned by the OS and that made flink crash. B.R.
