Re: File idempotent store problem
Hi Yeah I can see the point that its better to have the default implementation work better. I have reworked the implementation so the file store and cache are independent and the cache is only used for quick lookup. Also the default sizes of the file store has been increased to allow storing more entries, and it now "free space" by deleting the oldest 1000 entries (all configurable). On Wed, Jan 3, 2018 at 8:51 PM, Krzysztof Hołdanowicz wrote: > Hi, > > regarding CAMEL-12058 I don't know if you are aware of all consequences of > wrong order in the idempotent file store. > The wrong order in the file is not the problem itself as long as elemens > are added and eviceted on runtime, because caffeine provides an api for > ordering like: > >- @Override public Map coldest(int limit) >- @Override public Map hottest(int limit) >- @Override public Map oldest(int limit), >- @Override public Map youngest(int limit) ) > > however the consequences of this appears after RESTART. The memory cache > does not contain the proper entries (in case of reaching the max limit > size) because it does not load elements from hottest to coldest but with > the file entries order hence some of the files are consumed multiple times. > It means that current implementation of file idempotent store is not usable > at all anymore. Ignoring the issue (CAMEL-12058) means that Camel library > does not provide any implementation of idempotent file store as the current > behaviour is completely wrong and causes consuming multiple times the same > file after reaching max size limit and after restarting application. > > > Regards > Kris > > sob., 2 gru 2017 o 15:14 użytkownik Krzysztof Hołdanowicz < > holdanow...@gmail.com> napisał: > >> I don't know if I understood you correctly. >> Instead of looping via cache.keySet() you mean looping via: >> Map.Entry entry : cache.entrySet() or cache.foreach((k, v) >> -> {...})? >> >> If yes what is the difference? Isn't it the same unordered collection? >> If Caffeince returns unordered collection how we can get ordered entries? >> Isn't it related with: >> https://github.com/ben-manes/caffeine/issues/86 >> >> Shouldn't we use a kind of LinkedHashMap implementation? >> >> Regards >> Kris >> >> wt., 28 lis 2017 o 18:40 użytkownik Claus Ibsen-2 [via Camel] < >> ml+s465427n5815878...@n5.nabble.com> napisał: >> >>> Hi >>> >>> Ah well spotted. >>> I think we should for loop via Map.Entry (or (k,v) via lambda) which I >>> think will be in the correct order. >>> >>> You are welcome to log a JIRA. And also work on unit test and patch. >>> http://camel.apache.org/contributing >>> >>> On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz >>> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=5815878&i=0>> >>> wrote: >>> >>> > Hi all, >>> > >>> > I recently noticed that there is wrong entry order in file using >>> > FileIdempotentRepository implementation. >>> > The effect is that instead of having order like: >>> > >>> > file1.txt.20171123 >>> > file2.txt.20171123 >>> > file1.txt.20171124 >>> > file3.txt.20171125 >>> > file2.txt.20171126 >>> > >>> > we have: >>> > >>> > file1.txt.20171123 >>> > file1.txt.20171124 >>> > file2.txt.20171123 >>> > file2.txt.20171126 >>> > file3.txt.20171125 >>> > >>> > where date extension represents order in which particular file was >>> consumed >>> > by the idempotent file consumer. >>> > As a consequence instead of initializing memory cache with newest >>> values, >>> > it is initialized (probably) based on hash function from truncStore >>> method >>> > and we consume same file more than once: >>> > >>> > protected void trunkStore() { >>> > LOG.info("Trunking idempotent filestore: {}", fileStore); >>> > FileOutputStream fos = null; >>> > try { >>> > fos = new FileOutputStream(fileStore); >>> > for (String key : *cache.keySet()*) { >>> > fos.write(key.getBytes()); >>> > fos.write(STORE_DELIMITER.getBytes()); >>> > } >>> > } catch (IOException e) { >>> > throw ObjectHelper.wrapRuntimeCamelE
Re: File idempotent store problem
DELIMITER.getBytes()); > >> > } > >> > } catch (IOException e) { > >> > throw ObjectHelper.wrapRuntimeCamelException(e); > >> > } finally { > >> > IOHelper.close(fos, "Trunking file idempotent repository", > >> LOG); > >> > } > >> > } > >> > > >> > LRUCache: > >> > > >> > @Override > >> > public Set keySet() { > >> > return map.keySet(); > >> > } > >> > > >> > where previously it was: > >> > > >> > @Override > >> > public Set keySet() { > >> > return map.*ascendingKeySet*(); > >> > } > >> > > >> > Regards > >> > Kris > >> > -- > >> > > >> > Pozdrawiam > >> > > >> > Krzysztof Hołdanowicz > >> > >> > >> > >> -- > >> Claus Ibsen > >> - > >> http://davsclaus.com @davsclaus > >> Camel in Action 2: https://www.manning.com/ibsen2 > >> > >> > >> -- > >> If you reply to this email, your message will be added to the discussion > >> below: > >> > >> > http://camel.465427.n5.nabble.com/File-idempotent-store-problem-since-2-20-tp5815868p5815878.html > >> To unsubscribe from Camel - Users, click here > >> < > http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=aG9sZGFub3dpY3pAZ21haWwuY29tfDQ2NTQyOHwtMTk2MTQxMTc0NQ== > > > >> . > >> NAML > >> < > http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml > > > >> > > -- > > > > Pozdrawiam > > > > Krzysztof Hołdanowicz > > > > > -- > > Pozdrawiam > > Krzysztof Hołdanowicz >
Re: File idempotent store problem
Hi, regarding CAMEL-12058 I don't know if you are aware of all consequences of wrong order in the idempotent file store. The wrong order in the file is not the problem itself as long as elemens are added and eviceted on runtime, because caffeine provides an api for ordering like: - @Override public Map coldest(int limit) - @Override public Map hottest(int limit) - @Override public Map oldest(int limit), - @Override public Map youngest(int limit) ) however the consequences of this appears after RESTART. The memory cache does not contain the proper entries (in case of reaching the max limit size) because it does not load elements from hottest to coldest but with the file entries order hence some of the files are consumed multiple times. It means that current implementation of file idempotent store is not usable at all anymore. Ignoring the issue (CAMEL-12058) means that Camel library does not provide any implementation of idempotent file store as the current behaviour is completely wrong and causes consuming multiple times the same file after reaching max size limit and after restarting application. Regards Kris sob., 2 gru 2017 o 15:14 użytkownik Krzysztof Hołdanowicz < holdanow...@gmail.com> napisał: > I don't know if I understood you correctly. > Instead of looping via cache.keySet() you mean looping via: > Map.Entry entry : cache.entrySet() or cache.foreach((k, v) > -> {...})? > > If yes what is the difference? Isn't it the same unordered collection? > If Caffeince returns unordered collection how we can get ordered entries? > Isn't it related with: > https://github.com/ben-manes/caffeine/issues/86 > > Shouldn't we use a kind of LinkedHashMap implementation? > > Regards > Kris > > wt., 28 lis 2017 o 18:40 użytkownik Claus Ibsen-2 [via Camel] < > ml+s465427n5815878...@n5.nabble.com> napisał: > >> Hi >> >> Ah well spotted. >> I think we should for loop via Map.Entry (or (k,v) via lambda) which I >> think will be in the correct order. >> >> You are welcome to log a JIRA. And also work on unit test and patch. >> http://camel.apache.org/contributing >> >> On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz >> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=5815878&i=0>> >> wrote: >> >> > Hi all, >> > >> > I recently noticed that there is wrong entry order in file using >> > FileIdempotentRepository implementation. >> > The effect is that instead of having order like: >> > >> > file1.txt.20171123 >> > file2.txt.20171123 >> > file1.txt.20171124 >> > file3.txt.20171125 >> > file2.txt.20171126 >> > >> > we have: >> > >> > file1.txt.20171123 >> > file1.txt.20171124 >> > file2.txt.20171123 >> > file2.txt.20171126 >> > file3.txt.20171125 >> > >> > where date extension represents order in which particular file was >> consumed >> > by the idempotent file consumer. >> > As a consequence instead of initializing memory cache with newest >> values, >> > it is initialized (probably) based on hash function from truncStore >> method >> > and we consume same file more than once: >> > >> > protected void trunkStore() { >> > LOG.info("Trunking idempotent filestore: {}", fileStore); >> > FileOutputStream fos = null; >> > try { >> > fos = new FileOutputStream(fileStore); >> > for (String key : *cache.keySet()*) { >> > fos.write(key.getBytes()); >> > fos.write(STORE_DELIMITER.getBytes()); >> > } >> > } catch (IOException e) { >> > throw ObjectHelper.wrapRuntimeCamelException(e); >> > } finally { >> > IOHelper.close(fos, "Trunking file idempotent repository", >> LOG); >> > } >> > } >> > >> > LRUCache: >> > >> > @Override >> > public Set keySet() { >> > return map.keySet(); >> > } >> > >> > where previously it was: >> > >> > @Override >> > public Set keySet() { >> > return map.*ascendingKeySet*(); >> > } >> > >> > Regards >> > Kris >> > -- >> > >> > Pozdrawiam >> > >> > Krzysztof Hołdanowicz >> >> >> >> -- >> Claus Ibsen >> - >> http://davsclaus.com @davscl
Re: File idempotent store problem
I don't know if I understood you correctly. Instead of looping via cache.keySet() you mean looping via: Map.Entry entry : cache.entrySet() or cache.foreach((k, v) -> {...})? If yes what is the difference? Isn't it the same unordered collection? If Caffeince returns unordered collection how we can get ordered entries? Isn't it related with: https://github.com/ben-manes/caffeine/issues/86 Shouldn't we use a kind of LinkedHashMap implementation? Regards Kris wt., 28 lis 2017 o 18:40 użytkownik Claus Ibsen-2 [via Camel] < ml+s465427n5815878...@n5.nabble.com> napisał: > Hi > > Ah well spotted. > I think we should for loop via Map.Entry (or (k,v) via lambda) which I > think will be in the correct order. > > You are welcome to log a JIRA. And also work on unit test and patch. > http://camel.apache.org/contributing > > On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz > <[hidden email] <http:///user/SendEmail.jtp?type=node&node=5815878&i=0>> > wrote: > > > Hi all, > > > > I recently noticed that there is wrong entry order in file using > > FileIdempotentRepository implementation. > > The effect is that instead of having order like: > > > > file1.txt.20171123 > > file2.txt.20171123 > > file1.txt.20171124 > > file3.txt.20171125 > > file2.txt.20171126 > > > > we have: > > > > file1.txt.20171123 > > file1.txt.20171124 > > file2.txt.20171123 > > file2.txt.20171126 > > file3.txt.20171125 > > > > where date extension represents order in which particular file was > consumed > > by the idempotent file consumer. > > As a consequence instead of initializing memory cache with newest > values, > > it is initialized (probably) based on hash function from truncStore > method > > and we consume same file more than once: > > > > protected void trunkStore() { > > LOG.info("Trunking idempotent filestore: {}", fileStore); > > FileOutputStream fos = null; > > try { > > fos = new FileOutputStream(fileStore); > > for (String key : *cache.keySet()*) { > > fos.write(key.getBytes()); > > fos.write(STORE_DELIMITER.getBytes()); > > } > > } catch (IOException e) { > > throw ObjectHelper.wrapRuntimeCamelException(e); > > } finally { > > IOHelper.close(fos, "Trunking file idempotent repository", > LOG); > > } > > } > > > > LRUCache: > > > > @Override > > public Set keySet() { > > return map.keySet(); > > } > > > > where previously it was: > > > > @Override > > public Set keySet() { > > return map.*ascendingKeySet*(); > > } > > > > Regards > > Kris > > -- > > > > Pozdrawiam > > > > Krzysztof Hołdanowicz > > > > -- > Claus Ibsen > - > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 > > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://camel.465427.n5.nabble.com/File-idempotent-store-problem-since-2-20-tp5815868p5815878.html > To unsubscribe from Camel - Users, click here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=aG9sZGFub3dpY3pAZ21haWwuY29tfDQ2NTQyOHwtMTk2MTQxMTc0NQ==> > . > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- Pozdrawiam Krzysztof Hołdanowicz
Re: File idempotent store problem
Hi Ah well spotted. I think we should for loop via Map.Entry (or (k,v) via lambda) which I think will be in the correct order. You are welcome to log a JIRA. And also work on unit test and patch. http://camel.apache.org/contributing On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz wrote: > Hi all, > > I recently noticed that there is wrong entry order in file using > FileIdempotentRepository implementation. > The effect is that instead of having order like: > > file1.txt.20171123 > file2.txt.20171123 > file1.txt.20171124 > file3.txt.20171125 > file2.txt.20171126 > > we have: > > file1.txt.20171123 > file1.txt.20171124 > file2.txt.20171123 > file2.txt.20171126 > file3.txt.20171125 > > where date extension represents order in which particular file was consumed > by the idempotent file consumer. > As a consequence instead of initializing memory cache with newest values, > it is initialized (probably) based on hash function from truncStore method > and we consume same file more than once: > > protected void trunkStore() { > LOG.info("Trunking idempotent filestore: {}", fileStore); > FileOutputStream fos = null; > try { > fos = new FileOutputStream(fileStore); > for (String key : *cache.keySet()*) { > fos.write(key.getBytes()); > fos.write(STORE_DELIMITER.getBytes()); > } > } catch (IOException e) { > throw ObjectHelper.wrapRuntimeCamelException(e); > } finally { > IOHelper.close(fos, "Trunking file idempotent repository", LOG); > } > } > > LRUCache: > > @Override > public Set keySet() { > return map.keySet(); > } > > where previously it was: > > @Override > public Set keySet() { > return map.*ascendingKeySet*(); > } > > Regards > Kris > -- > > Pozdrawiam > > Krzysztof Hołdanowicz -- Claus Ibsen - http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
File idempotent store problem
Hi all, I recently noticed that there is wrong entry order in file using FileIdempotentRepository implementation. The effect is that instead of having order like: file1.txt.20171123 file2.txt.20171123 file1.txt.20171124 file3.txt.20171125 file2.txt.20171126 we have: file1.txt.20171123 file1.txt.20171124 file2.txt.20171123 file2.txt.20171126 file3.txt.20171125 where date extension represents order in which particular file was consumed by the idempotent file consumer. As a consequence instead of initializing memory cache with newest values, it is initialized (probably) based on hash function from truncStore method and we consume same file more than once: protected void trunkStore() { LOG.info("Trunking idempotent filestore: {}", fileStore); FileOutputStream fos = null; try { fos = new FileOutputStream(fileStore); for (String key : *cache.keySet()*) { fos.write(key.getBytes()); fos.write(STORE_DELIMITER.getBytes()); } } catch (IOException e) { throw ObjectHelper.wrapRuntimeCamelException(e); } finally { IOHelper.close(fos, "Trunking file idempotent repository", LOG); } } LRUCache: @Override public Set keySet() { return map.keySet(); } where previously it was: @Override public Set keySet() { return map.*ascendingKeySet*(); } Regards Kris -- Pozdrawiam Krzysztof Hołdanowicz