There are points i agree/disagree.
This can stay as the limitation imho. Because using cache as fifo sounds a
bit advance care of idempotent repository feature and it would require
taking all the burdens of such implementation for a single minor feature by
diving into caffeine. This would be too much to implement in camel-core
ootb. in case you require such an advanced feature, idempotent repository
is a pluggable feature and it can be implemented with fifo based custom
cache as my first sight and imho.
On Wed, 3 Jan 2018 at 22:51, Krzysztof Hołdanowicz <holdanow...@gmail.com>
wrote:
> Hi,
>
> regarding CAMEL-12058 I don't know if you are aware of all consequences of
> wrong order in the idempotent file store.
> The wrong order in the file is not the problem itself as long as elemens
> are added and eviceted on runtime, because caffeine provides an api for
> ordering like:
>
> - @Override public Map<K, V> coldest(int limit)
> - @Override public Map<K, V> hottest(int limit)
> - @Override public Map<K, V> oldest(int limit),
> - @Override public Map<K, V> youngest(int limit) )
>
> however the consequences of this appears after RESTART. The memory cache
> does not contain the proper entries (in case of reaching the max limit
> size) because it does not load elements from hottest to coldest but with
> the file entries order hence some of the files are consumed multiple times.
> It means that current implementation of file idempotent store is not usable
> at all anymore. Ignoring the issue (CAMEL-12058) means that Camel library
> does not provide any implementation of idempotent file store as the current
> behaviour is completely wrong and causes consuming multiple times the same
> file after reaching max size limit and after restarting application.
>
>
> Regards
> Kris
>
> sob., 2 gru 2017 o 15:14 użytkownik Krzysztof Hołdanowicz <
> holdanow...@gmail.com> napisał:
>
> > I don't know if I understood you correctly.
> > Instead of looping via cache.keySet() you mean looping via:
> > Map.Entry<String, Object> entry : cache.entrySet() or cache.foreach((k,
> v)
> > -> {...})?
> >
> > If yes what is the difference? Isn't it the same unordered collection?
> > If Caffeince returns unordered collection how we can get ordered entries?
> > Isn't it related with:
> > https://github.com/ben-manes/caffeine/issues/86
> >
> > Shouldn't we use a kind of LinkedHashMap implementation?
> >
> > Regards
> > Kris
> >
> > wt., 28 lis 2017 o 18:40 użytkownik Claus Ibsen-2 [via Camel] <
> > ml+s465427n5815878...@n5.nabble.com> napisał:
> >
> >> Hi
> >>
> >> Ah well spotted.
> >> I think we should for loop via Map.Entry (or (k,v) via lambda) which I
> >> think will be in the correct order.
> >>
> >> You are welcome to log a JIRA. And also work on unit test and patch.
> >> http://camel.apache.org/contributing
> >>
> >> On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz
> >> <[hidden email] <http://
> /user/SendEmail.jtp?type=node&node=5815878&i=0>>
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I recently noticed that there is wrong entry order in file using
> >> > FileIdempotentRepository implementation.
> >> > The effect is that instead of having order like:
> >> >
> >> > file1.txt.20171123
> >> > file2.txt.20171123
> >> > file1.txt.20171124
> >> > file3.txt.20171125
> >> > file2.txt.20171126
> >> >
> >> > we have:
> >> >
> >> > file1.txt.20171123
> >> > file1.txt.20171124
> >> > file2.txt.20171123
> >> > file2.txt.20171126
> >> > file3.txt.20171125
> >> >
> >> > where date extension represents order in which particular file was
> >> consumed
> >> > by the idempotent file consumer.
> >> > As a consequence instead of initializing memory cache with newest
> >> values,
> >> > it is initialized (probably) based on hash function from truncStore
> >> method
> >> > and we consume same file more than once:
> >> >
> >> > protected void trunkStore() {
> >> > LOG.info("Trunking idempotent filestore: {}", fileStore);
> >> > FileOutputStream fos = null;
> >> > try {
> >> > fos = new FileOutputStream(fileStore);
> >> > for (String key : *cache.keySet()*) {
> >> > fos.write(key.getBytes());
> >> > fos.write(STORE_DELIMITER.getBytes());
> >> > }
> >> > } catch (IOException e) {
> >> > throw ObjectHelper.wrapRuntimeCamelException(e);
> >> > } finally {
> >> > IOHelper.close(fos, "Trunking file idempotent repository",
> >> LOG);
> >> > }
> >> > }
> >> >
> >> > LRUCache:
> >> >
> >> > @Override
> >> > public Set<K> keySet() {
> >> > return map.keySet();
> >> > }
> >> >
> >> > where previously it was:
> >> >
> >> > @Override
> >> > public Set<K> keySet() {
> >> > return map.*ascendingKeySet*();
> >> > }
> >> >
> >> > Regards
> >> > Kris
> >> > --
> >> >
> >> > Pozdrawiam
> >> >
> >> > Krzysztof Hołdanowicz
> >>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> -----------------
> >> http://davsclaus.com @davsclaus
> >> Camel in Action 2: https://www.manning.com/ibsen2
> >>
> >>
> >> ------------------------------
> >> If you reply to this email, your message will be added to the discussion
> >> below:
> >>
> >>
> http://camel.465427.n5.nabble.com/File-idempotent-store-problem-since-2-20-tp5815868p5815878.html
> >> To unsubscribe from Camel - Users, click here
> >> <
> http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=aG9sZGFub3dpY3pAZ21haWwuY29tfDQ2NTQyOHwtMTk2MTQxMTc0NQ==
> >
> >> .
> >> NAML
> >> <
> http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >>
> > --
> >
> > Pozdrawiam
> >
> > Krzysztof Hołdanowicz
> >
>
>
> --
>
> Pozdrawiam
>
> Krzysztof Hołdanowicz
>
