Hi,
regarding CAMEL-12058 I don't know if you are aware of all consequences of
wrong order in the idempotent file store.
The wrong order in the file is not the problem itself as long as elemens
are added and eviceted on runtime, because caffeine provides an api for
ordering like:
- @Override public Map<K, V> coldest(int limit)
- @Override public Map<K, V> hottest(int limit)
- @Override public Map<K, V> oldest(int limit),
- @Override public Map<K, V> youngest(int limit) )
however the consequences of this appears after RESTART. The memory cache
does not contain the proper entries (in case of reaching the max limit
size) because it does not load elements from hottest to coldest but with
the file entries order hence some of the files are consumed multiple times.
It means that current implementation of file idempotent store is not usable
at all anymore. Ignoring the issue (CAMEL-12058) means that Camel library
does not provide any implementation of idempotent file store as the current
behaviour is completely wrong and causes consuming multiple times the same
file after reaching max size limit and after restarting application.
Regards
Kris
sob., 2 gru 2017 o 15:14 użytkownik Krzysztof Hołdanowicz <
holdanow...@gmail.com> napisał:
> I don't know if I understood you correctly.
> Instead of looping via cache.keySet() you mean looping via:
> Map.Entry<String, Object> entry : cache.entrySet() or cache.foreach((k, v)
> -> {...})?
>
> If yes what is the difference? Isn't it the same unordered collection?
> If Caffeince returns unordered collection how we can get ordered entries?
> Isn't it related with:
> https://github.com/ben-manes/caffeine/issues/86
>
> Shouldn't we use a kind of LinkedHashMap implementation?
>
> Regards
> Kris
>
> wt., 28 lis 2017 o 18:40 użytkownik Claus Ibsen-2 [via Camel] <
> ml+s465427n5815878...@n5.nabble.com> napisał:
>
>> Hi
>>
>> Ah well spotted.
>> I think we should for loop via Map.Entry (or (k,v) via lambda) which I
>> think will be in the correct order.
>>
>> You are welcome to log a JIRA. And also work on unit test and patch.
>> http://camel.apache.org/contributing
>>
>> On Tue, Nov 28, 2017 at 8:55 AM, Krzysztof Hołdanowicz
>> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=5815878&i=0>>
>> wrote:
>>
>> > Hi all,
>> >
>> > I recently noticed that there is wrong entry order in file using
>> > FileIdempotentRepository implementation.
>> > The effect is that instead of having order like:
>> >
>> > file1.txt.20171123
>> > file2.txt.20171123
>> > file1.txt.20171124
>> > file3.txt.20171125
>> > file2.txt.20171126
>> >
>> > we have:
>> >
>> > file1.txt.20171123
>> > file1.txt.20171124
>> > file2.txt.20171123
>> > file2.txt.20171126
>> > file3.txt.20171125
>> >
>> > where date extension represents order in which particular file was
>> consumed
>> > by the idempotent file consumer.
>> > As a consequence instead of initializing memory cache with newest
>> values,
>> > it is initialized (probably) based on hash function from truncStore
>> method
>> > and we consume same file more than once:
>> >
>> > protected void trunkStore() {
>> > LOG.info("Trunking idempotent filestore: {}", fileStore);
>> > FileOutputStream fos = null;
>> > try {
>> > fos = new FileOutputStream(fileStore);
>> > for (String key : *cache.keySet()*) {
>> > fos.write(key.getBytes());
>> > fos.write(STORE_DELIMITER.getBytes());
>> > }
>> > } catch (IOException e) {
>> > throw ObjectHelper.wrapRuntimeCamelException(e);
>> > } finally {
>> > IOHelper.close(fos, "Trunking file idempotent repository",
>> LOG);
>> > }
>> > }
>> >
>> > LRUCache:
>> >
>> > @Override
>> > public Set<K> keySet() {
>> > return map.keySet();
>> > }
>> >
>> > where previously it was:
>> >
>> > @Override
>> > public Set<K> keySet() {
>> > return map.*ascendingKeySet*();
>> > }
>> >
>> > Regards
>> > Kris
>> > --
>> >
>> > Pozdrawiam
>> >
>> > Krzysztof Hołdanowicz
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://camel.465427.n5.nabble.com/File-idempotent-store-problem-since-2-20-tp5815868p5815878.html
>> To unsubscribe from Camel - Users, click here
>> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=aG9sZGFub3dpY3pAZ21haWwuY29tfDQ2NTQyOHwtMTk2MTQxMTc0NQ==>
>> .
>> NAML
>> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
> --
>
> Pozdrawiam
>
> Krzysztof Hołdanowicz
>
--
Pozdrawiam
Krzysztof Hołdanowicz
