Camel calling commit too early when using split+seda+file endpoint

[camel_el]

Hi,

  I am not sure if this is a bug, or if there is a way around it, but either
way, I wrote a small sample to reproduce the problem I am facing in my
application.

Setup
- camel 2.4.0
- running on Windows 32 bits
- Sun jdk 1.6.0_03

input
2 x .csv files where each one contains 4 tokens separated by commas

code
from("file:e:/test")
  .split(body().tokenize(","))
.to("seda:queue")
.process(new Processor()
{
  public void process(Exchange e) throws Exception
  { Thread.sleep(2000); } // Simulates long processing happening in my
real application
})
  .end()
  .log("End of file ${file:name}");

from("seda:queue)
  .log("Token: ${body}");

Problem
When this code gets executed, it will throw 2 exceptions about Camel cannot
rename the input file.  Camel is trying to rename the input file even though
it hasn't finished tokenizing the file.  As soon as the first "splitted
token" gets passed the sleep processor, the commitRenamer of the
FileStrategy is called and Camel tries to rename the file.  "tokenize(",")"
still has a hold on the file as an input stream I guess, so the file cannot
be renamed.  

Is there a way to delay that rename until the very end of the file (when
tokenize() has been through all the elements inside the file), or is this a
bug?
-- 
View this message in context: 
http://camel.465427.n5.nabble.com/Camel-calling-commit-too-early-when-using-split-seda-file-endpoint-tp2830894p2830894.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel calling commit too early when using split+seda+file endpoint

[camel_el]

This piece of code is a very simplified version of my real code.  My
application is running on an OSGi container and the different stages in my
processing are isolated in different bundles.  So when I need to pass a
message from one stage to another, I need to use a 'vm' endpoint, a.k.a. a
seda queue.

To give you more background, I am processing large files (each file can
contain up to 500,000 items) and I receive a lot of those.  All of my input
is multithreaded, so that I can read many files at the same time.

overview of my real route
from("file:input")
   .threads(5)
   .bean(ObjectIteratorFactory.class, "createIterator")  // this bean
creates an iterator that will read 1 item at a time
// and it
will keep the input file opened until it is done
   .split(body()).streaming()
  .process(firstProcessor)
  .to("vm:secondStageInAnotherBundle")
   .end()
   .log("Done processing file ${file:name}")

So that seda queue (or vm endpoint in my case) is needed since I don't want
to have all of my code inside the same bundle.  If I don't use a seda queue,
it works fine and the file gets deleted at then end, when ObjectIterator has
closed its input stream.  If the seda is involved, Camel tries to close the
file after processing the first item in the file...
-- 
View this message in context: 
http://camel.465427.n5.nabble.com/Camel-calling-commit-too-early-when-using-split-seda-file-endpoint-tp2830894p2834071.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel calling commit too early when using split+seda+file endpoint

[camel_el]

Is there a link I can use to download 2.5.0 (links on
http://camel.apache.org/camel-250-release.html do not work), or I need to go
through the code repo?
-- 
View this message in context: 
http://camel.465427.n5.nabble.com/Camel-calling-commit-too-early-when-using-split-seda-file-endpoint-tp2830894p2839057.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel calling commit too early when using split+seda+file endpoint

[camel_el]

I've tried smcduff's patch on 2.4.0 and it fixed my problem.  Like he has
mentioned on https://issues.apache.org/activemq/browse/CAMEL-3121, we had to
modify his original patch slightly to make it work at shutdown time.  I'll
still give 2.5.0 a try when I get access to it.
-- 
View this message in context: 
http://camel.465427.n5.nabble.com/Camel-calling-commit-too-early-when-using-split-seda-file-endpoint-tp2830894p2839063.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel calling commit too early when using split+seda+file endpoint

[camel_el]

Claus,

  can you try the following piece of code and check if it works or not?  (it
doesn't work on my workstation, using camel 2.5.0)

Code
from("file:e:/test")
  .split(body().tokenize(","))
.log("Split line ${body}")
.process(new Processor()
{
  public void process(Exchange e) throws Exception
  { Thread.sleep(2000); } // Simulates long processing happening in my
real application
})
  .end()
  .log("End of file ${file:name}");

Important: make sure you are on Windows and you need at least two input
files in e:\test to see the error.  It always happen when camel tries to
move the second file (could be a race condition?)

Let me know.
Thanks

-- 
View this message in context: 
http://camel.465427.n5.nabble.com/Camel-calling-commit-too-early-when-using-split-seda-file-endpoint-tp2830894p2839249.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel calling commit too early when using split+seda+file endpoint

[camel_el]

Claus,

  is there a new snapshot of 2.5.0 that I can use to test your fix?  The one
on the Maven repo
(https://repository.apache.org/content/repositories/snapshots/org/apache/camel/camel-core/2.5-SNAPSHOT/)
is too old (Aug 10th).  All my stuff is on a private network, so I can't
access your SVN directly.  It would be nice if I can avoid building it.  

If not, were your changes isolated to Splitter.java only?  If so, I can
easily steal that class only and replace it in my .jar.

Let me know
Thanks!
-- 
View this message in context: 
http://camel.465427.n5.nabble.com/Camel-calling-commit-too-early-when-using-split-seda-file-endpoint-tp2830894p2842711.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel calling commit too early when using split+seda+file endpoint

[camel_el]

Sorry for the late reply, but I had the chance to try out your latest version
of 2.5.0 this morning and I can confirm that your fix worked.  

Thanks!  :)
-- 
View this message in context: 
http://camel.465427.n5.nabble.com/Camel-calling-commit-too-early-when-using-split-seda-file-endpoint-tp2830894p2846824.html
Sent from the Camel - Users mailing list archive at Nabble.com.

UuidGenerator performance issue

[camel_el]

Hi,

  my system is generating a lot of exchanges (many concurrent threads), so
many that creating an exchange becomes a bottleneck.  An exchangeId is
created by the UuidGenerator class, which is using
java.security.SecureRandom.  SecureRandom then calls its secureRandomSpi:

UuidGenerator.generateUuid() -> UUID.randomUUID() ->
secureRandom.nextBytes(n) -> secureRandomSpi.engineNextBytes(n) 

where secureRandomSpi.engineNextBytes(n) is synchronized.  Obviously, this
call becomes a bottleneck.  I have modified UuidGenerator so that it creates
a 1-up number (probably not the best, but it worked for the time being) and
I saw a major performance gain.

Would it be possible to:

1- Change the way Camel creates the exchangeId?
 - or -
2- Provide a hook where we could supply our own ExchangeIdFactory?  
If we use our own factory, what are the constraints?

Thanks.
-- 
View this message in context: 
http://camel.465427.n5.nabble.com/UuidGenerator-performance-issue-tp2846866p2846866.html
Sent from the Camel - Users mailing list archive at Nabble.com.