Re: handling large files

2009-10-21 Thread mcarson
This problem was solved by loosening the configuration of ActiveMQ to allow for fast producers. bsnyder wrote: > > > I'm not sure what kind of configuration you're using for ActiveMQ, but > are you utilizing a connection pooler so that the the connection, > session and producer are not recreat

Re: handling large files

2009-10-21 Thread mcarson
bsnyder wrote: > > On Fri, Oct 16, 2009 at 7:41 AM, mcarson wrote: >> >> Using the scanner seems to work for parsing down the huge file based upon >> a >> delimiter.  However it appears that either the JmsTemplate I'm using to >> send >> message

Re: handling large files

2009-10-16 Thread mcarson
Using the scanner seems to work for parsing down the huge file based upon a delimiter. However it appears that either the JmsTemplate I'm using to send messages or ActiveMQ cannot keep pace. Somewhere between 250K - 500K sends, I get this stack trace: Exception in thread "main" org.springframew

Re: handling large files

2009-10-14 Thread mcarson
14, 2009 at 4:21 PM, Claus Ibsen >> wrote: >>> Hi >>> >>> On Wed, Oct 14, 2009 at 4:16 PM, mcarson wrote: >>>> >>>> It looks like the scanner might provide me with the capabilities I was >>>> looking for regarding reading in a fi

Re: handling large files

2009-10-14 Thread mcarson
It looks like the scanner might provide me with the capabilities I was looking for regarding reading in a file in delimited chunks. I'm assuming I would implement this as a bean... can the bean component be used as a "from" in a camel route? I'm new to Camel, and I have never seen that done. Is

Re: handling large files

2009-10-14 Thread mcarson
I would like to split using xpath if possible: /Envelope/Body/person Claus Ibsen-2 wrote: > > Hi > > How do you want to split the file? > Is there a special character that denotes a new "record" > > Using java.util.Scanner is great as it can do streaming. And also what > Camel can do if you

Re: handling large files

2009-10-14 Thread mcarson
I have to solve this exact same problem! I am also trying to: -- read an extremely large file from a file system -- doing a split on that file -- send each split record to a JMS queue However I am relatively new to Camel.. I do not understand how the stream component can help specify ho