Hi What about the line separators on the big file? Are it just \n or is it maybe \n\r or \r etc? Maybe the tokenizer need to be adjusted?
You can also try a little java main app and try to use the java util Scanner to read the same file and see if you can do that without any issues. And is there something special in about row 3000 where it "seems to stop" ? On Thu, Feb 13, 2014 at 11:53 PM, jay <mylifemyrulesmyw...@gmail.com> wrote: > Hi I have a requirement to process 2 huge files from remote server. > > Step1:i have written route to SFTP the file from remote server to local > server - working fine. > > from(from).autoStartup(true).to(to) > > from--> > sftp://user@xxxx?password=xxxx&localWorkDirectory=/tmp&readLock=changed&idempotent=true&move=processed > to--> file:///opt/mydata > > Step2: consuming the file from local directory and splitting that and > sending to queue. > > from(from) > .routeId("split route") > > .split(body().tokenize("\n")).streaming().convertBodyTo(String.class) > .process(new Processor()) > .to(splitQueue); > > from-->file:///opt/mydata?move=../processed&readLock=changed&idempotent=true > splitQueue--->activemq:splitqueue > > > I am running the above in Fuse ESB and is working OK in Macbook - but when > i deploy the bundle in linux server - first file (small file 3500 records) > went through OK but when it process the second file Step2 route is keep on > reading the huge file (6million records) and keep on sending it to queue > with out stopping (i have noticed it read only 3000 rows and keep on sending > the same 3000 rows). > > > exception: > 21:44:26,537 | ERROR | /opt/mydata | DefaultErrorHandler | 147 > - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Failed delivery for > (MessageId: ID-ABC-41834-1392325070932-5-167921 on ExchangeId: > ID-ABC-41834-1392325070932-5-167922). Exhausted after delivery attempt: 1 > caught: org.apache.camel.RuntimeCamelException: Scanner aborted because of > an IOException! > org.apache.camel.RuntimeCamelException: Scanner aborted because of an > IOException! > at > org.apache.camel.processor.Splitter$1$1.hasNext(Splitter.java:156)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047] > > Caused by: java.nio.charset.MalformedInputException: Input length = 1 > at > java.nio.charset.CoderResult.throwException(CoderResult.java:260)[:1.6.0_45] > at > sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:319)[:1.6.0_45] > > can some one help me please as it is really stopper issue for my delivery? > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/File-Component-route-issue-tp5747281.html > Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen ----------------- Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen Make your Camel applications look hawt, try: http://hawt.io
