Thanks Sven, I owe you a beer. zipSplitter works great. Cheers, Thomas.
Am 03.09.2014 um 07:36 schrieb Sven Nold <sven.n...@isb-ag.de>: > Hi, > > sorry didn't fix it, still the same error. > > > > -----Original Message----- > From: Claus Ibsen [mailto:claus.ib...@gmail.com] > Sent: Tuesday, September 02, 2014 4:07 PM > To: users@camel.apache.org > Subject: Re: camel 2.13.2 zipdataformat > > Hi > > Thanks. I suspect its because this code creates a new input stream in the > message body > https://github.com/apache/camel/blob/master/components/camel-zipfile/src/main/java/org/apache/camel/dataformat/zipfile/ZipIterator.java#L106 > > That is not explicit closed. > > Can you try changing > > answer.setBody(new ZipInputStreamWrapper(zipInputStream)); > > To > > answer.setBody(zipInputStream); > > And give it a test on your Windows system? > > On Tue, Sep 2, 2014 at 3:13 PM, Sven Nold <sven.n...@isb-ag.de> wrote: >> Hi, >> >> same result, but workaround is to use zipSplitter like this: >> >> <bean id="zipSplitter" >> class="org.apache.camel.dataformat.zipfile.ZipSplitter" /> >> >> <route id="readZip" streamCache="true"> >> <from >> uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/> >> <!-- <unmarshal ref="zipFileDataFormat"/> --> >> <camel:split streaming="true"> >> <!-- <camel:simple>${body}</camel:simple> --> >> <camel:ref>zipSplitter</camel:ref> >> <camel:to >> uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/> >> <camel:to uri="file://target/unzipped"/> >> </camel:split> >> <camel:to >> uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/> >> </route> >> >> ZipSplitter seems not to have this issue. Also works for 2.13.2. which could >> be helpful for you @Thomas. >> >> So from my point of view CAMEL-7577 is not RESOLVED. >> >> Best regards, >> >> -----Ursprüngliche Nachricht----- >> Von: Claus Ibsen [mailto:claus.ib...@gmail.com] >> Gesendet: Dienstag, 2. September 2014 14:29 >> An: users@camel.apache.org >> Betreff: Re: camel 2.13.2 zipdataformat >> >> You need to build the code to make sure the snapshot is using recent code. >> >> On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <sven.n...@isb-ag.de> wrote: >>> Hi Claus, >>> >>> I changed to 2.13.3-SNAPSHOT but still the same issue. >>> >>> Best regards, >>> >>> Sven >>> >>> -----Ursprüngliche Nachricht----- >>> Von: Claus Ibsen [mailto:claus.ib...@gmail.com] >>> Gesendet: Dienstag, 2. September 2014 13:52 >>> An: users@camel.apache.org >>> Betreff: Re: camel 2.13.2 zipdataformat >>> >>> Hi >>> >>> See CAMEL-7577 >>> >>> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <sven.n...@isb-ag.de> wrote: >>>> Hi, >>>> >>>> interesting is also that if I only have on zipEntry and define the >>>> zipDataFormat as useIterator=false it works fine. So my guess is that >>>> ZipIterator somehow holds a lock on the file: >>>> >>>> <bean id="zipFileDataFormat" >>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat"> >>>> <property name="usingIterator" value="false"/> >>>> </bean> >>>> >>>> <bean id="zipAggregationStrategy" >>>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationSt >>>> r >>>> a >>>> tegy"/> >>>> >>>> <camelContext xmlns="http://camel.apache.org/schema/spring";> >>>> <!-- write the zip so we can process something repeatedly --> >>>> <route id="writeZip"> >>>> <from >>>> uri="file://src/main/data?antInclude=message1.xml&noop=true"/> >>>> <aggregate strategyRef="zipAggregationStrategy" >>>> completionFromBatchConsumer="true" eagerCheckCompletion="true" >>>> completionSize="2"> >>>> <correlationExpression> >>>> <constant>true</constant> >>>> </correlationExpression> >>>> <to uri="file://target?fileName=any.xxx"/> >>>> </aggregate> >>>> </route> >>>> >>>> <!-- extract the zip --> >>>> <route id="readZip" streamCache="true"> >>>> <from >>>> uri="file://target?fileName=any.xxx&moveFailed=.error&move=.done&delay=5000"/> >>>> <unmarshal ref="zipFileDataFormat"/> >>>> <!--<camel:split streaming="true"> >>>> <camel:simple>${body}</camel:simple> >>>> <camel:to >>>> uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/> >>>> <camel:to uri="file://target/unzipped"/> >>>> </camel:split> --> >>>> <camel:to >>>> uri="log:org.apache.camel?level=INFO&showAll=true&multiline=true&showStreams=true"/> >>>> </route> >>>> </camelContext> >>>> >>>> Works fine. >>>> >>>> Best regards, >>>> >>>> >>>> >>>> -----Ursprüngliche Nachricht----- >>>> Von: Sven Nold [mailto:sven.n...@isb-ag.de] >>>> Gesendet: Dienstag, 2. September 2014 13:38 >>>> An: users@camel.apache.org >>>> Betreff: AW: camel 2.13.2 zipdataformat >>>> >>>> Hi, >>>> >>>> thanks for this test now I think here is the problem: >>>> >>>> [l-1) thread #1 - file://target] GenericFileOnCompletion WARN >>>> Error during commit. Exchange[any.zip]. Caused by: >>>> [org.apache.camel.component.file.GenericFileOperationFailedException - >>>> Error renaming file from >>>> C:\temp\camel-examples-master\zip-file-processing\target\any.zip to >>>> target\.done\any.zip] >>>> org.apache.camel.component.file.GenericFileOperationFailedException: Error >>>> renaming file from >>>> C:\temp\camel-examples-master\zip-file-processing\target\any.zip to >>>> target\.done\any.zip >>>> at >>>> org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77) >>>> at >>>> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113) >>>> at >>>> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88) >>>> at >>>> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124) >>>> at >>>> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80) >>>> at >>>> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54) >>>> at >>>> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100) >>>> at >>>> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228) >>>> at >>>> org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61) >>>> at >>>> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613) >>>> at >>>> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581) >>>> at >>>> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240) >>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:106) >>>> at >>>> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175) >>>> at >>>> org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) >>>> at >>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.io.IOException: Renaming file from >>>> 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' to >>>> 'target\.done\any.zip' failed: Cannot delete file >>>> 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip' after >>>> copy succeeded >>>> at >>>> org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456) >>>> at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428) >>>> at >>>> org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74) >>>> ... 25 more >>>> >>>> This is why it grabs the file all the time. >>>> >>>> Regarding your questions: >>>> >>>> 1) well thanks to your test you can see error above >>>> 2) yes it is processing it over and over again >>>> 3) no doesn't move to .error >>>> 4) no file is not updated, in your test case I am not quite sure, will >>>> increase delay or as you suggested preMove option. >>>> >>>> Due to known problems off windows and zip files I even renamed it to .xxx >>>> to keep away explorer from analyzing, but still it seems there is a handle >>>> on that file. >>>> And again if I remove the unmarshal and split it works fine. >>>> >>>> Thanks for any help. >>>> >>>> Best regards, >>>> >>>> >>>> -----Ursprüngliche Nachricht----- >>>> Von: Jimmi Dyson [mailto:jimmidy...@gmail.com] >>>> Gesendet: Dienstag, 2. September 2014 10:37 >>>> An: users@camel.apache.org >>>> Betreff: Re: camel 2.13.2 zipdataformat >>>> >>>> Hi, >>>> >>>> I just added an example at >>>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing. >>>> This example creates the zip file first (so that it's repeatable) & then >>>> does pretty much what you're doing: read from file uri, unmarshal, log >>>> each file contents (careful with large files) & finally writes each file >>>> to a file. Seems to be doing what it should... >>>> >>>> Few questions if the example doesn't help you: >>>> >>>> - Are there any hints in the logs (try increasing logging level if not)? >>>> - If the file is back in the source dir does it try to process again? >>>> - Is there anything in the .error (moveFailed dir)? >>>> - Could the source file be being updated while you're processing it - if >>>> so, can you try the preMove option? >>>> >>>> Thanks, >>>> >>>> >>>> On 2 September 2014 08:46, Walzer, Thomas >>>> <thomas.wal...@integratix.net> >>>> wrote: >>>> >>>>> Hi Sven, >>>>> >>>>> I seem to have a similar problem. I also use 2.13.2. >>>>> My code is almost identical. The only difference is my in-url, that >>>>> looks >>>>> like: >>>>> >>>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${ >>>>> f i l e:name}.${date:now:HHmmss}&antInclude=*.ZIP >>>>> >>>>> I get a random number of retrievals (about 4-20) and therefore the >>>>> same number of files in the „archive“ directory. The timestamps are >>>>> 6 seconds apart. >>>>> >>>>> Cheers, Thomas. >>>>> >>>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <sven.n...@isb-ag.de>: >>>>> >>>>>> Hi Guys, >>>>>> >>>>>> I am trying to read Zipentries and process them separately. But >>>>>> using >>>>> the file component it seems the file is not correctly closed. I can >>>>> see the file for a few seconds in the .done and source directory. >>>>> But then it disappears in the .done folder and is back in the source >>>>> directory : >>>>>> >>>>>> >>>>>> <bean id="zipFileDataFormat" >>>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat"> >>>>>> <property name="usingIterator" value="true" /> </bean> >>>>>> >>>>>> <route id="readZip"> >>>>>> <from >>>>> uri="file://C:/temp/?fileName=any.zip&moveFailed=.error&move=.done&delay=10000" >>>>> /> >>>>>> <unmarshal ref="zipFileDataFormat" /> >>>>>> <!-- <split streaming="true"> --> >>>>>> <!-- <simple>$simple{body}</simple> --> >>>>>> <!-- <to >>>>> uri="log://OUT?level=INFO&showAll=true&multiline=true" /> >>>>> --> >>>>>> <!-- </split> --> >>>>>> <to uri="log://OUT?level=INFO&showAll=true&multiline=true" >>>>> /> >>>>>> </route> >>>>>> >>>>>> If I remove the unmarshal the file component works fine (it moves >>>>> processed file to .done). >>>>>> >>>>>> Any Ideas? I guess I could do a workaround by using noop=true and >>>>>> moving >>>>> the file myself, but it would be nice if it works as expected. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Sven >>>>> >>>>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> Red Hat, Inc. >>> Email: cib...@redhat.com >>> Twitter: davsclaus >>> Blog: http://davsclaus.com >>> Author of Camel in Action: http://www.manning.com/ibsen >>> hawtio: http://hawt.io/ >>> fabric8: http://fabric8.io/ >> >> >> >> -- >> Claus Ibsen >> ----------------- >> Red Hat, Inc. >> Email: cib...@redhat.com >> Twitter: davsclaus >> Blog: http://davsclaus.com >> Author of Camel in Action: http://www.manning.com/ibsen >> hawtio: http://hawt.io/ >> fabric8: http://fabric8.io/ > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > Email: cib...@redhat.com > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen > hawtio: http://hawt.io/ > fabric8: http://fabric8.io/
