Hi Claus. Ack, sorry about that. I'm pulling through github rather than SVN directly and then running via a Maven project pointing at my locally built Camel, so it's sometimes confusing how up-to-date I am.
So I think I have current code, including checkin "CAMEL-3174: Changing dir with ftp must do one dir at a time." and I think SFTP polling is now broken for both absolute and relative URLs. See debug walk-throughs and stack traces below. Absolute URI with double leading /: sftp://u...@host//home/user/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=****** SftpOperations.changeCurrentDirectory("home/user/subdir") //note there's no leading slash on the supposedly absolute path! //splits into chunks successfully, starts looping through chunks doChangeDirectory("home") ChannelSftp.cd("home") //path=remoteAbsolutePath(path); //yields "/home/user/home". Doh! Relative URI with single leading /: sftp://u...@host/subdir?idempotent=true&idempotentRepository=#myCustomIR&include=.*.log&noop=true&password=****** SftpOperations.changeCurrentDirectory("subdir") //splits into single chunk, starts looping through chunks doChangeDirectory("subdir") ChannelSftp.cd("home") //path=remoteAbsolutePath(path); //yields "/home/user/subdir". yay! ..but then, back in SftpOperations.retrieveFileToStreamInBody, it hits the "change back to current directory code" changeCurrentDirectory("/home/tomcat"); //now we have a leading slash, unlike before //splits info chunks "", "home", and "tomcat". Doh! doChangeDirectory("") ChannelSftp.cd("") //path=remoteAbsolutePath(path) throws a StringIndexOutOfBoundsException when it receives empty string. Stack trace when attempting to supply absolute path: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: home at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372) at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361) at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:428) at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407) at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299) at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155) at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121) at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) Caused by: 2: No such file at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289) at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822) at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268) at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370) ... 16 more Stack trace when relative URL tries to restore current working directory: SftpConsumer 2010-10-01 13:41:58,529 -- ERROR -- Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot change directory to: ] org.apache.camel.component.file.GenericFileOperationFailedException: Cannot change directory to: at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:372) at org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:361) at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:435) at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:407) at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299) at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155) at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121) at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) Caused by: 4: at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:285) at org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:370) ... 16 more Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.charAt(String.java:686) at com.jcraft.jsch.ChannelSftp.remoteAbsolutePath(ChannelSftp.java:2359) at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:264) ... 17 more Cheers -Lorrin On Sep 29, 2010, at 9:37 PM, Claus Ibsen wrote: > Hi > > You have not used the latest source code. The stacktrace points to > empty lines etc. > Make sure you build camel-core and camel-ftp using latest source code. > > Revision: 1002946 > Node Kind: directory > Schedule: normal > Last Changed Author: davsclaus > Last Changed Rev: 1002821 > Last Changed Date: 2010-09-29 21:31:23 +0200 (Wed, 29 Sep 2010) > > > On Wed, Sep 29, 2010 at 11:55 PM, Lorrin Nelson > <lhn_git...@nerdylorrin.net> wrote: >> I'm pretty sure I was already at revision 1002541 before. Now I definitely >> am. Same Exception. I think the problem is the lack of leading /. >> >> SftpOperations.retrieveFileToStreamInBody computes path as "tmp/mult", calls >> changeCurrentDirectory(path), which in turn calls channel.cd(path), which >> (now we're in com.jcraft.jsch.ChannelSftp) calls >> path=remoteAbsolutePath(path); and computes path: "/home/test/tmp/mult", >> which does not exist. >> >> SftpConsumer 2010-09-29 14:51:50,487 -- ERROR -- Caused by: >> [org.apache.camel.component.file.GenericFileOperationFailedException - >> Cannot change current directory to: tmp/mult] >> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot >> change current directory to: tmp/mult >> at >> org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352) >> at >> org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408) >> at >> org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387) >> at >> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299) >> at >> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155) >> at >> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121) >> at >> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:637) >> Caused by: 2: No such file >> at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289) >> at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822) >> at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268) >> at >> org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350) >> ... 15 more >> >> >> Cheers! >> -Lorrin >> >> On Sep 29, 2010, at 12:32 PM, Claus Ibsen wrote: >> >>> Hi >>> >>> Can you try again with latest source code from trunk. I have changed >>> the logic to change dir one folder at a time. >>> >>> >>> On Wed, Sep 29, 2010 at 7:47 PM, Lorrin Nelson >>> <lhn_git...@nerdylorrin.net> wrote: >>>> Thanks for the quick response. I'm seeing a different failure now ("Cannot >>>> change current directory to: tmp/mult" rather than "Cannot retrieve file: >>>> tmp/mult/dummy-5.log"). Is it suspicious that in either case there is no >>>> leading /? >>>> >>>> I tried uris >>>> sftp://t...@host//tmp/mult?include=.*\.log >>>> and >>>> sftp://t...@host///tmp/mult?include=.*\.log >>>> >>>> SftpConsumer 2010-09-29 10:31:48,401 -- ERROR -- Caused by: >>>> [org.apache.camel.component.file.GenericFileOperationFailedException - >>>> Cannot change current directory to: tmp/mult] >>>> org.apache.camel.component.file.GenericFileOperationFailedException: >>>> Cannot change current directory to: tmp/mult >>>> at >>>> org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:352) >>>> at >>>> org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:408) >>>> at >>>> org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121) >>>> at >>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:637) >>>> Caused by: 2: No such file >>>> at >>>> com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289) >>>> at com.jcraft.jsch.ChannelSftp._realpath(ChannelSftp.java:1822) >>>> at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:268) >>>> at >>>> org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:350) >>>> ... 15 more >>>> >>>> On a tangentially related note, there's some file name processing during >>>> the move step that throws a NullPointerException when the URI contains a >>>> trailing slash: >>>> >>>> sftp://t...@host/tmp/mult/?include=.*\.log >>>> >>>> The problem is that GenericFile.changeFileName calls >>>> newFileName = ObjectHelper.after(newFileName, endpointPath + >>>> getFileSeparator()); >>>> ...without first checking if endPointPath contains a trailing separator. >>>> >>>> E.g. >>>> newFileName: "tmp/mult/.done/dummy-2.log" >>>> endPointPath: "tmp/mult/" >>>> >>>> ObjectHelper.after(): >>>> text: "tmp/mult/.done/dummy-2.log" >>>> after: "tmp/mult//" >>>> >>>> text doesn't contain after (because of double //), returns null, >>>> changeFileName() doesn't expect null, throws NPE. >>>> >>>> GenericFileOnCompletion 2010-09-29 10:37:37,888 -- ERROR -- Caused by: >>>> [java.lang.NullPointerException - null] >>>> java.lang.NullPointerException >>>> at java.io.File.<init>(File.java:222) >>>> at >>>> org.apache.camel.component.file.GenericFile.changeFileName(GenericFile.java:169) >>>> at >>>> org.apache.camel.component.file.strategy.GenericFileExpressionRenamer.renameFile(GenericFileExpressionRenamer.java:41) >>>> at >>>> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:82) >>>> at >>>> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:121) >>>> at >>>> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83) >>>> at >>>> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52) >>>> at >>>> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:55) >>>> at >>>> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:173) >>>> at >>>> org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121) >>>> at >>>> org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36) >>>> at >>>> org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106) >>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:130) >>>> at >>>> org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99) >>>> at >>>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70) >>>> at >>>> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) >>>> at >>>> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) >>>> at >>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155) >>>> at >>>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121) >>>> at >>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:637) >>>> >>>> >>>> >>>> >>>> On Sep 29, 2010, at 2:46 AM, Claus Ibsen wrote: >>>> >>>>> Hi >>>>> >>>>> I have committed a fix to the SFTP component. Can you try with latest >>>>> source code from trunk? >>>>> >>>>> >>>>> On Tue, Sep 28, 2010 at 7:36 PM, Lorrin <lhn_git...@nerdylorrin.net> >>>>> wrote: >>>>>> >>>>>> I just bumped into the same problem. I'm running 2.5-SNAPSHOT. >>>>>> >>>>>> SftpConsumer 2010-09-28 10:31:28,094 -- ERROR -- Caused by: >>>>>> [org.apache.camel.component.file.GenericFileOperationFailedException - >>>>>> Cannot retrieve file: tmp/mult/dummy-5.log] >>>>>> org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>> Cannot >>>>>> retrieve file: tmp/mult/dummy-5.log >>>>>> at >>>>>> org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:403) >>>>>> at >>>>>> org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:387) >>>>>> at >>>>>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:299) >>>>>> at >>>>>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155) >>>>>> at >>>>>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121) >>>>>> at >>>>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>>>>> at >>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>>>>> at >>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>>> at java.lang.Thread.run(Thread.java:637) >>>>>> Caused by: 2: No such file >>>>>> at >>>>>> com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289) >>>>>> at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:901) >>>>>> at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:870) >>>>>> at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:849) >>>>>> at >>>>>> org.apache.camel.component.file.remote.SftpOperations.retrieveFileToStreamInBody(SftpOperations.java:400) >>>>>> ... 14 more >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-tp2806817p2857187.html >>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Claus Ibsen >>>>> Apache Camel Committer >>>>> >>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>> Open Source Integration: http://fusesource.com >>>>> Blog: http://davsclaus.blogspot.com/ >>>>> Twitter: http://twitter.com/davsclaus >>>>> >>>> >>>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> Apache Camel Committer >>> >>> Author of Camel in Action: http://www.manning.com/ibsen/ >>> Open Source Integration: http://fusesource.com >>> Blog: http://davsclaus.blogspot.com/ >>> Twitter: http://twitter.com/davsclaus >>> >> >> > > > > -- > Claus Ibsen > Apache Camel Committer > > Author of Camel in Action: http://www.manning.com/ibsen/ > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/ > Twitter: http://twitter.com/davsclaus >
