Joe -
Still working with the SFTP chain. We have been testing and have
encountered a different problem. During some transfers but not all, when
doing a transfer of a large number of files (>10), some of the files are
not transferring.
I'm getting no errors at all.
Here is the chain I'm using:
ListFile --> FetchFile --> PutSFTP
I used putfile processors for the failure and reject and terminating on
success. No files are ending up in the directories I set aside for Failure
and Reject.
Any suggestions on how best to debug this?
Thanks.
On Thu, Apr 6, 2017 at 5:24 PM Joe Witt <joe.w...@gmail.com> wrote:
> I just mean while designing/interacting with the flow you can
> start/stop processors, you can click on the connections and say 'list
> queue' and then you can click on each object in the queue and see its
> attributes and content. Really helps step through the flow at each
> step.
>
> On Thu, Apr 6, 2017 at 5:08 PM, James Keeney <nextves...@gmail.com> wrote:
> > Thanks again. When you refer to live queue listing and data viewing what
> are
> > you referring to? The dashboard or something else.
> >
> > Jim K.
> >
> > On Thu, Apr 6, 2017 at 4:49 PM Joe Witt <joe.w...@gmail.com> wrote:
> >>
> >> No problem. Remember you can use live queue listing and data viewing
> >> to see all the attributes we know about the object at each stage.
> >> That is exactly how I figured out how to wire this together and what I
> >> needed from each step.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, Apr 6, 2017 at 4:43 PM, James Keeney <nextves...@gmail.com>
> wrote:
> >> > Thank you. That was the final detail I was not getting. The use of the
> >> > ${path} expression variable. I now see that I needed to look at the
> >> > writes
> >> > attributes section of the ListFile processor.
> >> >
> >> > Jim K.
> >> >
> >> > On Thu, Apr 6, 2017 at 2:30 PM Joe Witt <joe.w...@gmail.com> wrote:
> >> >>
> >> >> Jim,
> >> >>
> >> >> Yep I understand your question and how to support that is what I was
> >> >> trying to convey.
> >> >>
> >> >> ListFile should pull from "/home/source". Lets say it finds that
> >> >> 'home/source/test/newfile.txt file.
> >> >>
> >> >> The resulting flowfile will have an attribute called 'path' that says
> >> >> 'test'
> >> >>
> >> >> Then you use FetchFile to actually pull in the bytes. 'path' still
> >> >> says
> >> >> 'test'
> >> >>
> >> >> Then you use PutSFTP with the 'Remote Path' set to
> "/www/files/${path}"
> >> >>
> >> >> I have just verified that this works myself locally using
> >> >> List/Fetch/PutFile. In your case you'd use PutSFTP.
> >> >>
> >> >> Thanks
> >> >> Joe
> >> >>
> >> >> On Thu, Apr 6, 2017 at 12:51 PM, James Keeney <nextves...@gmail.com>
> >> >> wrote:
> >> >> > Thanks for getting back to me. I will follow up on the
> documentation
> >> >> > Pull
> >> >> > Request.
> >> >> >
> >> >> > As to the directory question, I wasn't specific enough. I've
> already
> >> >> > configured the setting you described.
> >> >> >
> >> >> > Here is what is going on:
> >> >> >
> >> >> > Say the source directory is /home/source and the destination is
> >> >> > /www/files
> >> >> >
> >> >> > This works:
> >> >> >
> >> >> > If a user drops the file text.txt into /home/source then I want
> that
> >> >> > to
> >> >> > be
> >> >> > /www/files/text.txt That is working as expected.
> >> >> >
> >> >> > This does not work
> >> >> >
> >> >> > If a user creates a subdirectory and drop a file, so
> >> >> > /home/source/test/newfile.txt then I want the destination to
> reflect
> >> >> > the
> >> >> > subdirectory as in /www/files/test/newfile.txt But what happens is
> >> >> > the
> >> >> > file
> >> >> > is being placed into the destination directory without the new
> >> >> > subdirectory.
> >> >> > So what is getting created is /www/files/newfile.txt and not
> >> >> > /www/files/test/newfile.txt
> >> >> >
> >> >> > Any suggestions?
> >> >> >
> >> >> > Jim K.
> >> >> >
> >> >> >
> >> >> > On Thu, Apr 6, 2017 at 12:19 PM Joe Witt <joe.w...@gmail.com>
> wrote:
> >> >> >>
> >> >> >> Jim,
> >> >> >>
> >> >> >> Glad you've made progress on the SFTP side. Please file a JIRA
> with
> >> >> >> your suggestions for the docs and the ideal case then is you'd
> file
> >> >> >> a
> >> >> >> Pull Request
> >> >> >> (
> https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide)
> >> >> >> which actually provides the suggested documentation changes.
> >> >> >>
> >> >> >> For the ListFile/FetchFile -> PutSFTP[1] side the key property on
> >> >> >> PutSFTP to set is 'Remote Path'. You'll want this value to have
> the
> >> >> >> base directory you need to write to which could be './' or could
> be
> >> >> >> 'some/place/to/write/to' and you'll also want it to reflect the
> >> >> >> directory structure from which you fetched the file locally. This
> >> >> >> will be available to you from the 'path' attribute of the
> flowfile.
> >> >> >> This is set by the ListFile processor (see writes attributes) [2].
> >> >> >>
> >> >> >> So putting these together you want your PutSFTP processor to have
> as
> >> >> >> a
> >> >> >> value for 'Remote Path' something like
> "thebasedir/fordata/${path}".
> >> >> >>
> >> >> >>
> >> >> >> [1]
> >> >> >>
> >> >> >>
> >> >> >>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.PutSFTP/index.html
> >> >> >> [2]
> >> >> >>
> >> >> >>
> >> >> >>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index.html
> >> >> >>
> >> >> >> On Thu, Apr 6, 2017 at 11:43 AM, James Keeney <
> nextves...@gmail.com>
> >> >> >> wrote:
> >> >> >> > Joe and Juan -
> >> >> >> >
> >> >> >> > Thank you very much for the help. It turned out to be the prompt
> >> >> >> > for
> >> >> >> > verifying the authenticity of the host.
> >> >> >> >
> >> >> >> > With that fixed, I have a new question:
> >> >> >> >
> >> >> >> > I'm using ListFile and FetchFile to identify new files as they
> are
> >> >> >> > added
> >> >> >> > to
> >> >> >> > a directory. When they are I am using SFTP to transfer to
> another
> >> >> >> > server
> >> >> >> > behind the firewall. I'd like to be able to preserve the
> directory
> >> >> >> > structure
> >> >> >> > when I transfer the files. I set Create Directory to true but
> the
> >> >> >> > SFTP
> >> >> >> > transfer is always putting the files in the root.
> >> >> >> >
> >> >> >> > Any ideas?
> >> >> >> >
> >> >> >> > Here is how I resolve the first issue:
> >> >> >> >
> >> >> >> > I added the host keys to the known_hosts file and that did the
> >> >> >> > trick.
> >> >> >> >
> >> >> >> > From a documentation perspective I'd suggest adding a little
> more
> >> >> >> > guidance
> >> >> >> > for people who are not familiar with SFTP. It was not clear to
> me
> >> >> >> > what
> >> >> >> > the
> >> >> >> > two parameters are:
> >> >> >> >
> >> >> >> > Host Key File
> >> >> >> > Private Key Path
> >> >> >> >
> >> >> >> > Since I didn't understand the use of the known_hosts file (using
> >> >> >> > key-scan to
> >> >> >> > get their keys and adding those to the file I had no idea what
> to
> >> >> >> > put
> >> >> >> > in
> >> >> >> > Host Key File. Also, private key path confused me since it is
> the
> >> >> >> > public
> >> >> >> > key
> >> >> >> > that is being shared. Also, we might want to highlight that the
> >> >> >> > SFTP
> >> >> >> > process
> >> >> >> > needs to go forward without prompts. I had gotten use to the
> >> >> >> > prompt
> >> >> >> > so
> >> >> >> > when
> >> >> >> > I tested I didn't even think about that issue.
> >> >> >> >
> >> >> >> > How might I go about adding my sweat equity to update the
> >> >> >> > documentation
> >> >> >> > to
> >> >> >> > make it a little clearer? Let me know and I will take a crack at
> >> >> >> > expandign
> >> >> >> > the information to help other users.
> >> >> >> >
> >> >> >> > Thanks.
> >> >> >> >
> >> >> >> > Jim K.
> >> >> >> >
> >> >> >> > On Tue, Apr 4, 2017 at 12:31 PM Joe Witt <joe.w...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> definitely agree with Juan's suggestion to get more details on
> >> >> >> >> what
> >> >> >> >> the actual authentication process is when trying ssh -vvv.
> Also,
> >> >> >> >> be
> >> >> >> >> sure to check what order of authorization occurs. It is
> possible
> >> >> >> >> it
> >> >> >> >> is trying keyboard-interactive before the certs and this could
> >> >> >> >> create
> >> >> >> >> problems so ordering there, on the server side, will really
> >> >> >> >> matter.
> >> >> >> >>
> >> >> >> >> On Tue, Apr 4, 2017 at 12:27 PM, Juan Sequeiros
> >> >> >> >> <helloj...@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> > Good afternoon,
> >> >> >> >> >
> >> >> >> >> > I would try this command from command line:
> >> >> >> >> >
> >> >> >> >> > ssh -vvv -i <path to rsa file> user@server
> >> >> >> >> >
> >> >> >> >> > Example:
> >> >> >> >> >
> >> >> >> >> > ssh -vvv -i /some/path/.ssh/id_rsa nifi@10.10.10.10
> >> >> >> >> >
> >> >> >> >> > If that works then I would double check the "private key
> path"
> >> >> >> >> > property
> >> >> >> >> > of
> >> >> >> >> > your GetSFTP it should point to the fully qualified file to
> the
> >> >> >> >> > private
> >> >> >> >> > key
> >> >> >> >> > path.
> >> >> >> >> >
> >> >> >> >> > If it does not work then the -vvv option should give you more
> >> >> >> >> > error.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Tue, Apr 4, 2017 at 9:47 AM James Keeney
> >> >> >> >> > <nextves...@gmail.com>
> >> >> >> >> > wrote:
> >> >> >> >> >>
> >> >> >> >> >> I am using SFTP to transfer files between two servers. I
> have
> >> >> >> >> >> tried
> >> >> >> >> >> multiple configurations to try to get the authentication to
> >> >> >> >> >> work
> >> >> >> >> >> but
> >> >> >> >> >> i
> >> >> >> >> >> keep
> >> >> >> >> >> getting the Auth Fail error. I'm able to go onto the Nifi
> >> >> >> >> >> server
> >> >> >> >> >> sftp
> >> >> >> >> >> over
> >> >> >> >> >> to the destination server but I cannot get it to work in
> Nifi.
> >> >> >> >> >> I'm
> >> >> >> >> >> just
> >> >> >> >> >> not
> >> >> >> >> >> sure how to debug this so I was hoping someone could help.
> >> >> >> >> >> Here
> >> >> >> >> >> are
> >> >> >> >> >> the
> >> >> >> >> >> setting I'm using (I've replaced all the important details
> >> >> >> >> >> with
> >> >> >> >> >> placeholders):
> >> >> >> >> >>
> >> >> >> >> >> Hostname: <Hostname. I've tried both IP and hostname without
> >> >> >> >> >> luck>
> >> >> >> >> >> Port: 22 <I've confirmed this by connecting directly from
> >> >> >> >> >> server
> >> >> >> >> >> to
> >> >> >> >> >> server)
> >> >> >> >> >> username: <username>
> >> >> >> >> >> Host key File: <location on the server of the public key
> file>
> >> >> >> >> >>
> >> >> >> >> >> The error I am receiving is below. Any help would be greatly
> >> >> >> >> >> appreciated.
> >> >> >> >> >>
> >> >> >> >> >> Thanks.
> >> >> >> >> >>
> >> >> >> >> >> Jim K
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> 2017-03-29 13:59:51,705 ERROR [Timer-Driven Process
> Thread-1]
> >> >> >> >> >> o.a.nifi.processors.standard.PutSFTP
> >> >> >> >> >> PutSFTP[id=fcdd2eb4-015a-1000-80c5-7406e6fca4c5] Unable to
> >> >> >> >> >> transfer
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> StandardFlowFileRecord[uuid=a3569afa-7c80-4cec-9239-2424172e30d1,claim=StandardContentClaim
> >> >> >> >> >> [resourceClaim=StandardResourceClaim[id=1490795991637-151,
> >> >> >> >> >> container=default, section=151], offset=0,
> >> >> >> >> >>
> >> >> >> >> >>
> length=2917388],offset=0,name=tulips_248_042214.jpg,size=2917388]
> >> >> >> >> >> to
> >> >> >> >> >> remote
> >> >> >> >> >> host <servername> due to
> >> >> >> >> >> org.apache.nifi.processor.exception.ProcessException:
> >> >> >> >> >> IOException
> >> >> >> >> >> thrown
> >> >> >> >> >> from PutSFTP[id=fcdd2eb4-015a-1000-80c5-7406e6fca4c5]:
> >> >> >> >> >> java.io.IOException:
> >> >> >> >> >> Failed to obtain connection to remote host due to
> >> >> >> >> >> com.jcraft.jsch.JSchException: Auth fail:
> java.io.IOException:
> >> >> >> >> >> Failed
> >> >> >> >> >> to
> >> >> >> >> >> obtain connection to remote host due to
> >> >> >> >> >> com.jcraft.jsch.JSchException:
> >> >> >> >> >> Auth
> >> >> >> >> >> fail; routing to failure: java.io.IOException: Failed to
> >> >> >> >> >> obtain
> >> >> >> >> >> connection
> >> >> >> >> >> to remote host due to com.jcraft.jsch.JSchException: Auth
> fail
> >> >> >> >> >> 2017-03-29 13:59:51,706 ERROR [Timer-Driven Process
> Thread-1]
> >> >> >> >> >> o.a.nifi.processors.standard.PutSFTP
> >> >> >> >> >> java.io.IOException: Failed to obtain connection to remote
> >> >> >> >> >> host
> >> >> >> >> >> due
> >> >> >> >> >> to
> >> >> >> >> >> com.jcraft.jsch.JSchException: Auth fail
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:447)
> >> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:529)
> >> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.processors.standard.PutFileTransfer$1.process(PutFileTransfer.java:135)
> >> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082)
> >> >> >> >> >> ~[na:na]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053)
> >> >> >> >> >> ~[na:na]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.processors.standard.PutFileTransfer.onTrigger(PutFileTransfer.java:127)
> >> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> >> >> >> >> >> [nifi-api-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> >> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> >> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> >> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> >> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> >> at
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> >> >> >> >> [na:1.8.0_121]
> >> >> >> >
> >> >> >> > --
> >> >> >> > Jim Keeney
> >> >> >> > President, FitterWeb
> >> >> >> > E: j...@fitterweb.com
> >> >> >> > M: 703-568-5887 <(703)%20568-5887>
> >> >> >> >
> >> >> >> > FitterWeb Consulting
> >> >> >> > Are you lean and agile enough?
> >> >> >
> >> >> > --
> >> >> > Jim Keeney
> >> >> > President, FitterWeb
> >> >> > E: j...@fitterweb.com
> >> >> > M: 703-568-5887 <(703)%20568-5887>
> >> >> >
> >> >> > FitterWeb Consulting
> >> >> > Are you lean and agile enough?
> >> >
> >> > --
> >> > Jim Keeney
> >> > President, FitterWeb
> >> > E: j...@fitterweb.com
> >> > M: 703-568-5887 <(703)%20568-5887>
> >> >
> >> > FitterWeb Consulting
> >> > Are you lean and agile enough?
> >
> > --
> > Jim Keeney
> > President, FitterWeb
> > E: j...@fitterweb.com
> > M: 703-568-5887 <(703)%20568-5887>
> >
> > FitterWeb Consulting
> > Are you lean and agile enough?
>
--
Jim Keeney
President, FitterWeb
E: j...@fitterweb.com
M: 703-568-5887
*FitterWeb Consulting*
*Are you lean and agile enough? *
