Thanks again. When you refer to live queue listing and data viewing what
are you referring to? The dashboard or something else.
Jim K.
On Thu, Apr 6, 2017 at 4:49 PM Joe Witt <joe.w...@gmail.com> wrote:
> No problem. Remember you can use live queue listing and data viewing
> to see all the attributes we know about the object at each stage.
> That is exactly how I figured out how to wire this together and what I
> needed from each step.
>
> Thanks
> Joe
>
> On Thu, Apr 6, 2017 at 4:43 PM, James Keeney <nextves...@gmail.com> wrote:
> > Thank you. That was the final detail I was not getting. The use of the
> > ${path} expression variable. I now see that I needed to look at the
> writes
> > attributes section of the ListFile processor.
> >
> > Jim K.
> >
> > On Thu, Apr 6, 2017 at 2:30 PM Joe Witt <joe.w...@gmail.com> wrote:
> >>
> >> Jim,
> >>
> >> Yep I understand your question and how to support that is what I was
> >> trying to convey.
> >>
> >> ListFile should pull from "/home/source". Lets say it finds that
> >> 'home/source/test/newfile.txt file.
> >>
> >> The resulting flowfile will have an attribute called 'path' that says
> >> 'test'
> >>
> >> Then you use FetchFile to actually pull in the bytes. 'path' still says
> >> 'test'
> >>
> >> Then you use PutSFTP with the 'Remote Path' set to "/www/files/${path}"
> >>
> >> I have just verified that this works myself locally using
> >> List/Fetch/PutFile. In your case you'd use PutSFTP.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, Apr 6, 2017 at 12:51 PM, James Keeney <nextves...@gmail.com>
> >> wrote:
> >> > Thanks for getting back to me. I will follow up on the documentation
> >> > Pull
> >> > Request.
> >> >
> >> > As to the directory question, I wasn't specific enough. I've already
> >> > configured the setting you described.
> >> >
> >> > Here is what is going on:
> >> >
> >> > Say the source directory is /home/source and the destination is
> >> > /www/files
> >> >
> >> > This works:
> >> >
> >> > If a user drops the file text.txt into /home/source then I want that
> to
> >> > be
> >> > /www/files/text.txt That is working as expected.
> >> >
> >> > This does not work
> >> >
> >> > If a user creates a subdirectory and drop a file, so
> >> > /home/source/test/newfile.txt then I want the destination to reflect
> the
> >> > subdirectory as in /www/files/test/newfile.txt But what happens is the
> >> > file
> >> > is being placed into the destination directory without the new
> >> > subdirectory.
> >> > So what is getting created is /www/files/newfile.txt and not
> >> > /www/files/test/newfile.txt
> >> >
> >> > Any suggestions?
> >> >
> >> > Jim K.
> >> >
> >> >
> >> > On Thu, Apr 6, 2017 at 12:19 PM Joe Witt <joe.w...@gmail.com> wrote:
> >> >>
> >> >> Jim,
> >> >>
> >> >> Glad you've made progress on the SFTP side. Please file a JIRA with
> >> >> your suggestions for the docs and the ideal case then is you'd file a
> >> >> Pull Request
> >> >> (https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide)
> >> >> which actually provides the suggested documentation changes.
> >> >>
> >> >> For the ListFile/FetchFile -> PutSFTP[1] side the key property on
> >> >> PutSFTP to set is 'Remote Path'. You'll want this value to have the
> >> >> base directory you need to write to which could be './' or could be
> >> >> 'some/place/to/write/to' and you'll also want it to reflect the
> >> >> directory structure from which you fetched the file locally. This
> >> >> will be available to you from the 'path' attribute of the flowfile.
> >> >> This is set by the ListFile processor (see writes attributes) [2].
> >> >>
> >> >> So putting these together you want your PutSFTP processor to have as
> a
> >> >> value for 'Remote Path' something like "thebasedir/fordata/${path}".
> >> >>
> >> >>
> >> >> [1]
> >> >>
> >> >>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.PutSFTP/index.html
> >> >> [2]
> >> >>
> >> >>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index.html
> >> >>
> >> >> On Thu, Apr 6, 2017 at 11:43 AM, James Keeney <nextves...@gmail.com>
> >> >> wrote:
> >> >> > Joe and Juan -
> >> >> >
> >> >> > Thank you very much for the help. It turned out to be the prompt
> for
> >> >> > verifying the authenticity of the host.
> >> >> >
> >> >> > With that fixed, I have a new question:
> >> >> >
> >> >> > I'm using ListFile and FetchFile to identify new files as they are
> >> >> > added
> >> >> > to
> >> >> > a directory. When they are I am using SFTP to transfer to another
> >> >> > server
> >> >> > behind the firewall. I'd like to be able to preserve the directory
> >> >> > structure
> >> >> > when I transfer the files. I set Create Directory to true but the
> >> >> > SFTP
> >> >> > transfer is always putting the files in the root.
> >> >> >
> >> >> > Any ideas?
> >> >> >
> >> >> > Here is how I resolve the first issue:
> >> >> >
> >> >> > I added the host keys to the known_hosts file and that did the
> trick.
> >> >> >
> >> >> > From a documentation perspective I'd suggest adding a little more
> >> >> > guidance
> >> >> > for people who are not familiar with SFTP. It was not clear to me
> >> >> > what
> >> >> > the
> >> >> > two parameters are:
> >> >> >
> >> >> > Host Key File
> >> >> > Private Key Path
> >> >> >
> >> >> > Since I didn't understand the use of the known_hosts file (using
> >> >> > key-scan to
> >> >> > get their keys and adding those to the file I had no idea what to
> put
> >> >> > in
> >> >> > Host Key File. Also, private key path confused me since it is the
> >> >> > public
> >> >> > key
> >> >> > that is being shared. Also, we might want to highlight that the
> SFTP
> >> >> > process
> >> >> > needs to go forward without prompts. I had gotten use to the prompt
> >> >> > so
> >> >> > when
> >> >> > I tested I didn't even think about that issue.
> >> >> >
> >> >> > How might I go about adding my sweat equity to update the
> >> >> > documentation
> >> >> > to
> >> >> > make it a little clearer? Let me know and I will take a crack at
> >> >> > expandign
> >> >> > the information to help other users.
> >> >> >
> >> >> > Thanks.
> >> >> >
> >> >> > Jim K.
> >> >> >
> >> >> > On Tue, Apr 4, 2017 at 12:31 PM Joe Witt <joe.w...@gmail.com>
> wrote:
> >> >> >>
> >> >> >> definitely agree with Juan's suggestion to get more details on
> what
> >> >> >> the actual authentication process is when trying ssh -vvv. Also,
> be
> >> >> >> sure to check what order of authorization occurs. It is possible
> it
> >> >> >> is trying keyboard-interactive before the certs and this could
> >> >> >> create
> >> >> >> problems so ordering there, on the server side, will really
> matter.
> >> >> >>
> >> >> >> On Tue, Apr 4, 2017 at 12:27 PM, Juan Sequeiros
> >> >> >> <helloj...@gmail.com>
> >> >> >> wrote:
> >> >> >> > Good afternoon,
> >> >> >> >
> >> >> >> > I would try this command from command line:
> >> >> >> >
> >> >> >> > ssh -vvv -i <path to rsa file> user@server
> >> >> >> >
> >> >> >> > Example:
> >> >> >> >
> >> >> >> > ssh -vvv -i /some/path/.ssh/id_rsa nifi@10.10.10.10
> >> >> >> >
> >> >> >> > If that works then I would double check the "private key path"
> >> >> >> > property
> >> >> >> > of
> >> >> >> > your GetSFTP it should point to the fully qualified file to the
> >> >> >> > private
> >> >> >> > key
> >> >> >> > path.
> >> >> >> >
> >> >> >> > If it does not work then the -vvv option should give you more
> >> >> >> > error.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Tue, Apr 4, 2017 at 9:47 AM James Keeney <
> nextves...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> I am using SFTP to transfer files between two servers. I have
> >> >> >> >> tried
> >> >> >> >> multiple configurations to try to get the authentication to
> work
> >> >> >> >> but
> >> >> >> >> i
> >> >> >> >> keep
> >> >> >> >> getting the Auth Fail error. I'm able to go onto the Nifi
> server
> >> >> >> >> sftp
> >> >> >> >> over
> >> >> >> >> to the destination server but I cannot get it to work in Nifi.
> >> >> >> >> I'm
> >> >> >> >> just
> >> >> >> >> not
> >> >> >> >> sure how to debug this so I was hoping someone could help. Here
> >> >> >> >> are
> >> >> >> >> the
> >> >> >> >> setting I'm using (I've replaced all the important details with
> >> >> >> >> placeholders):
> >> >> >> >>
> >> >> >> >> Hostname: <Hostname. I've tried both IP and hostname without
> >> >> >> >> luck>
> >> >> >> >> Port: 22 <I've confirmed this by connecting directly from
> server
> >> >> >> >> to
> >> >> >> >> server)
> >> >> >> >> username: <username>
> >> >> >> >> Host key File: <location on the server of the public key file>
> >> >> >> >>
> >> >> >> >> The error I am receiving is below. Any help would be greatly
> >> >> >> >> appreciated.
> >> >> >> >>
> >> >> >> >> Thanks.
> >> >> >> >>
> >> >> >> >> Jim K
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> 2017-03-29 13:59:51,705 ERROR [Timer-Driven Process Thread-1]
> >> >> >> >> o.a.nifi.processors.standard.PutSFTP
> >> >> >> >> PutSFTP[id=fcdd2eb4-015a-1000-80c5-7406e6fca4c5] Unable to
> >> >> >> >> transfer
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> StandardFlowFileRecord[uuid=a3569afa-7c80-4cec-9239-2424172e30d1,claim=StandardContentClaim
> >> >> >> >> [resourceClaim=StandardResourceClaim[id=1490795991637-151,
> >> >> >> >> container=default, section=151], offset=0,
> >> >> >> >>
> length=2917388],offset=0,name=tulips_248_042214.jpg,size=2917388]
> >> >> >> >> to
> >> >> >> >> remote
> >> >> >> >> host <servername> due to
> >> >> >> >> org.apache.nifi.processor.exception.ProcessException:
> IOException
> >> >> >> >> thrown
> >> >> >> >> from PutSFTP[id=fcdd2eb4-015a-1000-80c5-7406e6fca4c5]:
> >> >> >> >> java.io.IOException:
> >> >> >> >> Failed to obtain connection to remote host due to
> >> >> >> >> com.jcraft.jsch.JSchException: Auth fail: java.io.IOException:
> >> >> >> >> Failed
> >> >> >> >> to
> >> >> >> >> obtain connection to remote host due to
> >> >> >> >> com.jcraft.jsch.JSchException:
> >> >> >> >> Auth
> >> >> >> >> fail; routing to failure: java.io.IOException: Failed to obtain
> >> >> >> >> connection
> >> >> >> >> to remote host due to com.jcraft.jsch.JSchException: Auth fail
> >> >> >> >> 2017-03-29 13:59:51,706 ERROR [Timer-Driven Process Thread-1]
> >> >> >> >> o.a.nifi.processors.standard.PutSFTP
> >> >> >> >> java.io.IOException: Failed to obtain connection to remote host
> >> >> >> >> due
> >> >> >> >> to
> >> >> >> >> com.jcraft.jsch.JSchException: Auth fail
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:447)
> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:529)
> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.processors.standard.PutFileTransfer$1.process(PutFileTransfer.java:135)
> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082)
> >> >> >> >> ~[na:na]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053)
> >> >> >> >> ~[na:na]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.processors.standard.PutFileTransfer.onTrigger(PutFileTransfer.java:127)
> >> >> >> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> >> >> >> >> [nifi-api-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> >> >> >> >> [nifi-framework-core-1.1.0.jar:1.1.0]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> at
> >> >> >> >>
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> >> >> >> [na:1.8.0_121]
> >> >> >> >> at
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> >> >> >> [na:1.8.0_121]
> >> >> >
> >> >> > --
> >> >> > Jim Keeney
> >> >> > President, FitterWeb
> >> >> > E: j...@fitterweb.com
> >> >> > M: 703-568-5887 <(703)%20568-5887>
> >> >> >
> >> >> > FitterWeb Consulting
> >> >> > Are you lean and agile enough?
> >> >
> >> > --
> >> > Jim Keeney
> >> > President, FitterWeb
> >> > E: j...@fitterweb.com
> >> > M: 703-568-5887 <(703)%20568-5887>
> >> >
> >> > FitterWeb Consulting
> >> > Are you lean and agile enough?
> >
> > --
> > Jim Keeney
> > President, FitterWeb
> > E: j...@fitterweb.com
> > M: 703-568-5887 <(703)%20568-5887>
> >
> > FitterWeb Consulting
> > Are you lean and agile enough?
>
--
Jim Keeney
President, FitterWeb
E: j...@fitterweb.com
M: 703-568-5887
*FitterWeb Consulting*
*Are you lean and agile enough? *
