Any thought Joe? thank so much
On Tue, Oct 30, 2018 at 2:28 PM Alfredo De Luca <alfredo.del...@gmail.com>
wrote:
> I have attached the nifi properties. Also these are more info ...
>
> - ListSftp is running on primary node of the cluster and with 1 concurrent
> tasks
> - nifi is in cluster with 3 nodes (pod in kubernetes)
> ---
> *Loading of nifi*
> how many threads does nifi show are running?
> *150 in average*
> ---
> How often does the problem happen? How often does the connection/flow
> work?
>
> *Often, some processors does not list any file on sftp server because of
> this problem. Other Processors could list file on server sftp but at
> certain time listing fails.*
> *In addition sometimes fails also the unique fetchSftp processor*
>
> SFTP Server
> 1 CPU
> 2Gb Ram
> Load avg. load average: 0.20, 0.27, 0.31
>
> Cheers
>
>
>
> On Mon, Oct 29, 2018 at 5:41 PM Alfredo De Luca <alfredo.del...@gmail.com>
> wrote:
>
>> Thanks Joe. Appreciated. I will get back to you asap with the info.
>>
>> Cheers
>>
>>
>> On Mon, Oct 29, 2018 at 5:26 PM Joe Witt <joe.w...@gmail.com> wrote:
>>
>>> Alfredo
>>>
>>> There are quite a few details that will be needed to help folks guess
>>> at possible issues.
>>>
>>> Configuration of the machine running nifi
>>> how many cores?, etc..
>>>
>>> Configuration of nifi
>>> you shared common properties from nifi.properties which is good.
>>> how many threads does the flow controller have?
>>> how many processors are on the graph?
>>> How many threads does ListSFTP have?
>>> Is this a NiFi cluster or single node?
>>>
>>> Loading of the system running nifi
>>> cpu load avg
>>> garbage collection behavior in the jVM?
>>>
>>> Loading of nifi
>>> how many threads does nifi show are running?
>>>
>>> Similar info for the SFTP server side
>>> System error 4 could mean a lot of things. Is that system taxed and
>>> unable to establish connections at times?
>>>
>>> How often does the problem happen? How often does the connection/flow
>>> work?
>>>
>>> Have you tried manually connecting the way nifi would by running
>>> commands like 'sftp u...@host.sftp.server -vvv' repeatedly to see if
>>> you can reproduce the issue?
>>>
>>> Does a file system get mounted for the sftp user and periodically is
>>> this failing?
>>>
>>> Unfortunately it could be a lot of things so we'll probably need to
>>> unpack a lot of info to get there.
>>>
>>> Thanks
>>> On Mon, Oct 29, 2018 at 12:14 PM Alfredo De Luca
>>> <alfredo.del...@gmail.com> wrote:
>>> >
>>> > hi all. any idea? I am trying everything..but not sure what cause this
>>> error.
>>> >
>>> > Cheers
>>> >
>>> > On Mon, Oct 29, 2018 at 12:34 PM Alfredo De Luca <
>>> alfredo.del...@gmail.com> wrote:
>>> >>
>>> >> nifi.swap.in.threads=1
>>> >> nifi.swap.out.threads=4
>>> >> nifi.cluster.node.protocol.threads=25
>>> >> nifi.cluster.node.protocol.max.threads=50
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Oct 29, 2018 at 12:28 PM Alfredo De Luca <
>>> alfredo.del...@gmail.com> wrote:
>>> >>>
>>> >>> anyway...this is the error
>>> >>> 2018-10-29 11:21:06,768 ERROR [Timer-Driven Process Thread-13]
>>> SimpleProcessLogger.java:254
>>> ListSFTP[id=766ac418-27ce-335a-97c9-8823f2cf5a96] Failed to perform listing
>>> on remote host due to java.io.IOException: Failed to obtain connection to
>>> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
>>> >>>
>>> >>>
>>> >>> On Mon, Oct 29, 2018 at 12:27 PM Alfredo De Luca <
>>> alfredo.del...@gmail.com> wrote:
>>> >>>>
>>> >>>> Hi Joe.
>>> >>>> Sftp server allows 200 connections... not sure about the nifi. I ll
>>> check
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Oct 29, 2018 at 12:21 PM Joe Witt <joe.w...@gmail.com>
>>> wrote:
>>> >>>>>
>>> >>>>> how many connections does your sftp server allow at once and how
>>> many threads is nifi given for communication?
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Mon, Oct 29, 2018, 7:10 AM Alfredo De Luca <
>>> alfredo.del...@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Hi all. We have some issue with our nifi (v. 1.6) where
>>> occasionally we have an error saying ...
>>> >>>>>> Auth fail.
>>> >>>>>>
>>> >>>>>> On the sftp server side we get
>>> >>>>>>
>>> >>>>>> Oct 29 11:48:06 sftp sshd[13845]: pam_sss(sshd:account): Access
>>> denied for user <sftp user>: 4 (System error)
>>> >>>>>>
>>> >>>>>> Any idea?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Alfredo
>>> >>>>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Alfredo
>>> >>>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Alfredo
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> Alfredo
>>> >>
>>> >
>>> >
>>> > --
>>> > Alfredo
>>> >
>>>
>>
>>
>> --
>> *Alfredo*
>>
>>
>
> --
> *Alfredo*
>
>
--
*Alfredo*
