Hi David Thanks a lot for answering here in this topic.
A few comments to your reply. * SSHD config and the “MaxSession” setting -> when we started troubleshooting we thought as well that this could help to fix our issue. However based on the sshd_config manpage (see below) this setting is correlating only to session multiplexing, which I don’t expect that NiFi does it. That was one of the reasons why I created my mail. When we started using the SFTP Server increased as well the “MaxStartups” to 100 as potentially a lot of PutSFTP processors can try to open a connection to the SFTP server in parallel. MaxSessions Specifies the maximum number of open shell, login or subsystem (e.g. sftp) sessions permitted per network connection. Multiple sessions may be established by clients that support connection multiplexing. Setting MaxSessions to 1 will effectively disable session multiplexing, whereas setting it to 0 will prevent all shell, login and subsystem sessions while still permitting forwarding. The default is 10. * Reason for having 50 PutSFTP instance -> We fetch data from a lot of different data sources, each of them are processed and the original files will be store on the SFTP server. The reason why we created not a single SFTP processor was clarity on the GUI. It would need a lot of local Input/Output ports and would mess up the canvas. But you are right, less SFTP processor would reduce the load of the SFTP server as we would use less total processors to transfer data. However we use a quite powerful Synology NAS (SA3600) as SFTP server and based on the datasheet it should be possible to handle 4000 concurrent sessions (sadly not more). Based on your explanation how NiFi handles SFTP we decided to restart and upgrade our NAS to the newest firmware. In the last few hours the issue happened only two times, but we will monitor it. It’s very likely that the issue is or was related to our SFTP server and not to NiFi. Cheers Josef From: David Handermann <[email protected]> Date: Thursday, 16 February 2023 at 15:51 To: [email protected] <[email protected]> Subject: Re: NiFi 1.20.0 PutSFTP: SSH Client connection failed -> Timeout expired Hi Josef, The FetchSFTP Processor implements some connection reuse, but GetSFTP, ListSFTP, and PutSFTP create a new connection for every invocation of the processor. I have considered a new approach SFTP components using a Controller Service with connection pooling, but it still requires some design work prior to implementation. Based on the current SFTP processor behavior, it is possible to have connection timeouts when the SFTP server is not accepting new connections. Every SFTP server is different, but OpenSSH uses the MaxSessions setting in sshd_config [1] to limit the number of simultaneous open sessions. Using the example of 50 PutSFTP processors connecting to the same server, it is very possible to encounter a connection timeout if NiFi triggers all of them in rapid succession. The Connection Timeout property in PutSFTP controls how long the processor will wait before throwing the ClientConnectException. The number of Concurrent Tasks for each PutSFTP processor also influences the number of simultaneous connections. Increasing the Connection Timeout in PutSFTP may hide the problem, but if the destination SFTP server can handle the load, it may be helpful to increase the number of maximum sessions. On the other hand, is there a reason for having 50 separate instances of PutSFTP sending to the same server? It should be possible to design the flow and parameterize the destination using FlowFile attributes. Depending on the number of CPU cores and available threads, having more SFTP connections results in poor performance, so smaller numbers can be better. Regards, David Handermann [1] https://linux.die.net/man/5/sshd_config<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinux.die.net%2Fman%2F5%2Fsshd_config&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CjGVtb6tqsQsNelhECCV3aNvvFwgZwEYEC9gGiEnWwQ%3D&reserved=0> On Thu, Feb 16, 2023 at 1:33 AM <[email protected]<mailto:[email protected]>> wrote: Hi guys It was upgrade time again on our side, we just upgraded from 1.19.1 to 1.20.0. Since 1.20.0 we do see significantly more SSH Connection Timeout errors on the PutSFTP processor… PutSFTP Processor ERROR: 2023-02-16 07:44:07,905 ERROR [Timer-Driven Process Thread-50] o.a.nifi.processors.standard.PutSFTP PutSFTP[id=12563431-c40a-1af7-b09b-16de27d887b7] Unable to transfer StandardFlowFileRecord[uuid=a1adadb1-61f9-414f-99e5-aad4331165ef,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1676529847583-196507, container=default, section=923], offset=0, length=6582249],offset=0,name=myfile.avro.gz,size=6582249] to remote host nas.local.com<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnas.local.com%2F&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8u5wureyR4zVU%2FUdXb6TqTWpY9V890qTb3a1cTe%2BVt4%3D&reserved=0> due to org.apache.nifi.processors.standard.socket.ClientConnectException: SSH Client connection failed [nas.local.com<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnas.local.com%2F&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8u5wureyR4zVU%2FUdXb6TqTWpY9V890qTb3a1cTe%2BVt4%3D&reserved=0>:22]: net.schmizz.sshj.transport.TransportException: Timeout expired: 30000 MILLISECONDS; routing to failure net.schmizz.sshj.transport.TransportException: Timeout expired: 30000 MILLISECONDS I know that David Handermann implemented a fix (https://issues.apache.org/jira/browse/NIFI-9989<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9989&data=05%7C01%7CJosef.Zahner1%40swisscom.com%7Cca0dea0c3781417181e008db102d598a%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C638121559173781274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lojp%2BvP%2BoGq9DVzZnFsugKh0FdYgdP4prYmtsXsoAZI%3D&reserved=0>) for SSHJ, but I don’t know if it really is related. May be it’s just a configuration issue (number of allowed concurrent connections?) on the SFTP Server side. Let’s make an example, let’s say we do have 50 PutSFTP processors to the same destination and all of them are getting data in an interval of 15min. Does NiFi keep the SSH connection established for this 50 processors or will it be closed after each flow has been transferred? If it isn’t closed after each flow, how can we influence the timeout? I see only the two timeouts below, which let me assume that it’s closed after each flow… But may be one of you guys can bring some light into the dark. [Background pattern Description automatically generated with low confidence] Cheers Josef
smime.p7s
Description: S/MIME Cryptographic Signature
