Jordan Some of the references to Gbps are in reference to network terms where rates are truly measured in bits and on a base of 1000 where as some reference Gbps referring to actual throughput/sizes and usually those are in Bytes and on a base of 1024. I'm making some assumptions on what is meant from the writeup.
Rate/Capabilities: - NIC on the machine NiFi runs on: 10Gbps = 1.22GB/sec theoretical max. Factor in some overhead and such and good to assume around 1GB/sec is safe with 1.2GB being feasible in some cases (like you mentioned). - 100 GB files then in theory should take close to 2 mins to transfer assuming the NIC and network is otherwise available. We'd need to know more about the type of disks involved. They're network mapped which is important but sounds like the network is pretty solid design. The type of disks then do matter at least in terms of the rate they're meant to sustain. As you noted though you can fix some variables and achieve 1+GB/sec copies so likely these disks are quite fast. Prob would need to better understand the setup such as where nifi is, vs where the storage is, vs where the data comes from vs where it goes to. Just to consider the various factors at play there. All that said... "usually we see transfer speeds of more than 1gbps and files move over > pretty quickly. In the last two months....5MB/sec" Nothing in NiFi changed but the rates went from around 1GB to 5MB/sec? Is there any sign of packet loss between the systems? That can certainly do it. Ways to check this vary but an old school technique is to send various sized ICMP messages and just monitor their health. In a LAN setup anything less than 100% would be interesting. The *most common* culprit for sudden performance changes like this though have to do with virus/malware scanning software being put on machines like this. It is certainly a good idea but the performance penalty for some of these tools is severe. They engage as every bit of data is read/written in many cases and do their analysis. These are often turned on by others sometimes without the admins of a given system even knowing. Any chance that can be a factor here? Thanks On Tue, Sep 5, 2023 at 3:17 AM POMEROY, Jordan 7094 <jordan.pome...@herts.police.uk.invalid> wrote: > Hello > > Wonder if you can help me with an issue that is being reported to us via > our users. > > We use NiFi to transfer evidential phone data across our network and place > it on to our networked file storage server. Most files are in excess of > 100gb, and we have allocated the server for this work a 10gb NIC. Although > we do not expect the transfers to reach these speeds, usually we see > transfer speeds of more than 1gbps and files move over pretty quickly. > > In the last two months, our users have reported to us that it is taking > significantly longer for files to reach the destination. Upon looking, the > transfer speeds have dropped to roughly 5mbps. > > I have asked our Network team to look at firewall rules to see if there is > a bottleneck on the Network anywhere, but there isn't. we've also migrated > the virtual server which NiFi is configured on and changed network switch > ports etc to see if this has made any difference, but nothing. > > To test the network even further, I mapped from the NiFi virtual server to > the file storage server and transferred over a large file to test speeds. > This moved over at 1.2gbps with no issues at all. > > The "get file" and "put file" commands have not been touched since they > were originally configured, and we are currently using NiFi version 1.5.0. > The server is running OS Windows Server 2022. > > Any help you can provide as to what we can check next to get this working > would be very much appreciated. > > Thanks, > Jordan. > > Jordan Pomeroy > Server and Storage Engineer > Infrastructure team > Herts, Beds and Cambs Police ICT Department > The best way to contact me is via Teams or Email as I have no fixed desk. > jordan.pome...@herts.pnn.police.uk<mailto: > jordan.pome...@herts.pnn.police.uk> > > ________________________________ > Internet e-mail is not to be treated as a secure means of communication. > Hertfordshire Constabulary monitors all internet e-mail activity and > content. This communication is confidential and intended for the > addressee(s) only. Please notify the sender if you have received this in > error. Unauthorised use or disclosure of the contents may be unlawful. > Opinions expressed in this document may not be official policy. > > For more details please see Hertfordshire Constabulary Privacy Policy < > https://www.herts.police.uk/hyg/fpnbch/privacy-notice/> >
