As a first step, I decided to create a ram disk and move the ts_export folders in /var/lib/ntopng/[-1 - 24]/ to the ram disk. I figured this seemed relatively safe as the data in this folder seems to be a holding area for data to be submitted to influx. It's been about 15 minutes and haven't gotten an error message yet.
As a side note, a folder starting with a hyphen is a pain in the butt. On Fri, Jul 17, 2020 at 8:23 AM Marsh-Mossadeghi, Dariush < dariush.marsh-mossade...@lv.com> wrote: > Hi, > > > > Ouch !! > > > > You may get some benefit from examining what settings you have at your > disposal in the following areas: > > - Whichever database backend you are using, tuning its disk write > policies or working memory allocations > > - Tuning the filesystem layer in your OS > > - Tuning firmware setting in your disk controller and/or disks > > HTH > > Dariush > > > > *From:* ntop-boun...@listgateway.unipi.it [mailto: > ntop-boun...@listgateway.unipi.it] *On Behalf Of *Munroe Sollog > *Sent:* 17 July 2020 13:02 > *To:* n...@unipi.it > *Subject:* Re: [Ntop] Performance pains > > > > ATTENTION: External email - This message has been sent from outside LV=. > > I meant to include this as well: > > > > [image: image.png] > > > > On Fri, Jul 17, 2020 at 7:59 AM Munroe Sollog <m...@lehigh.edu> wrote: > > While they may not me NVMe, I currently have 6 ssds in a raid 10 > configuration. > > > > On Fri, Jul 17, 2020 at 3:06 AM Marsh-Mossadeghi, Dariush < > dariush.marsh-mossade...@lv.com> wrote: > > Hi, > > > > I’d guess your disks can’t keep up. > > As a rule of thumb, if you’re using any timeseries database and you’re not > using solid state storage in 2020 you’re gonna have a bad time ;-) > > > > I’ve had ntopng monitoring a busy 10G pipe with upwards of 8k active hosts > on the ‘local’ network and running comfortably at a load average of no more > than 3-4 on a 16 core Xeon, using up less than 100G of memory. But it does > need some serious disk IO. > > > > Use iostat to check what %iowait looks like, it’s the percentage of the > time the CPU’s are waiting around for disk IO to complete. If that’s > anything more than 1-2% your bottleneck is most likely disk IO. > > > > HTH > > Dariush > > > > *From:* ntop-boun...@listgateway.unipi.it [mailto: > ntop-boun...@listgateway.unipi.it] *On Behalf Of *Munroe Sollog > *Sent:* 16 July 2020 19:21 > *To:* ntop@listgateway.unipi.it > *Subject:* [Ntop] Performance pains > > > > ATTENTION: External email - This message has been sent from outside LV=. > > looking in the ntopng log I am getting a lot of: > > > > [ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local > hosts timeseries dump. Disk slow or too many local hosts? > > ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or > too many local hosts? > > > > > I'm assuming the host can't keep up. Our local network is defined as a > /16 and it looks like we received about 50-60k flows per minute. For > reference my cpu: > > > > CPU(s): 56 > > On-line CPU(s) list: 0-55 > > Thread(s) per core: 2 > > Core(s) per socket: 14 > > Socket(s): 2 > > NUMA node(s): 2 > > Vendor ID: GenuineIntel > > CPU family: 6 > > Model: 63 > > Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz > > > > # free -h > > total used free shared buff/cache > available > > Mem: 251Gi 186Gi 45Gi 209Mi 20Gi > 161Gi > > Swap: 29Gi 347Mi 29Gi > > > > > > > > -- > > Munroe Sollog > > Senior Network Engineer > > mun...@lehigh.edu > > > This email (including any attachment) may contain confidential and/ or > legally privileged information. If you are not the intended recipient, > please notify us at d...@lv.com and destroy it and any copies. > Unauthorised access, use, disclosure, storage or copying of this email is > not permitted and, unless you are the intended recipient, you are not > entitled to rely on it in any way. Any opinions expressed in this email are > those of the individual sending it and not necessarily those of LV=. > > This email is believed to be free of any virus or other defect. However, > communication by email cannot be guaranteed to be free from defect, error > free or secure. If you choose to communicate with us by email you must > realise that there can be no guarantee of privacy and you should carry out > your own security checks before opening any email or attachment. > > LV= accepts no liability for any loss or damage which may be caused by any > lack of privacy, software viruses or other defect. > > LV= reserves the right to monitor and inspect any email (including any > attachment) sent to and/or from LV= for reasons of security and for > monitoring internal compliance with our office policies. LV= may use email > monitoring or blocking software at its discretion. You are responsible for > ensuring that any email you send is appropriate and within the bounds of > the law. > > LV= and Liverpool Victoria are trademarks of Liverpool Victoria Financial > Services Limited and LV= and Liverpool Victoria are trading styles of the > Liverpool Victoria group of companies. Liverpool Victoria Financial > Services Limited , registered in England with registration number 12383237 > is authorised by the Prudential Regulation Authority and regulated by the > Financial Conduct Authority and the Prudential Regulation Authority, > register number 110035.The registered office address for all LV= companies > is County Gates, Bournemouth, BH1 2NF. Information about the LV= group of > companies can be found via this link www.lv.com/legal/lvcompanies > > _______________________________________________ > Ntop mailing list > Ntop@listgateway.unipi.it > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > -- > > Munroe Sollog > > Senior Network Engineer > > mun...@lehigh.edu > > > > > -- > > Munroe Sollog > > Senior Network Engineer > > mun...@lehigh.edu > _______________________________________________ > Ntop mailing list > Ntop@listgateway.unipi.it > http://listgateway.unipi.it/mailman/listinfo/ntop -- Munroe Sollog Senior Network Engineer mun...@lehigh.edu
_______________________________________________ Ntop mailing list Ntop@listgateway.unipi.it http://listgateway.unipi.it/mailman/listinfo/ntop
