Re: [Ntop] Performance pains
As a first step, I decided to create a ram disk and move the ts_export folders in /var/lib/ntopng/[-1 - 24]/ to the ram disk. I figured this seemed relatively safe as the data in this folder seems to be a holding area for data to be submitted to influx. It's been about 15 minutes and haven't gotten an error message yet. As a side note, a folder starting with a hyphen is a pain in the butt. On Fri, Jul 17, 2020 at 8:23 AM Marsh-Mossadeghi, Dariush < dariush.marsh-mossade...@lv.com> wrote: > Hi, > > > > Ouch !! > > > > You may get some benefit from examining what settings you have at your > disposal in the following areas: > > -Whichever database backend you are using, tuning its disk write > policies or working memory allocations > > -Tuning the filesystem layer in your OS > > -Tuning firmware setting in your disk controller and/or disks > > HTH > > Dariush > > > > *From:* ntop-boun...@listgateway.unipi.it [mailto: > ntop-boun...@listgateway.unipi.it] *On Behalf Of *Munroe Sollog > *Sent:* 17 July 2020 13:02 > *To:* n...@unipi.it > *Subject:* Re: [Ntop] Performance pains > > > > ATTENTION: External email - This message has been sent from outside LV=. > > I meant to include this as well: > > > > [image: image.png] > > > > On Fri, Jul 17, 2020 at 7:59 AM Munroe Sollog wrote: > > While they may not me NVMe, I currently have 6 ssds in a raid 10 > configuration. > > > > On Fri, Jul 17, 2020 at 3:06 AM Marsh-Mossadeghi, Dariush < > dariush.marsh-mossade...@lv.com> wrote: > > Hi, > > > > I’d guess your disks can’t keep up. > > As a rule of thumb, if you’re using any timeseries database and you’re not > using solid state storage in 2020 you’re gonna have a bad time ;-) > > > > I’ve had ntopng monitoring a busy 10G pipe with upwards of 8k active hosts > on the ‘local’ network and running comfortably at a load average of no more > than 3-4 on a 16 core Xeon, using up less than 100G of memory. But it does > need some serious disk IO. > > > > Use iostat to check what %iowait looks like, it’s the percentage of the > time the CPU’s are waiting around for disk IO to complete. If that’s > anything more than 1-2% your bottleneck is most likely disk IO. > > > > HTH > > Dariush > > > > *From:* ntop-boun...@listgateway.unipi.it [mailto: > ntop-boun...@listgateway.unipi.it] *On Behalf Of *Munroe Sollog > *Sent:* 16 July 2020 19:21 > *To:* ntop@listgateway.unipi.it > *Subject:* [Ntop] Performance pains > > > > ATTENTION: External email - This message has been sent from outside LV=. > > looking in the ntopng log I am getting a lot of: > > > > [ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local > hosts timeseries dump. Disk slow or too many local hosts? > > ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or > too many local hosts? > > > > > I'm assuming the host can't keep up. Our local network is defined as a > /16 and it looks like we received about 50-60k flows per minute. For > reference my cpu: > > > > CPU(s): 56 > > On-line CPU(s) list: 0-55 > > Thread(s) per core: 2 > > Core(s) per socket: 14 > > Socket(s): 2 > > NUMA node(s):2 > > Vendor ID: GenuineIntel > > CPU family: 6 > > Model: 63 > > Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz > > > > # free -h > > totalusedfree shared buff/cache > available > > Mem: 251Gi 186Gi45Gi 209Mi20Gi > 161Gi > > Swap: 29Gi 347Mi29Gi > > > > > > > > -- > > Munroe Sollog > > Senior Network Engineer > > mun...@lehigh.edu > > > This email (including any attachment) may contain confidential and/ or > legally privileged information. If you are not the intended recipient, > please notify us at d...@lv.com and destroy it and any copies. > Unauthorised access, use, disclosure, storage or copying of this email is > not permitted and, unless you are the intended recipient, you are not > entitled to rely on it in any way. Any opinions expressed in this email are > those of the individual sending it and not necessarily those of LV=. > > This email is believed to be free of any virus or other defect. However, > communication by email cannot be guaranteed to be free from defect, error > free or secure. If you choose to communicate with us by email you must > realise that there can be no guarantee of privacy and you should car
Re: [Ntop] Performance pains
Hi, Ouch !! You may get some benefit from examining what settings you have at your disposal in the following areas: -Whichever database backend you are using, tuning its disk write policies or working memory allocations -Tuning the filesystem layer in your OS -Tuning firmware setting in your disk controller and/or disks HTH Dariush From: ntop-boun...@listgateway.unipi.it [mailto:ntop-boun...@listgateway.unipi.it] On Behalf Of Munroe Sollog Sent: 17 July 2020 13:02 To: n...@unipi.it Subject: Re: [Ntop] Performance pains ATTENTION: External email - This message has been sent from outside LV=. I meant to include this as well: [image.png] On Fri, Jul 17, 2020 at 7:59 AM Munroe Sollog mailto:m...@lehigh.edu>> wrote: While they may not me NVMe, I currently have 6 ssds in a raid 10 configuration. On Fri, Jul 17, 2020 at 3:06 AM Marsh-Mossadeghi, Dariush mailto:dariush.marsh-mossade...@lv.com>> wrote: Hi, I’d guess your disks can’t keep up. As a rule of thumb, if you’re using any timeseries database and you’re not using solid state storage in 2020 you’re gonna have a bad time ;-) I’ve had ntopng monitoring a busy 10G pipe with upwards of 8k active hosts on the ‘local’ network and running comfortably at a load average of no more than 3-4 on a 16 core Xeon, using up less than 100G of memory. But it does need some serious disk IO. Use iostat to check what %iowait looks like, it’s the percentage of the time the CPU’s are waiting around for disk IO to complete. If that’s anything more than 1-2% your bottleneck is most likely disk IO. HTH Dariush From: ntop-boun...@listgateway.unipi.it<mailto:ntop-boun...@listgateway.unipi.it> [mailto:ntop-boun...@listgateway.unipi.it<mailto:ntop-boun...@listgateway.unipi.it>] On Behalf Of Munroe Sollog Sent: 16 July 2020 19:21 To: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it> Subject: [Ntop] Performance pains ATTENTION: External email - This message has been sent from outside LV=. looking in the ntopng log I am getting a lot of: [ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts? ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts? I'm assuming the host can't keep up. Our local network is defined as a /16 and it looks like we received about 50-60k flows per minute. For reference my cpu: CPU(s): 56 On-line CPU(s) list: 0-55 Thread(s) per core: 2 Core(s) per socket: 14 Socket(s): 2 NUMA node(s):2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz # free -h totalusedfree shared buff/cache available Mem: 251Gi 186Gi45Gi 209Mi20Gi 161Gi Swap: 29Gi 347Mi29Gi -- Munroe Sollog Senior Network Engineer mun...@lehigh.edu<mailto:mun...@lehigh.edu> This email (including any attachment) may contain confidential and/ or legally privileged information. If you are not the intended recipient, please notify us at d...@lv.com<mailto:d...@lv.com> and destroy it and any copies. Unauthorised access, use, disclosure, storage or copying of this email is not permitted and, unless you are the intended recipient, you are not entitled to rely on it in any way. Any opinions expressed in this email are those of the individual sending it and not necessarily those of LV=. This email is believed to be free of any virus or other defect. However, communication by email cannot be guaranteed to be free from defect, error free or secure. If you choose to communicate with us by email you must realise that there can be no guarantee of privacy and you should carry out your own security checks before opening any email or attachment. LV= accepts no liability for any loss or damage which may be caused by any lack of privacy, software viruses or other defect. LV= reserves the right to monitor and inspect any email (including any attachment) sent to and/or from LV= for reasons of security and for monitoring internal compliance with our office policies. LV= may use email monitoring or blocking software at its discretion. You are responsible for ensuring that any email you send is appropriate and within the bounds of the law. LV= and Liverpool Victoria are trademarks of Liverpool Victoria Financial Services Limited and LV= and Liverpool Victoria are trading styles of the Liverpool Victoria group of companies. Liverpool Victoria Financial Services Limited , registered in England with registration number 12383237 is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority, register number 110035.The registered office address for all
Re: [Ntop] Performance pains
I meant to include this as well: [image: image.png] On Fri, Jul 17, 2020 at 7:59 AM Munroe Sollog wrote: > While they may not me NVMe, I currently have 6 ssds in a raid 10 > configuration. > > On Fri, Jul 17, 2020 at 3:06 AM Marsh-Mossadeghi, Dariush < > dariush.marsh-mossade...@lv.com> wrote: > >> Hi, >> >> >> >> I’d guess your disks can’t keep up. >> >> As a rule of thumb, if you’re using any timeseries database and you’re >> not using solid state storage in 2020 you’re gonna have a bad time ;-) >> >> >> >> I’ve had ntopng monitoring a busy 10G pipe with upwards of 8k active >> hosts on the ‘local’ network and running comfortably at a load average of >> no more than 3-4 on a 16 core Xeon, using up less than 100G of memory. But >> it does need some serious disk IO. >> >> >> >> Use iostat to check what %iowait looks like, it’s the percentage of the >> time the CPU’s are waiting around for disk IO to complete. If that’s >> anything more than 1-2% your bottleneck is most likely disk IO. >> >> >> >> HTH >> >> Dariush >> >> >> >> *From:* ntop-boun...@listgateway.unipi.it [mailto: >> ntop-boun...@listgateway.unipi.it] *On Behalf Of *Munroe Sollog >> *Sent:* 16 July 2020 19:21 >> *To:* ntop@listgateway.unipi.it >> *Subject:* [Ntop] Performance pains >> >> >> >> ATTENTION: External email - This message has been sent from outside LV=. >> >> looking in the ntopng log I am getting a lot of: >> >> >> >> [ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local >> hosts timeseries dump. Disk slow or too many local hosts? >> >> ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or >> too many local hosts? >> >> >> >> >> I'm assuming the host can't keep up. Our local network is defined as a >> /16 and it looks like we received about 50-60k flows per minute. For >> reference my cpu: >> >> >> >> CPU(s): 56 >> >> On-line CPU(s) list: 0-55 >> >> Thread(s) per core: 2 >> >> Core(s) per socket: 14 >> >> Socket(s): 2 >> >> NUMA node(s):2 >> >> Vendor ID: GenuineIntel >> >> CPU family: 6 >> >> Model: 63 >> >> Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz >> >> >> >> # free -h >> >> totalusedfree shared buff/cache >> available >> >> Mem: 251Gi 186Gi45Gi 209Mi20Gi >> 161Gi >> >> Swap: 29Gi 347Mi29Gi >> >> >> >> >> >> >> >> -- >> >> Munroe Sollog >> >> Senior Network Engineer >> >> mun...@lehigh.edu >> >> This email (including any attachment) may contain confidential and/ or >> legally privileged information. If you are not the intended recipient, >> please notify us at d...@lv.com and destroy it and any copies. >> Unauthorised access, use, disclosure, storage or copying of this email is >> not permitted and, unless you are the intended recipient, you are not >> entitled to rely on it in any way. Any opinions expressed in this email are >> those of the individual sending it and not necessarily those of LV=. >> >> This email is believed to be free of any virus or other defect. However, >> communication by email cannot be guaranteed to be free from defect, error >> free or secure. If you choose to communicate with us by email you must >> realise that there can be no guarantee of privacy and you should carry out >> your own security checks before opening any email or attachment. >> >> LV= accepts no liability for any loss or damage which may be caused by >> any lack of privacy, software viruses or other defect. >> >> LV= reserves the right to monitor and inspect any email (including any >> attachment) sent to and/or from LV= for reasons of security and for >> monitoring internal compliance with our office policies. LV= may use email >> monitoring or blocking software at its discretion. You are responsible for >> ensuring that any email you send is appropriate and within the bounds of >> the law. >> >> LV= and Liverpool Victoria are trademarks of Liverpool Victoria Financial >> Services Limited and LV= and Liverpool Victoria are trading styles of the >> Liverpool Victoria group of companies. Liverpool Victo
Re: [Ntop] Performance pains
While they may not me NVMe, I currently have 6 ssds in a raid 10 configuration. On Fri, Jul 17, 2020 at 3:06 AM Marsh-Mossadeghi, Dariush < dariush.marsh-mossade...@lv.com> wrote: > Hi, > > > > I’d guess your disks can’t keep up. > > As a rule of thumb, if you’re using any timeseries database and you’re not > using solid state storage in 2020 you’re gonna have a bad time ;-) > > > > I’ve had ntopng monitoring a busy 10G pipe with upwards of 8k active hosts > on the ‘local’ network and running comfortably at a load average of no more > than 3-4 on a 16 core Xeon, using up less than 100G of memory. But it does > need some serious disk IO. > > > > Use iostat to check what %iowait looks like, it’s the percentage of the > time the CPU’s are waiting around for disk IO to complete. If that’s > anything more than 1-2% your bottleneck is most likely disk IO. > > > > HTH > > Dariush > > > > *From:* ntop-boun...@listgateway.unipi.it [mailto: > ntop-boun...@listgateway.unipi.it] *On Behalf Of *Munroe Sollog > *Sent:* 16 July 2020 19:21 > *To:* ntop@listgateway.unipi.it > *Subject:* [Ntop] Performance pains > > > > ATTENTION: External email - This message has been sent from outside LV=. > > looking in the ntopng log I am getting a lot of: > > > > [ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local > hosts timeseries dump. Disk slow or too many local hosts? > > ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or > too many local hosts? > > > > > I'm assuming the host can't keep up. Our local network is defined as a > /16 and it looks like we received about 50-60k flows per minute. For > reference my cpu: > > > > CPU(s): 56 > > On-line CPU(s) list: 0-55 > > Thread(s) per core: 2 > > Core(s) per socket: 14 > > Socket(s): 2 > > NUMA node(s):2 > > Vendor ID: GenuineIntel > > CPU family: 6 > > Model: 63 > > Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz > > > > # free -h > > totalusedfree shared buff/cache > available > > Mem: 251Gi 186Gi45Gi 209Mi20Gi > 161Gi > > Swap: 29Gi 347Mi29Gi > > > > > > > > -- > > Munroe Sollog > > Senior Network Engineer > > mun...@lehigh.edu > > This email (including any attachment) may contain confidential and/ or > legally privileged information. If you are not the intended recipient, > please notify us at d...@lv.com and destroy it and any copies. > Unauthorised access, use, disclosure, storage or copying of this email is > not permitted and, unless you are the intended recipient, you are not > entitled to rely on it in any way. Any opinions expressed in this email are > those of the individual sending it and not necessarily those of LV=. > > This email is believed to be free of any virus or other defect. However, > communication by email cannot be guaranteed to be free from defect, error > free or secure. If you choose to communicate with us by email you must > realise that there can be no guarantee of privacy and you should carry out > your own security checks before opening any email or attachment. > > LV= accepts no liability for any loss or damage which may be caused by any > lack of privacy, software viruses or other defect. > > LV= reserves the right to monitor and inspect any email (including any > attachment) sent to and/or from LV= for reasons of security and for > monitoring internal compliance with our office policies. LV= may use email > monitoring or blocking software at its discretion. You are responsible for > ensuring that any email you send is appropriate and within the bounds of > the law. > > LV= and Liverpool Victoria are trademarks of Liverpool Victoria Financial > Services Limited and LV= and Liverpool Victoria are trading styles of the > Liverpool Victoria group of companies. Liverpool Victoria Financial > Services Limited , registered in England with registration number 12383237 > is authorised by the Prudential Regulation Authority and regulated by the > Financial Conduct Authority and the Prudential Regulation Authority, > register number 110035.The registered office address for all LV= companies > is County Gates, Bournemouth, BH1 2NF. Information about the LV= group of > companies can be found via this link www.lv.com/legal/lvcompanies > ___ > Ntop mailing list > Ntop@listgateway.unipi.it > http://listgateway.unipi.it/mailman/listinfo/ntop -- Munroe Sollog Senior Network Engineer mun...@lehigh.edu ___ Ntop mailing list Ntop@listgateway.unipi.it http://listgateway.unipi.it/mailman/listinfo/ntop
Re: [Ntop] Performance pains
Hi, I’d guess your disks can’t keep up. As a rule of thumb, if you’re using any timeseries database and you’re not using solid state storage in 2020 you’re gonna have a bad time ;-) I’ve had ntopng monitoring a busy 10G pipe with upwards of 8k active hosts on the ‘local’ network and running comfortably at a load average of no more than 3-4 on a 16 core Xeon, using up less than 100G of memory. But it does need some serious disk IO. Use iostat to check what %iowait looks like, it’s the percentage of the time the CPU’s are waiting around for disk IO to complete. If that’s anything more than 1-2% your bottleneck is most likely disk IO. HTH Dariush From: ntop-boun...@listgateway.unipi.it [mailto:ntop-boun...@listgateway.unipi.it] On Behalf Of Munroe Sollog Sent: 16 July 2020 19:21 To: ntop@listgateway.unipi.it Subject: [Ntop] Performance pains ATTENTION: External email - This message has been sent from outside LV=. looking in the ntopng log I am getting a lot of: [ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts? ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts? I'm assuming the host can't keep up. Our local network is defined as a /16 and it looks like we received about 50-60k flows per minute. For reference my cpu: CPU(s): 56 On-line CPU(s) list: 0-55 Thread(s) per core: 2 Core(s) per socket: 14 Socket(s): 2 NUMA node(s):2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz # free -h totalusedfree shared buff/cache available Mem: 251Gi 186Gi45Gi 209Mi20Gi 161Gi Swap: 29Gi 347Mi29Gi -- Munroe Sollog Senior Network Engineer mun...@lehigh.edu<mailto:mun...@lehigh.edu> This email (including any attachment) may contain confidential and/ or legally privileged information. If you are not the intended recipient, please notify us at d...@lv.com and destroy it and any copies. Unauthorised access, use, disclosure, storage or copying of this email is not permitted and, unless you are the intended recipient, you are not entitled to rely on it in any way. Any opinions expressed in this email are those of the individual sending it and not necessarily those of LV=. This email is believed to be free of any virus or other defect. However, communication by email cannot be guaranteed to be free from defect, error free or secure. If you choose to communicate with us by email you must realise that there can be no guarantee of privacy and you should carry out your own security checks before opening any email or attachment. LV= accepts no liability for any loss or damage which may be caused by any lack of privacy, software viruses or other defect. LV= reserves the right to monitor and inspect any email (including any attachment) sent to and/or from LV= for reasons of security and for monitoring internal compliance with our office policies. LV= may use email monitoring or blocking software at its discretion. You are responsible for ensuring that any email you send is appropriate and within the bounds of the law. LV= and Liverpool Victoria are trademarks of Liverpool Victoria Financial Services Limited and LV= and Liverpool Victoria are trading styles of the Liverpool Victoria group of companies. Liverpool Victoria Financial Services Limited , registered in England with registration number 12383237 is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority, register number 110035.The registered office address for all LV= companies is County Gates, Bournemouth, BH1 2NF. Information about the LV= group of companies can be found via this link www.lv.com/legal/lvcompanies<http://www.lv.com/legal/lvcompanies/> ___ Ntop mailing list Ntop@listgateway.unipi.it http://listgateway.unipi.it/mailman/listinfo/ntop
[Ntop] Performance pains
looking in the ntopng log I am getting a lot of: [ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts? ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts? I'm assuming the host can't keep up. Our local network is defined as a /16 and it looks like we received about 50-60k flows per minute. For reference my cpu: CPU(s): 56 On-line CPU(s) list: 0-55 Thread(s) per core: 2 Core(s) per socket: 14 Socket(s): 2 NUMA node(s):2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz # free -h totalusedfree shared buff/cache available Mem: 251Gi 186Gi45Gi 209Mi20Gi 161Gi Swap: 29Gi 347Mi29Gi -- Munroe Sollog Senior Network Engineer mun...@lehigh.edu ___ Ntop mailing list Ntop@listgateway.unipi.it http://listgateway.unipi.it/mailman/listinfo/ntop
