Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
I think I understand better. Now .. but is there still a security benefit from having the different services in their own jails ? (even if the jail cells come with their own metaphorical swimming pool and armoury ) or is it that the jails don’t offer enough compared with the additional workload of managing multiple copies of libraries/binaries in the system... ? On Thu, 9 Mar 2023 at 12:29, Stuart Henderson wrote: > > On 2023/03/08 10:10, Glen Gunsalus wrote: > > > > On 3/7/23 15:33, Stuart Henderson wrote: > > > On 2023-03-07, Glen Gunsalus wrote: > > > > To get this running cp'd perl (/usr/bin/perl) and relevant perl libs > > > > (/usr/lib/[libs.so|libm.so|libperl.so] /usr/libexec/ld.so) to > > > > /var/www/usr/[bin|lib|libexec] > > > > > > You shouldn't need that bit (and it is safer not to) - smokeping_fcgi > > > does not chroot. > > > > > > > > Hmm, I did this on the basis of a post by you (5/11/20) in response to Tom > > (5/10/20) which I interpreted as needing several files moved into www > > "jail." > > No that was me saying "this software is not really meant to work with > chroot and if you're copying enough into the chroot that it works, > you're providing a lot of extra tools to someone who is able to execute > code within the jail" > > > quote-- > > bgplg is designed to run in a jail, it is a small C program and even > > then it needs specially compiled versions of the external dependencies > > (ping, bgpctl etc). > > > > Smokeping isn't - if you want to run the graph generating part of > > smokeping (i.e. the cgi/fcgi script) inside a chroot jail, a whole lot > > more is needed - a copy of perl and various modules, rrdtool, > > rrdtool's library dependencies, fonts, and I think there were config > > files for some of the libraries. I did this in the past but it's a > > real mess and easy to break at update time, and the amount of things > > copied in means that the chroot ends up more as "luxury camping" than > > "jail" > > end quote--- > > > > I had been running smokeping and mrtg with apache for a number of years, > > but when OpenBSD abandoned apache I looked at nginx for transition then > > httpd came along and looked both more attractive and likely to be more long > > lived under OpenBSD. > > > > It was Tom's post that got me started down the httpd path. I have been > > running with httpd since that time. > > I can't remember the details, but think I initially tried w/o the cp'd > > files, but was not successful so began incrementally moving goodies into > > /var/www until it worked. > > I will try rm'ing or mv'ing those in /var/www and see how it goes. > > > > Thanks for your help. > > > > Regards, Glen > -- Kindest regards, Tom Smyth.
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On 2023/03/08 10:10, Glen Gunsalus wrote: > > On 3/7/23 15:33, Stuart Henderson wrote: > > On 2023-03-07, Glen Gunsalus wrote: > > > To get this running cp'd perl (/usr/bin/perl) and relevant perl libs > > > (/usr/lib/[libs.so|libm.so|libperl.so] /usr/libexec/ld.so) to > > > /var/www/usr/[bin|lib|libexec] > > > > You shouldn't need that bit (and it is safer not to) - smokeping_fcgi > > does not chroot. > > > > > Hmm, I did this on the basis of a post by you (5/11/20) in response to Tom > (5/10/20) which I interpreted as needing several files moved into www "jail." No that was me saying "this software is not really meant to work with chroot and if you're copying enough into the chroot that it works, you're providing a lot of extra tools to someone who is able to execute code within the jail" > quote-- > bgplg is designed to run in a jail, it is a small C program and even > then it needs specially compiled versions of the external dependencies > (ping, bgpctl etc). > > Smokeping isn't - if you want to run the graph generating part of > smokeping (i.e. the cgi/fcgi script) inside a chroot jail, a whole lot > more is needed - a copy of perl and various modules, rrdtool, > rrdtool's library dependencies, fonts, and I think there were config > files for some of the libraries. I did this in the past but it's a > real mess and easy to break at update time, and the amount of things > copied in means that the chroot ends up more as "luxury camping" than > "jail" > end quote--- > > I had been running smokeping and mrtg with apache for a number of years, but > when OpenBSD abandoned apache I looked at nginx for transition then httpd > came along and looked both more attractive and likely to be more long lived > under OpenBSD. > > It was Tom's post that got me started down the httpd path. I have been > running with httpd since that time. > I can't remember the details, but think I initially tried w/o the cp'd files, > but was not successful so began incrementally moving goodies into /var/www > until it worked. > I will try rm'ing or mv'ing those in /var/www and see how it goes. > > Thanks for your help. > > Regards, Glen
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
Morning Glen, Stuart all, yep ... Stuarts comments re chroot glamping vs chroot jails made me gigle all right... the way I think I have it working is that smokeping and rrdcached are running outside the jail with symbolic links to sockets inside the httpd chroot jail /var/www/... and httpd picks up those sockets and plays with them inside the jail... relevant output from my ps -aux list USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND _smokepi 98525 9.7 1.3 98040 111580 ?? S 6:31AM1:03.00 /usr/bin/perl /usr/local/bin/smokeping_cgi /etc/smokeping/config _rrdcach 67082 0.0 0.1 9272 7952 ?? S 6:31AM0:03.21 /usr/local/bin/rrdcached -b /var/db/smokeping -B -m 770 -l unix:/var/www/run/rrd _smokepi 25394 0.0 0.1 43244 10536 ?? I 6:31AM0:00.03 /usr/bin/perl /usr/local/bin/smokeping _smokepi 57899 0.0 0.3 43752 21276 ?? S 6:31AM0:01.31 perl: /usr/local/bin/smokeping [FPing] (perl) _smokepi 74710 0.4 0.3 43244 21480 ?? S 6:31AM0:03.49 perl: /usr/local/bin/smokeping [DNS] (perl) _smokepi 76253 0.2 0.0 2892 2916 ?? Sp 6:47AM0:00.15 /usr/local/sbin/fping -C 61 -q -B1 -r1 -b64 -t125 -i10 -p1 10.20.127.2 10.139.25... when I get around to it ... I would like rrdcached and smokeping in another / separate glamping site / luxury chroot jail to the cgi binary... Comments thoughts welcome ... On Wed, 8 Mar 2023 at 19:26, Glen Gunsalus wrote: > > > On 3/7/23 15:33, Stuart Henderson wrote: > > On 2023-03-07, Glen Gunsalus wrote: > >> To get this running cp'd perl (/usr/bin/perl) and relevant perl libs > >> (/usr/lib/[libs.so|libm.so|libperl.so] /usr/libexec/ld.so) to > >> /var/www/usr/[bin|lib|libexec] > > > > You shouldn't need that bit (and it is safer not to) - smokeping_fcgi > > does not chroot. > > > > > Hmm, I did this on the basis of a post by you (5/11/20) in response to Tom > (5/10/20) which I interpreted as needing several files moved into www "jail." > > quote-- > bgplg is designed to run in a jail, it is a small C program and even > then it needs specially compiled versions of the external dependencies > (ping, bgpctl etc). > > Smokeping isn't - if you want to run the graph generating part of > smokeping (i.e. the cgi/fcgi script) inside a chroot jail, a whole lot > more is needed - a copy of perl and various modules, rrdtool, > rrdtool's library dependencies, fonts, and I think there were config > files for some of the libraries. I did this in the past but it's a > real mess and easy to break at update time, and the amount of things > copied in means that the chroot ends up more as "luxury camping" than > "jail" > end quote--- > > I had been running smokeping and mrtg with apache for a number of years, but > when OpenBSD abandoned apache I looked at nginx for transition then httpd > came along and looked both more attractive and likely to be more long lived > under OpenBSD. > > It was Tom's post that got me started down the httpd path. I have been > running with httpd since that time. > I can't remember the details, but think I initially tried w/o the cp'd files, > but was not successful so began incrementally moving goodies into /var/www > until it worked. > I will try rm'ing or mv'ing those in /var/www and see how it goes. > > Thanks for your help. > > Regards, Glen > -- Kindest regards, Tom Smyth.
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On 3/7/23 15:33, Stuart Henderson wrote: On 2023-03-07, Glen Gunsalus wrote: To get this running cp'd perl (/usr/bin/perl) and relevant perl libs (/usr/lib/[libs.so|libm.so|libperl.so] /usr/libexec/ld.so) to /var/www/usr/[bin|lib|libexec] You shouldn't need that bit (and it is safer not to) - smokeping_fcgi does not chroot. Hmm, I did this on the basis of a post by you (5/11/20) in response to Tom (5/10/20) which I interpreted as needing several files moved into www "jail." quote-- bgplg is designed to run in a jail, it is a small C program and even then it needs specially compiled versions of the external dependencies (ping, bgpctl etc). Smokeping isn't - if you want to run the graph generating part of smokeping (i.e. the cgi/fcgi script) inside a chroot jail, a whole lot more is needed - a copy of perl and various modules, rrdtool, rrdtool's library dependencies, fonts, and I think there were config files for some of the libraries. I did this in the past but it's a real mess and easy to break at update time, and the amount of things copied in means that the chroot ends up more as "luxury camping" than "jail" end quote--- I had been running smokeping and mrtg with apache for a number of years, but when OpenBSD abandoned apache I looked at nginx for transition then httpd came along and looked both more attractive and likely to be more long lived under OpenBSD. It was Tom's post that got me started down the httpd path. I have been running with httpd since that time. I can't remember the details, but think I initially tried w/o the cp'd files, but was not successful so began incrementally moving goodies into /var/www until it worked. I will try rm'ing or mv'ing those in /var/www and see how it goes. Thanks for your help. Regards, Glen
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
Folks,
Just to say keeping rrdcached for smokeping, and just using the smokeping.sock
server "default" {
listen on * port 80
location "/smokeping/smokeping.fcgi*" {
fastcgi {
socket "/run/smokeping.sock"
}
}
}
is way faster ... for the user interface... ... Ill let you know if
there are any negative impact on the graphs ...
Thanks
Tom Smyth
On Wed, 8 Mar 2023 at 15:21, Tom Smyth wrote:
>
> Hello
> I found that RRDCached helps with the gaps in the graphs... (write
> i/o burst smoothing) (which is the main reason I went with rrdcached
>
> but it did not help so much on the user interface / web rendering front ...
> (perhaps I could try (if it is even possible) to try the following
>
> write rrds using smokeping --> rrdcached-->rrdfile
> and separately read
> rrdfile --> smokeping_fcgi --> httpd
> or does rrdcahced need to exclusively manage I/O ( read and write)
> with the rrd files. ?
> Ill investgate this a bit more ... (comments and ideas welcome ..
>
> On Wed, 8 Mar 2023 at 14:16, Stuart Henderson wrote:
> >
> > On 2023/03/07 14:38, Tom Smyth wrote:
> > > the config below seems to get rrdcached working with httpd in OpenBSD.
> > > ...
> >
> > Thanks, I've added this to the pkg-readme.
> >
> > > the loading of the smokeping detailed graphs still takes a while ... but
> > > I
> > > will do further dianostics...
> >
> > Do check to make sure that using rrdcached does actually improve things
> > for your setup, you might find that it doesn't.
> >
>
>
> --
> Kindest regards,
> Tom Smyth.
--
Kindest regards,
Tom Smyth.
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
Hello I found that RRDCached helps with the gaps in the graphs... (write i/o burst smoothing) (which is the main reason I went with rrdcached but it did not help so much on the user interface / web rendering front ... (perhaps I could try (if it is even possible) to try the following write rrds using smokeping --> rrdcached-->rrdfile and separately read rrdfile --> smokeping_fcgi --> httpd or does rrdcahced need to exclusively manage I/O ( read and write) with the rrd files. ? Ill investgate this a bit more ... (comments and ideas welcome .. On Wed, 8 Mar 2023 at 14:16, Stuart Henderson wrote: > > On 2023/03/07 14:38, Tom Smyth wrote: > > the config below seems to get rrdcached working with httpd in OpenBSD. ... > > Thanks, I've added this to the pkg-readme. > > > the loading of the smokeping detailed graphs still takes a while ... but I > > will do further dianostics... > > Do check to make sure that using rrdcached does actually improve things > for your setup, you might find that it doesn't. > -- Kindest regards, Tom Smyth.
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On 2023/03/07 14:38, Tom Smyth wrote: > the config below seems to get rrdcached working with httpd in OpenBSD. ... Thanks, I've added this to the pkg-readme. > the loading of the smokeping detailed graphs still takes a while ... but I > will do further dianostics... Do check to make sure that using rrdcached does actually improve things for your setup, you might find that it doesn't.
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
Hello folks
Stuart mentioned that the package readme only contains info on
rrdcached and smokeping running with nginx ( i have been running it
with httpd for a year or so)
here is my configs to get smokeping running with rrdcached on
openbsd's httpd (base)...
comments welcome...
the only gotcha is that when adding in sub menus,
you need to set the permissions on the directory structure created in
/var/db/smokeping
smoke1# pwd
/var/db/smokeping
smoke1# ls -al
total 160
drwxr-xr-x 40 _smokeping _rrdcached 1024 Mar 8 05:49 .
drwxr-xr-x 12 rootwheel512 Mar 7 12:50 ..
drwxrwxr-x 5 _smokeping _rrdcached 512 Mar 7 06:40 CORE
/var/db/smokeping/top_level_menu/sub_level_menu/
chmod 775 /var/db/smokeping/top_level_menu
chmod 775 /var/db/smokeping/top_level_menu/sub_level_menu
#rc.conf.local
pkg_scripts=rrdcached smokeping smokeping_fcgi
rrdcached_flags=-b /var/db/smokeping -B -m 770 -l
unix:/var/www/run/rrd/rrdcached.sock -j /var/db/smokeping/journal -p
/var/www/run/rrd/rrdcached.pid
#
#httpd.conf###
server "default" {
listen on * port 80
location "/smokeping/smokeping.fcgi*" {
fastcgi {
socket "/run/smokeping.sock"
param RRDCACHED_ADDRESS "unix:/var/www/run/rrd/rrdcached.sock"
}
root "/"
}
###
###etc-smokeping-config###
smoke1$ cat /etc/smokeping/config
*** General ***
owner= Wireless Connect Managed Networks
contact = [Redacted] @wirelessconnect.eu
mailhost = [redacted]
#sendmail = /usr/sbin/sendmail
# NOTE: do not put the Image Cache below cgi-bin
# since all files under cgi-bin will be executed ... this is not
# good for images.
imgcache = /var/www/htdocs/smokeping/cache
imgurl = cache
datadir = /var/db/smokeping
piddir = /var/run/smokeping
cgiurl = http://[redacted]/smokeping/smokeping.fcgi
smokemail = /etc/smokeping/smokemail
tmail = /etc/smokeping/tmail
# specify this to get syslog logging
syslogfacility = local0
syslogpriority = debug
# each probe is now run in its own process
# disable this to revert to the old behaviour
concurrentprobes = yes
*** Alerts ***
to = [redacted] @ wirelessconnect.eu
from = [reacted] @wirelessconnect.eu
#mailtemplate = /var/somewhere/mailtemplate
+Sustained_5%_loss
type = loss
# in percent
edgetrigger = yes
pattern = >=5%,>=5%,>=5%
comment = Sustained Packet Loss more than 5%
+Sudden_10%_Loss
type = loss
# in percent
edgetrigger = yes
pattern = ==0%,==0%,>10%
comment = Single Incident Packet Loss more than 10% Detected!
+Sudden_20%_Loss
type= loss
# in percent
edgetrigger = yes
pattern = ==0%,==0%,>20%
comment = Single Incident Packet Loss more than 20% Detected!
+Sporadic_Loss
type = loss
# in percent
pattern = ==0%,==0%,*8*,*8*,>20%,*8*,>20%,>20%,*8*,>0%
comment = High Packet loss occured at least 4 times over the last two hours
#+anydelay
#type = rtt
# # in milliseconds
# pattern = >1
# comment = Just for testing
+Latency_Over_50ms
type = rtt
# in milliseconds
edgetrigger = yes
pattern = <45,<45,<45,<45,<45,<45,>50
comment = Sudden High Latency greater than 50ms!
+Latency_Over_200ms
type = rtt
# in milliseconds
edgetrigger = yes
pattern = <100,<100,>200
comment = Sudden High Latency greater than 200ms!
+Latency_Constantly_Over_100ms
type = rtt
# in milliseconds
edgetrigger = yes
pattern = >100,>100,>100,>100,>100
comment = Constant High Latency greater than 100 ms for over 5 samples in a row
+Offline_at_startup
type = rtt
# in milliseconds
edgetrigger = yes
pattern = ==S,==U
comment = System offline at startup!
*** Database ***
step = 90
pings= 60
# consfn mrhb steps total
AVERAGE 0.5 1 1008
AVERAGE 0.5 12 4320
MIN 0.5 12 4320
MAX 0.5 12 4320
AVERAGE 0.5 144 720
MAX 0.5 144 720
MIN 0.5 144 720
*** Presentation ***
template = /etc/smokeping/basepage.html
htmltitle = yes
graphborders = no
+ charts
menu = Charts
title = The most interesting destinations
++ stddev
sorter = StdDev(entries=>10)
title = Top Standard Deviation
menu = Std Deviation
format = Standard Deviation %f
++ max
sorter = Max(entries=>10)
title = Top Max Roundtrip Time
menu = by Max
format = Max Roundtrip Time %f seconds
++ loss
sorter = Loss(entries=>10)
title = Top Packet Loss
menu = Loss
format = Packets Lost %f
++ median
sorter = Median(entries=>10)
title = Top Median Roundtrip Time
menu = by Median
format = Median RTT %f seconds
+ overview
width = 800
height = 100
range = 36h
+ detail
width = 1024
height = 256
unison_tolerance = 2
"Last 8 Hours Performance"8h
"Last 12 Hours Performance"12h
"Last 36 Hours Performance"36h
"Last 7 Days Performance"7d
"Last 14 Days Performance"14d
"Last 28 Days Performance"28d
"Last 100 Days Performance" 100d
#+ hierarchies
#++ owner
#title = Host Owner
#++ location
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On 2023-03-07, Glen Gunsalus wrote: > To get this running cp'd perl (/usr/bin/perl) and relevant perl libs > (/usr/lib/[libs.so|libm.so|libperl.so] /usr/libexec/ld.so) to > /var/www/usr/[bin|lib|libexec] You shouldn't need that bit (and it is safer not to) - smokeping_fcgi does not chroot.
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On 2023/03/07 16:49, Fabio Martins wrote: > On Tuesday, March 7, 2023, Claudio Jeker > > > > > > No need to collect flamegraphs, the issue is massive contention on the > > kernel lock because of high IO load. I see similar behaviour with iogen. > > Currently competing read and write calls clash with the async buffer > > handling which also requires the kernel lock to finish their work. So more > > concurrency makes it worse. Fixing this is a major task. Confirmed; closef, read, doopenat are prominent. https://junkpile.org/kstack-nms2030307.html > Can a ramdisk improve the performance while there are no changes in the > code? Not unless the existing storage is really slow; the issue is kernel contention not disk access speed. MFS doesn't help this.
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On 3/7/23 00:36, Stuart Henderson wrote:
On 2023/03/07 07:10, Tom Smyth wrote:
I m running smokeping fcgi and rrdcached ontop of OpenbSD, to smokeping
about 150 devces
the page load times can take 30 seconds to 1 minute,
is there any way to speed this up.
im running 7.2 OpenBSD on amd64 vm on top of an SSD array
any tips tricks welccome ...
One quick thing to try is updating to -current, I made some changes to
the rrdtool port which may possibly help a little.
Check that smokeping is actually using rrdcached (watch top while
opening a page) - the pkg-readme only gives instructions for passing the
required fastcgi variable through for nginx, I don't know how to do that
for httpd (or whether it's actually possible).
FWIW I have been running smokeping and mrtg with about 200 targets using httpd
for quite some time.
Smokeping with 1 master and 2 slaves. fping probes, config w default
concurrentprobes (i.e., commented out in config).
Rest of config down to ***Targets*** looks like Tom's.
not running rddcached
We're a small wireless (5GHz backbone, 2GHz APs) co-op on private 10.42.0.0
net; ping to most distance site is ~30msec, but can climb substantially with
lousy weather and client contention.
I found using IPs for "host" entries helped by eliminating DNS lookups but
don't recall how much difference it made.
It takes <15sec to generate new page (/var/db/smokeping/* and
/var/www/htdocs/smokeping/cache/*), top indicates smokeping using 1 core.
Hardware is pcEngines apu2 (4 core amd64, 4Gb mem) with 16Gb mSATA; OpenBSD 7.2 release
To get this running cp'd perl (/usr/bin/perl) and relevant perl libs
(/usr/lib/[libs.so|libm.so|libperl.so] /usr/libexec/ld.so) to
/var/www/usr/[bin|lib|libexec]
httpd.conf:
ext_ip="[redacted]" # Change this value to your vultr IP
server "default" {
listen on $ext_ip port 80
}
types {
include "/usr/share/misc/mime.types"
}
server "[redacted]" {
listen on $ext_ip port 80
location "/smokeping/smokeping.fcgi*" {
fastcgi socket "/run/smokeping.sock"
root "/"
}
}
start smokeping and smokeping.fcgi with rcctl, (same for mrtg)
Cheers, Glen
Other than that, rrdtool/rrdcached is just slow on OpenBSD. If it's
anything like mine you'll see high cpu spin % in top while it's busy.
You can try changing the number of cores in the VM - if you've given it
lots of cores try *reducing* it a bit. To pick a number out of the air
I'd suggest probably 4-6. (mine is bare metal and I can't drop the
number short of kernel hacks to set more cores offline).
You can hide the slowness at the loss of dynamic functionality in the
web interface by pre rendering the html/graphs from a cron job rather
than using the fastcgi (see the pkg-readme). But other than the above
I'm out of ideas to actually make it run faster.
(If anyone interested in poking at kernel locks would like flamegraphs
from my monitoring box - librenms/smokeping/icinga2/mariadb with lots
of rrdtool/rrdcached - let me know. Spin is pretty brutal.)
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
Inline On Tuesday, March 7, 2023, Claudio Jeker > > > No need to collect flamegraphs, the issue is massive contention on the > kernel lock because of high IO load. I see similar behaviour with iogen. > Currently competing read and write calls clash with the async buffer > handling which also requires the kernel lock to finish their work. So more > concurrency makes it worse. Fixing this is a major task. Can a ramdisk improve the performance while there are no changes in the code? > > -- > :wq Claudio > > -- Atenciosamente, Fabio Martins (+5521) 97914-8106 (Signal) https://www.linkedin.com/in/fabio1337br/
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
Hi Stuart,...
Im running 2 cores as Im a miser with my VMS in terms of CPU allocation
... ( I dont like spending time on the bare metal spliting cherries ) (more
context switches than work being done) ...
Got my system upgraded... thanks ... and fixed my old /usr/sbin/dig
(old..nolonger working) to /usr/bin/dig
the initial load seems to be quicker ... and opening a page seems to put
more load on rrdcached... process alright
the config below seems to get rrdcached working with httpd in OpenBSD. ...
the loading of the smokeping detailed graphs still takes a while ... but I
will do further dianostics...
This is my setup
#httpd.conf###
server "default" {
listen on * port 80
location "/smokeping/smokeping.fcgi*" {
fastcgi {
socket "/run/smokeping.sock"
param RRDCACHED_ADDRESS
"unix:/var/www/run/rrd/rrdcached.sock"
}
root "/"
}
###
top output below when loading a web page
load averages: 2.09, 1.82, 1.07 smoke1
14:36:27
42 processes: 40 idle, 2 on processor up 0
days 00:11:09
CPU0 states: 53.2% user, 0.0% nice, 6.8% sys, 0.2% spin, 2.2% intr,
37.6% idle
CPU1 states: 33.1% user, 0.0% nice, 10.6% sys, 1.4% spin, 0.0% intr,
54.9% idle
Memory: Real: 208M/1758M act/tot Free: 6160M Cache: 882M Swap: 0K/0K
PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND
57245 _rrdcach 20 41M 36M onproc/0 kqread3:13 30.62%
rrdcached
99560 _smokepi 20 74M 88M sleep/1 netio 1:01 10.55% perl
73953 _smokepi 20 2632K 2660K sleep/0 kqread0:00 0.20% fping
77717 _smokepi 100 42M 20M sleep/0 nanoslp 0:02 0.00% perl
1 root 100 644K 628K idle wait 0:01 0.00% init
67291 _smokepi -60 42M 20M idle piperd0:01 0.00% perl
72553 root 30 948K 924K idle ttyin 0:00 0.00% ksh
84133 _pflogd40 776K 1620K sleep/0 bpf 0:00 0.00% pflogd
74456 _smtpq 20 1656K 3484K idle kqread0:00 0.00% smtpd
58541 _ntp 2 -20 1408K 3320K sleep/1 kqread0:00 0.00% ntpd
22630 root 20 1204K 4160K idle kqread0:00 0.00% sshd
20724 www20 1908K 3944K sleep/1 kqread0:00 0.00% httpd
27618 www20 2256K 4276K idle kqread0:00 0.00% httpd
81375 _syslogd 20 1228K 1524K idle kqread0:00 0.00% syslogd
77400 _smokepi 180 42M 10M idle sigsusp 0:00 0.00% perl
39827 root 280 1224K 2512K onproc/1 - 0:00 0.00% top
79586 _smtpd 20 1936K 4828K idle kqread0:00 0.00% smtpd
18799 fireman20 1396K 3340K sleep/0 kqread0:00 0.00% sshd
20179 www20 1320K 3324K idle kqread0:00 0.00% httpd
45288 root 180 944K 916K idle sigsusp 0:00 0.00% ksh
51902 root 20 760K 2548K idle netio 0:00 0.00% syslogd
37356 www20 1332K 3180K idle kqread0:00 0.00% httpd
82428 root 20 1472K 2284K idle kqread0:00 0.00% httpd
62829 _ntp 20 908K 2772K idle kqread0:00 0.00% ntpd
89278 root 20 872K 1524K idle kqread0:00 0.00% cron
16265 _smtpd 20 1652K 3472K idle kqread0:00 0.00% smtpd
46732 _smtpd 20 1456K 3304K idle kqread0:00 0.00% smtpd
3405 root 2 -20 1264K 1956K idle kqread0:00 0.00% ntpd
30532 root 20 1716K 2164K idle kqread0:00 0.00% smtpd
On Tue, 7 Mar 2023 at 08:36, Stuart Henderson wrote:
> On 2023/03/07 07:10, Tom Smyth wrote:
> > I m running smokeping fcgi and rrdcached ontop of OpenbSD, to smokeping
> > about 150 devces
> > the page load times can take 30 seconds to 1 minute,
> > is there any way to speed this up.
> >
> > im running 7.2 OpenBSD on amd64 vm on top of an SSD array
> >
> > any tips tricks welccome ...
>
> One quick thing to try is updating to -current, I made some changes to
> the rrdtool port which may possibly help a little.
>
> Check that smokeping is actually using rrdcached (watch top while
> opening a page) - the pkg-readme only gives instructions for passing the
> required fastcgi variable through for nginx, I don't know how to do that
> for httpd (or whether it's actually possible).
>
> Other than that, rrdtool/rrdcached is just slow on OpenBSD. If it's
> anything like mine you'll see high cpu spin % in top while it's busy.
> You can try changing the number of cores in the VM - if you've given it
> lots of cores try *reducing* it a bit. To pick a number out of the air
> I'd suggest probably 4-6. (mine is bare metal and I can't drop the
> number short of kernel hacks to set more cores offline).
>
> You can hide the slowness at the loss of dynamic functionality in the
> web interface by pre rendering the
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On Tue, Mar 07, 2023 at 08:36:24AM +, Stuart Henderson wrote: > On 2023/03/07 07:10, Tom Smyth wrote: > > I m running smokeping fcgi and rrdcached ontop of OpenbSD, to smokeping > > about 150 devces > > the page load times can take 30 seconds to 1 minute, > > is there any way to speed this up. > > > > im running 7.2 OpenBSD on amd64 vm on top of an SSD array > > > > any tips tricks welccome ... > > One quick thing to try is updating to -current, I made some changes to > the rrdtool port which may possibly help a little. > > Check that smokeping is actually using rrdcached (watch top while > opening a page) - the pkg-readme only gives instructions for passing the > required fastcgi variable through for nginx, I don't know how to do that > for httpd (or whether it's actually possible). > > Other than that, rrdtool/rrdcached is just slow on OpenBSD. If it's > anything like mine you'll see high cpu spin % in top while it's busy. > You can try changing the number of cores in the VM - if you've given it > lots of cores try *reducing* it a bit. To pick a number out of the air > I'd suggest probably 4-6. (mine is bare metal and I can't drop the > number short of kernel hacks to set more cores offline). > > You can hide the slowness at the loss of dynamic functionality in the > web interface by pre rendering the html/graphs from a cron job rather > than using the fastcgi (see the pkg-readme). But other than the above > I'm out of ideas to actually make it run faster. > > (If anyone interested in poking at kernel locks would like flamegraphs > from my monitoring box - librenms/smokeping/icinga2/mariadb with lots > of rrdtool/rrdcached - let me know. Spin is pretty brutal.) No need to collect flamegraphs, the issue is massive contention on the kernel lock because of high IO load. I see similar behaviour with iogen. Currently competing read and write calls clash with the async buffer handling which also requires the kernel lock to finish their work. So more concurrency makes it worse. Fixing this is a major task. -- :wq Claudio
Re: Folks are there any tips to improve page load times on smokeping running on OpenBSD
On 2023/03/07 07:10, Tom Smyth wrote: > I m running smokeping fcgi and rrdcached ontop of OpenbSD, to smokeping > about 150 devces > the page load times can take 30 seconds to 1 minute, > is there any way to speed this up. > > im running 7.2 OpenBSD on amd64 vm on top of an SSD array > > any tips tricks welccome ... One quick thing to try is updating to -current, I made some changes to the rrdtool port which may possibly help a little. Check that smokeping is actually using rrdcached (watch top while opening a page) - the pkg-readme only gives instructions for passing the required fastcgi variable through for nginx, I don't know how to do that for httpd (or whether it's actually possible). Other than that, rrdtool/rrdcached is just slow on OpenBSD. If it's anything like mine you'll see high cpu spin % in top while it's busy. You can try changing the number of cores in the VM - if you've given it lots of cores try *reducing* it a bit. To pick a number out of the air I'd suggest probably 4-6. (mine is bare metal and I can't drop the number short of kernel hacks to set more cores offline). You can hide the slowness at the loss of dynamic functionality in the web interface by pre rendering the html/graphs from a cron job rather than using the fastcgi (see the pkg-readme). But other than the above I'm out of ideas to actually make it run faster. (If anyone interested in poking at kernel locks would like flamegraphs from my monitoring box - librenms/smokeping/icinga2/mariadb with lots of rrdtool/rrdcached - let me know. Spin is pretty brutal.)
Folks are there any tips to improve page load times on smokeping running on OpenBSD
Folks, I m running smokeping fcgi and rrdcached ontop of OpenbSD, to smokeping about 150 devces the page load times can take 30 seconds to 1 minute, is there any way to speed this up. im running 7.2 OpenBSD on amd64 vm on top of an SSD array any tips tricks welccome ... ###etc-smokeping-config### smoke1$ cat /etc/smokeping/config *** General *** owner= Wireless Connect Managed Networks contact = [Redacted] @wirelessconnect.eu mailhost = [redacted] #sendmail = /usr/sbin/sendmail # NOTE: do not put the Image Cache below cgi-bin # since all files under cgi-bin will be executed ... this is not # good for images. imgcache = /var/www/htdocs/smokeping/cache imgurl = cache datadir = /var/db/smokeping piddir = /var/run/smokeping cgiurl = http://[redacted]/smokeping/smokeping.fcgi smokemail = /etc/smokeping/smokemail tmail = /etc/smokeping/tmail # specify this to get syslog logging syslogfacility = local0 syslogpriority = debug # each probe is now run in its own process # disable this to revert to the old behaviour concurrentprobes = yes *** Alerts *** to = [redacted] @ wirelessconnect.eu from = [reacted] @wirelessconnect.eu #mailtemplate = /var/somewhere/mailtemplate +Sustained_5%_loss type = loss # in percent edgetrigger = yes pattern = >=5%,>=5%,>=5% comment = Sustained Packet Loss more than 5% +Sudden_10%_Loss type = loss # in percent edgetrigger = yes pattern = ==0%,==0%,>10% comment = Single Incident Packet Loss more than 10% Detected! +Sudden_20%_Loss type= loss # in percent edgetrigger = yes pattern = ==0%,==0%,>20% comment = Single Incident Packet Loss more than 20% Detected! +Sporadic_Loss type = loss # in percent pattern = ==0%,==0%,*8*,*8*,>20%,*8*,>20%,>20%,*8*,>0% comment = High Packet loss occured at least 4 times over the last two hours #+anydelay #type = rtt # # in milliseconds # pattern = >1 # comment = Just for testing +Latency_Over_50ms type = rtt # in milliseconds edgetrigger = yes pattern = <45,<45,<45,<45,<45,<45,>50 comment = Sudden High Latency greater than 50ms! +Latency_Over_200ms type = rtt # in milliseconds edgetrigger = yes pattern = <100,<100,>200 comment = Sudden High Latency greater than 200ms! +Latency_Constantly_Over_100ms type = rtt # in milliseconds edgetrigger = yes pattern = >100,>100,>100,>100,>100 comment = Constant High Latency greater than 100 ms for over 5 samples in a row +Offline_at_startup type = rtt # in milliseconds edgetrigger = yes pattern = ==S,==U comment = System offline at startup! *** Database *** step = 90 pings= 60 # consfn mrhb steps total AVERAGE 0.5 1 1008 AVERAGE 0.5 12 4320 MIN 0.5 12 4320 MAX 0.5 12 4320 AVERAGE 0.5 144 720 MAX 0.5 144 720 MIN 0.5 144 720 *** Presentation *** template = /etc/smokeping/basepage.html htmltitle = yes graphborders = no + charts menu = Charts title = The most interesting destinations ++ stddev sorter = StdDev(entries=>10) title = Top Standard Deviation menu = Std Deviation format = Standard Deviation %f ++ max sorter = Max(entries=>10) title = Top Max Roundtrip Time menu = by Max format = Max Roundtrip Time %f seconds ++ loss sorter = Loss(entries=>10) title = Top Packet Loss menu = Loss format = Packets Lost %f ++ median sorter = Median(entries=>10) title = Top Median Roundtrip Time menu = by Median format = Median RTT %f seconds + overview width = 800 height = 100 range = 36h + detail width = 1024 height = 256 unison_tolerance = 2 "Last 8 Hours Performance"8h "Last 12 Hours Performance"12h "Last 36 Hours Performance"36h "Last 7 Days Performance"7d "Last 14 Days Performance"14d "Last 28 Days Performance"28d "Last 100 Days Performance" 100d #+ hierarchies #++ owner #title = Host Owner #++ location #title = Location *** Probes *** #+basefork #forks = 8 #offset = 50% #step = 90 #timeout = 1 + FPing blazemode = true binary = /usr/local/sbin/fping packetsize = 64 hostinterval = 0.001 timeout = 0.125 offset = random + DNS binary = /usr/sbin/dig # mandatory forks = 5 offset = 50% step = 30 timeout = 15 # The following variables can be overridden in each target section lookup = bbc.co.uk pings = 30 server = [redacted] + TCPPing binary = /usr/local/sbin/hping forks = 5 offset = 50% step = 300 timeout =10 #*** Slaves *** #secrets=/etc/smokeping/smokeping_secrets #+boomer #display_name=boomer #color=ff #+slave2 #display_name=another #color=00ff00 *** Targets *** probe = FPing menu = Top title = Wireless Connect Network Latency Grapher remark = SmokePing of Wireless Connect Ltd. \ This Tool Shows the latency of the \ Wireless Connectnetwork. alerts = Sustained_5%_loss,Sudden_10%_Loss,Sporadic_Loss,Latency_Over_50ms,Offline_at_startup #config-sniped# smoke1# rcctl ls started cron dhcpleased httpd ntpd pflogd resolvd rrdcached smokeping smokeping_fcgi smtpd sshd syslogd --
