[BlueOnyx:25875] Re: Real Time access analysis (and better stats)
Hi Juerg, chown admserv:admserv -R /var/lib/monitorix/www/imgs/ I wonder what other black magic I'll have to conjure to make it stick. :p Could it be, that the rpm post script runs, BEFORE the admserv user is created? By 1'000 packages... I removed the monitorix rpm manually and installed again and then permissions are ok. Ah, you did a fresh install? Hmmm ... yeah, then it is probably an RPM installation order issue and Monitorix got installed before the user and group admserv were present. That's ... unpretty. With the OS included we're at about 1800-1900 RPMs (if installed off the ISO) and dependencies are getting somewhat complicated. I just published a YUM update for Monitorix that fixes it for good - no matter what. An ExecPre in the Systemd Unit file now makes sure that permissions and ownerships are corrected whenever Monitorix starts. I could throw in some RPM "Requires" to try to enforce a better sort order, but as it is we already have some funky circular dependencies and I don't want to push them to the point where the install fails due to that. -- With best regards Michael Stauber ___ Blueonyx mailing list Blueonyx@mail.blueonyx.it http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25874] Re: Real Time access analysis (and better stats)
Hi Juerg,
I guess it'll work. Until you eventually try to shove several
gigabytes of logfiles through it to parse. :o)
I agree, large logfiles are a problem with variables...
The problem is line ~170 (I modified the script)
system("$GoAccess_cmd $go_access_params");
This is called BEFORE loop in ~270:
while (my $line = ) {
GoAccess grabs the STDIN run on command line (both way "cat xy | script"
or "script < xy")
Ah, ok. I see your point. And it's a good point. But I still don't want
to stuff the whole Apache access_log into a variable as that might blow
up in our face if the logfile snippet is large. And that might easily
happen.
So how about this:
https://devel.blueonyx.it/trac/changeset/4464/
In that change I moved the whole (first) GoAccess parsing block behind
any STDIN processing and that runs off the (copied) logfile itself
anyway and doesn't need STDIN anyway.
I think that should work better.
--
With best regards
Michael Stauber
___
Blueonyx mailing list
Blueonyx@mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25873] Re: DKIM , SPF und DMARC on BlueOnyx
hello Chris, Neal & Michael It's a while ago, I had this problem Chris mentioned — providers, e.g. gmail, tagged e-mails coming from my servers as spam and users complained that their mails weren't delivered anymore or landed in the Junk folder... So tried to find a remedy, asked the list in July and Michael installed then OpenDKIM. Chris, your guide to install DKIM would have been very helpful at that time, I had to figure it all out the hard way... First generating the key and then inserting the TXT record into the DNS. And I wanted to do this for each domain. Then it occurred to me that the SPF *and* the DMARC TXT records must also be entered into the DNS. Whether DKIM, SPF and DMARK are charlatan products is an open question, but installing all three «authentication techniques» significantly reduced spam tagging and undelivered mail on my servers. So my message: if you have a bunch of domains without those implementations, then do the following (as I did): • create all DKIM keys • prepare the DMARC TXT record for each domain (see NOTE 1) • prepare the SPF TXT record for each domain (see NOTE 2) Then you can switch to your DNS server and insert the three TXT records. It's some kind of a «/Das tapfere Schneiderlein/» (The Valiant Little Tailor) but with only tree flies:-) You'll have to do this one by one, unfortunately (I created a shell script that does this partially, see NOTE3) • generate all DKIM keys for each domain into /etc/opendkim/keys → Steps 1 to 4 in Chris' guide • Step 5 is important: chown -R opendkim:opendkim /etc/opendkim (that was a tough one to find out:-) • Step 6 and 7 Then you can switch to your DNS server for Step 8. Browse thru each domain in [Select Domain... v] and add the three TXT records • _dmarc . yourdomain.tld TXT v=DMARC1; p=quarantine; rua=mailto:rep...@yourdomain.tld; ruf=mailto:rep...@yourdomain.tld • yourdomain.tld. TXT v=spf1 ip4:ip.ip.ip.ip1/32 ip4:ip.ip.ip.ip2/32 include:_spf.google.com include:_spf.bluewin.ch ~all • default._domainkey . yourdomain.tld TXT v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDeQBM3pni6EN9A3+N47x10tiRHe3KUM4ciXUMBD9gABcv/dnpRQfdOXZOG1A8WrvwoKXywYIDv4MCyuBXgCHMppjkQ703lc8eKjuTZxGLheiQGQ/ISmTndbM2y+SG9tv+YvD9YwpVNLTuUJung3XpHeoiOXLr0HX8TfQPzG04hDQIDAQAB Save the record, then save again for the domain, and when you went through all domains, restart the DNS server. Goto Step 9 and test the DNS record using https://www.dmarcanalyzer.com/dkim/dkim-checker/ I noted it might take a while to get an ok, probably because of the DNS propagation. Best regards で⊃ Meaulnes Legler Zurich, Switzerland +41¦0 44 260-1660 I'm on *Wire* as @meaulnes — https://get.wire.com/ /no more Whatzap and so on!/ NOTE1 I'm not sure which arguments are the best, these rua= and ruf= addresses create errors, but they don't harm.. NOTE2 Also here I'm not sure: a? mx? ip4? Intuitively, I included _spf.google.com and _spf.bluewin.ch, a major telecom provider NOTE3 My shell script checks OpenDKIM and Postfix configuration and lists all virtual servers, the ones with already installed DKIM keyfiles and and the ones without: # ~/dkim_addDomain.sh OpenDKIM and Postfix configuration ok. ERROR: no domain specified to DKIM! dkim_addDomain.sh version 3 (9.2022) - Install DKIM record for a virtual domain. usage: /root/dkim_addDomain.sh domain.tld or sub.domain.tld List of 32 available domains on this server: ... List of 17 already installed domains with keyfiles: ... List of 15 domains that can be installed: ... If someone wants it, write me directly @ i...@waveweb.ch. As Chris points it out: Keep in mind all the usual disclaimers, it's made available as a courtesy, not guaranteed to work for your production use, etc etc blah blah:-) On 29.12.22 05:05, Chris Gebhardt - VIRTBIZ Internet wrote: Hi Michael, On 12/28/22 7:50 PM, Michael Stauber wrote: All that out of the way, here's the guide for adding DKIM to a BlueOnyx VSITE: https://www.virtbiz.com/client/index.php?rp=/knowledgebase/4996/Add-DKIM-for-BlueOnyx-VSITE.html Ah, you know what? I guess it's not *that* much work, so I think I'll build it into the DNS GUI. I'll throw OpenDKIM in as mandatory RPM and provide the GUI to create/manage the keys and TXT DNS records. And instantly make my guide obsolete? Wh! LOL, not really. I think that's a great solution if it's easy enough to integrate, much like the SPF generator but a step beyond since it will have to handle the key integration. I presume that will be something for 5211R and possibly backported to 5210R. If that's the case, I'll keep the guide active for those who want to run OpenDKIM for VSITEs on a 5209R, since those will still be knocking around for a while. When the feature is released, I'll update my KB entry noting the obsolescence. ___ Blueonyx mailing list
[BlueOnyx:25872] Re: Real Time access analysis (and better stats)
Hi Michael,
That may be one way to do it. As you can see in
/etc/logrotate.d/apache it calls split_logs this way:
https://devel.blueonyx.it/trac/browser/BlueOnyx/5211R/ui/base-sitestats.mod/src/sitestats-scripts/apache.logrotate
/usr/local/sbin/split_logs web < /tmp/.logrotate_apache_access 2>
/dev/null || true
So we're telling split_logs that this is a 'web' log and we're piping
/tmp/.logrotate_apache_access through it. This may be a frigging large
file, so that's why it's not shoved into a variable, but directly
processed by slit_logs.
Once *that* is done and we have the Server's web.log, we simply call
GoAccess to do it's deeds, because at that point we don't have to
worry about STDIN anymore, as there will be nothing else delivered by
STDIN anymore.
I have to test it if it's also work with logrotate.
I guess it'll work. Until you eventually try to shove several
gigabytes of logfiles through it to parse. :o)
I agree, large logfiles are a problem with variables...
The problem is line ~170 (I modified the script)
system("$GoAccess_cmd $go_access_params");
This is called BEFORE loop in ~270:
while (my $line = ) {
GoAccess grabs the STDIN run on command line (both way "cat xy | script"
or "script < xy")
SAMPLE:
#!/usr/bin/perl
#system("/usr/bin/xargs /usr/bin/echo YY ");
my @stdin = ;
#system("/usr/bin/xargs /usr/bin/echo YY ");
while ($line = shift(@stdin)) {
print "XX:".$line;
}
If you activate only the first system(), output is
[root@blueonyx9 admin]# echo test |./stdin.pl
YY test
(system grabs stdin, loop is empty)
If you activate only the second system(), output is
[root@blueonyx9 admin]# echo test |./stdin.pl
YY
XX:test
(@stdin grabs the input and system is empty)
So you have to call GoAccess AFTER parsing STDIN or backup STDIN in a
variable. It works when called in logrotate, but I don't know how,
GoAccess or perl must be handle different, when they are not executed on
an interactive shell. I don't like any script which reacts different
when they are called manually or by server daemon. And somewhere in the
future, the STDIN is maybe also grabbed when executed on logrotate...
Regards,
Juerg
___
Blueonyx mailing list
Blueonyx@mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25871] Re: Real Time access analysis (and better stats)
Hi Michael,
Should web.log have data or would it removed after statistic is
processed? I have to debug the process, when I migrated a site with
more traffic
See /etc/logrotate.d/siteX: During the course of the whole logrotate
process the Vsite web.logs's are rotated out entirely. This happens
way after split_logs.
Yes, I have seen and understood by now
/var/lib/monitorix/www/imgs should be owned by admserv:admserv and
should have 0755 permissions. Yours doesn't? The "monitorix" RPM
should set it in the POST install section.
In installed a fresh vm. And no, the permissions are wrong:
[root@blueonyx9 ~]# ls -lsa /var/lib/monitorix/www/imgs
insgesamt 8
4 drwxr-xr-x 2 root root 4096 27. Dez 17:52 .
4 drwxr-xr-x 5 root root 4096 29. Dez 10:34 ..
IMHO it was correct before the "perl-Config-General" update, but I'm
not sure.
I've seen this a couple of times now and it doesn't make sense. The
Monitorix RPM Specfile has this in it's %files section:
%attr(755,admserv,admserv) %{_localstatedir}/lib/%{name}/www/imgs
ADDITIONALLY it has this in the %post section that runs during
post-install of the RPM:
chown admserv:admserv -R /var/lib/monitorix/www/imgs/
I wonder what other black magic I'll have to conjure to make it stick. :p
Could it be, that the rpm post script runs, BEFORE the admserv user is
created? By 1'000 packages... I removed the monitorix rpm manually and
installed again and then permissions are ok. Only an idea, I don't know
in detail in which order post scripts are executed and when admserv is
created...
Regards,
Juerg
___
Blueonyx mailing list
Blueonyx@mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25870] Re: Real Time access analysis (and better stats)
Hi Juerg,
> My unterstanding of split_logs is now better than before. I'll
> check if array for STDIN also works in logrotate-environment.
> If so, I would change the script for me: I don't like any scripts
> I can't run manually... :) I'll give you feedback tomorrow if
> script change also worked in logrotate.
This "split_log" script? We inherited it from the BlueQuartz sources and
it may already have been there during the Cobalt days. It's in a coding
format and principle I don't like very much either, but short of a
complete rewrite there isn't much that can be done about it.
When you start split_logs in the command line,
system("$GoAccess_cmd $go_access_params");
clears the STDIN. If it's run in logrotate I think STDIN keeps alive,
else the file wouldn't be created. But maybe it would be good to backup
STDIN in script start (near variable declaration) with
my @stdin;
while () {
push(@stdin, $_);
}
and then loop trough the array instead of STDIN
while (my $line = ) {
while (my $line = shift(@stdin)) {
That may be one way to do it. As you can see in /etc/logrotate.d/apache
it calls split_logs this way:
https://devel.blueonyx.it/trac/browser/BlueOnyx/5211R/ui/base-sitestats.mod/src/sitestats-scripts/apache.logrotate
/usr/local/sbin/split_logs web < /tmp/.logrotate_apache_access 2>
/dev/null || true
So we're telling split_logs that this is a 'web' log and we're piping
/tmp/.logrotate_apache_access through it. This may be a frigging large
file, so that's why it's not shoved into a variable, but directly
processed by slit_logs.
Once *that* is done and we have the Server's web.log, we simply call
GoAccess to do it's deeds, because at that point we don't have to worry
about STDIN anymore, as there will be nothing else delivered by STDIN
anymore.
I have to test it if it's also work with logrotate.
I guess it'll work. Until you eventually try to shove several gigabytes
of logfiles through it to parse. :o)
And then I saw, you create a /etc/logrotate.d/siteX script for each
vsite which clears the web.log, so this is ok, that there's a 0 Byte
file (it's ok, the complete server logs in /var/logs/https are rotated,
so we don't need it).
That's fine. By the time web.log is rotated and set to 0 all relevant
data has been extracted by it already. But for testing purpose you can
comment out or temporarily remove the /etc/logrotate.d/siteX files.
--
With best regards
Michael Stauber
___
Blueonyx mailing list
Blueonyx@mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25869] Re: Real Time access analysis (and better stats)
Hi Juerg,
Should web.log have data or would it removed after statistic is
processed? I have to debug the process, when I migrated a site with more
traffic
See /etc/logrotate.d/siteX: During the course of the whole logrotate
process the Vsite web.logs's are rotated out entirely. This happens way
after split_logs.
/var/lib/monitorix/www/imgs should be owned by admserv:admserv and
should have 0755 permissions. Yours doesn't? The "monitorix" RPM
should set it in the POST install section.
In installed a fresh vm. And no, the permissions are wrong:
[root@blueonyx9 ~]# ls -lsa /var/lib/monitorix/www/imgs
insgesamt 8
4 drwxr-xr-x 2 root root 4096 27. Dez 17:52 .
4 drwxr-xr-x 5 root root 4096 29. Dez 10:34 ..
IMHO it was correct before the "perl-Config-General" update, but I'm not
sure.
I've seen this a couple of times now and it doesn't make sense. The
Monitorix RPM Specfile has this in it's %files section:
%attr(755,admserv,admserv) %{_localstatedir}/lib/%{name}/www/imgs
ADDITIONALLY it has this in the %post section that runs during
post-install of the RPM:
chown admserv:admserv -R /var/lib/monitorix/www/imgs/
I wonder what other black magic I'll have to conjure to make it stick. :p
--
With best regards
Michael Stauber
___
Blueonyx mailing list
Blueonyx@mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25868] Re: Mass SPF tool
Hello Meaulnes and Neal, for i in $(ls -1 /home/sites | sed 's/www.//g'); do echo $i ; dig $i txt | grep spf ; done Thank you for the tip. While it's not quite the automated tool that will then insert a fully-formed SPF record into a domain, it does work very well as a live-assist tool. It's greatly sped up the process of proactively adding SPF records rather than wait and handle on a break-fix basis. Happy New Year! -- Chris Gebhardt VIRTBIZ Internet Services Access, Web Hosting, Colocation, Dedicated www.virtbiz.com | toll-free (866) 4 VIRTBIZ ___ Blueonyx mailing list Blueonyx@mail.blueonyx.it http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25867] Re: Real Time access analysis (and better stats)
Hi Michael
I debugged the split_logs script, not quite easy.
> Yeah, it worked now. Was a special environment: I tested it with a vm
> and only log-files from today. Running split_logs manually created a
> folder for 2022/12/27 (yesterday, normally wanted), but in this folder
> were only records for today (which would be filtered, I think).
Yes, this is intentional. When logrotate runs we create a copy of the
log and work through it. The results go in a folder with yesterdays
date, as "todays" data is still in the live log.
If logrotate doesn't run at midnight we might end up with some data
from yesterday and today in it, but each dated folder after a while
will have consistently data that was aggregated until the logrotate
kicked in. So I don't consider this an issue.
I checked the logs today after automatic logrotate. The files are
created (and I don't know why), but it's strange. I have to less
traffic on the test-site to debug it (only manual 5 hits), but
web.log is a null byte file (also on other older installations)
there's a valid json file in /home/sites/{sitename}/var/logs/2022/12/28
in GUI I can only choose 2022/12/29, which shows no results (correct,
no json). 2022/12/28 is not clickable.
Should web.log have data or would it removed after statistic is
processed? I have to debug the process, when I migrated a site with
more traffic
When you start split_logs in the command line,
system("$GoAccess_cmd $go_access_params");
clears the STDIN. If it's run in logrotate I think STDIN keeps alive,
else the file wouldn't be created. But maybe it would be good to backup
STDIN in script start (near variable declaration) with
my @stdin;
while () {
push(@stdin, $_);
}
and then loop trough the array instead of STDIN
while (my $line = ) {
while (my $line = shift(@stdin)) {
I have to test it if it's also work with logrotate.
And then I saw, you create a /etc/logrotate.d/siteX script for each
vsite which clears the web.log, so this is ok, that there's a 0 Byte
file (it's ok, the complete server logs in /var/logs/https are rotated,
so we don't need it).
My unterstanding of split_logs is now better than before. I'll check if
array for STDIN also works in logrotate-environment. If so, I would
change the script for me: I don't like any scripts I can't run
manually... :) I'll give you feedback tomorrow if script change also
worked in logrotate.
Regards,
Juerg
___
Blueonyx mailing list
Blueonyx@mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx
[BlueOnyx:25866] Re: Real Time access analysis (and better stats)
Hi Michael,
> Yeah, it worked now. Was a special environment: I tested it with a vm
> and only log-files from today. Running split_logs manually created a
> folder for 2022/12/27 (yesterday, normally wanted), but in this folder
> were only records for today (which would be filtered, I think).
Yes, this is intentional. When logrotate runs we create a copy of the
log and work through it. The results go in a folder with yesterdays
date, as "todays" data is still in the live log.
If logrotate doesn't run at midnight we might end up with some data
from yesterday and today in it, but each dated folder after a while
will have consistently data that was aggregated until the logrotate
kicked in. So I don't consider this an issue.
I checked the logs today after automatic logrotate. The files are
created (and I don't know why), but it's strange. I have to less traffic
on the test-site to debug it (only manual 5 hits), but
web.log is a null byte file (also on other older installations)
there's a valid json file in /home/sites/{sitename}/var/logs/2022/12/28
in GUI I can only choose 2022/12/29, which shows no results (correct, no
json). 2022/12/28 is not clickable.
Should web.log have data or would it removed after statistic is
processed? I have to debug the process, when I migrated a site with more
traffic
I checked it again on a clean vm. The rpm package is installed now,
but the images would not be show, because /var/lib/monitorix/www/imgs
has no write access. Which process creates the images? monitorix runs
as root, the admserv user has no write access. Maybe it's an error on
my side, but it occurs on two test vms.
/var/lib/monitorix/www/imgs should be owned by admserv:admserv and
should have 0755 permissions. Yours doesn't? The "monitorix" RPM
should set it in the POST install section.
In installed a fresh vm. And no, the permissions are wrong:
[root@blueonyx9 ~]# ls -lsa /var/lib/monitorix/www/imgs
insgesamt 8
4 drwxr-xr-x 2 root root 4096 27. Dez 17:52 .
4 drwxr-xr-x 5 root root 4096 29. Dez 10:34 ..
IMHO it was correct before the "perl-Config-General" update, but I'm not
sure.
Regards,
Juerg
___
Blueonyx mailing list
Blueonyx@mail.blueonyx.it
http://mail.blueonyx.it/mailman/listinfo/blueonyx
