Re: pf anchors attached to irrelevant states
On 19/05/2024 19:35, Kapetanakis Giannis wrote: > On 19/05/2024 14:37, Stuart Henderson wrote: >> On 2024-05-19, Kapetanakis Giannis wrote: >>> This is a bit strange. pf works normal, but rules after an enchor an >>> being attached to the anchor (somehow). >>> >>> All states that are created from rules after the anchor, show the anchor >>> (pf rule) number instead of (only) the rule number in pfctl -vv and in >>> pflog. >> I can confirm this is a problem, definitely seen in 7.4, I can't remember >> if 7.3 was affected. 7.2 from Dec 22 seems ok. > > 7.3 release was also affected, just tested on a vm. > > G It seems that this was introduced with 1.1169 of pf.c (2023/01/05) https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c.diff?r1=1.1168=1.1169=date=h reverting to 1.1168 shows then rules numbers correct both in pflog and pfctl. Rest of kernel is on 2023-01-12 G
packet filter silently ignores a rule
Hello,
I use OpenBSD 7.5 stable amd64.
I uncommented an old rule and the corresponding macro in pf.conf
which definitely worked when the
machine was on version 7.3 and possibly 7.4.
After that:
pfctl -nf /etc/pf.conf shows nothing
pfctl -f /etc/pf.conf shows nothing
So Packet Filter seems to be happy with the config as a whole.
pfctl -vvsr shows the old rules WITHOUT the uncommented one.
pfctl -vvnf /etc/pf.conf warns that the uncommented macro
used in the uncommented rule is NOT used.
The output of pfctl -vvnf /etc/pf.conf is appended as
pfctl_vvnf file
The output of pfctl -vvsr is appended as
pfctl_vvsr file
Did I miss something when changing the configuration?
The uncommented section 1 is:
mail_ports = "{ submission imaps }"
The uncommented section 2 is:
pass in on egress inet proto tcp to (egress) \
port $mail_ports \
keep state (max-src-conn 20, \
max-src-conn-rate 35/300, overload \
flush global) rdr-to $mail_server
My whole pf.conf (all uncommented lines):
int_if = "{ vether1 em1 em2 em3 }"
table { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 \
169.254.0.0/16 172.16.0.0/12 192.0.2.0/24 \
192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 \
}
table persist
table persist file "/etc/mail/nospamd"
table persist file "/etc/pf/bad_ips"
transmission_server = "192.168.1.65"
mail_server = "192.168.1.171"
mail_ports = "{ submission imaps }"
block log all
set limit table-entries 100
set block-policy drop
set syncookies adaptive (start 29%, end 15%)
set skip on lo
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from (vether1:network) \
to any nat-to (egress:0)
block in quick on egress from to any
block return out quick on egress from any to
block quick from
pass out quick inet
pass in on $int_if inet
pass in on egress inet proto tcp \
to (egress) port 22 keep state \
(max-src-conn 2, max-src-conn-rate 2/300, \
overload flush global)
pass in on egress inet proto { tcp udp } \
to (egress) port domain keep state \
(max-src-states 10) \
rdr-to 127.0.0.1 port 8053
pass in on $int_if inet proto { tcp udp } from \
(vether1:network) to (egress) port domain
pass in on egress inet proto { tcp udp } \
to (egress) port 5 \
rdr-to $transmission_server
pass in on egress inet proto tcp to (egress) \
port $mail_ports \
keep state (max-src-conn 20, \
max-src-conn-rate 35/300, overload \
flush global) rdr-to $mail_server
pass in on egress proto tcp to (egress) \
port smtp divert-to 127.0.0.1 port spamd
pass in on egress proto tcp from to (egress) \
port smtp rdr-to $mail_server
pass in log on egress proto tcp from \
to (egress) port smtp \
rdr-to $mail_server
pass out on egress proto tcp to (egress) port smtp
--
Best regards
Maksim Rodin
warning: macro 'mail_ports' not used
Loaded 714 passive OS fingerprints
int_if = "{ vether1 em1 em2 em3 }"
table { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16
172.16.0.0/12 192.0.2.0/24 192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 }
table persist
table persist file "/etc/mail/nospamd"
table persist file "/etc/pf/bad_ips"
transmission_server = "192.168.1.65"
mail_server = "192.168.1.171"
mail_ports = "{ submission imaps }"
set limit table-entries 100
set block-policy drop
set syncookies adaptive (start 29%, end 15%)
set skip on { lo }
@0 block drop log all
@1 match in all scrub (no-df random-id max-mss 1440)
@2 match out on egress inet from (vether1:network:*) to any nat-to (egress:0:*)
round-robin
@3 block drop in quick on egress from to any
@4 block return out quick on egress from any to
@5 block drop quick from to any
@6 pass out quick inet all flags S/SA
@7 pass in on vether1 inet all flags S/SA
@8 pass in on em1 inet all flags S/SA
@9 pass in on em2 inet all flags S/SA
@10 pass in on em3 inet all flags S/SA
@11 pass in on egress inet proto tcp from any to (egress:*) port = 22 flags
S/SA keep state (source-track rule, max-src-conn 2, max-src-conn-rate 2/300,
overload flush global, src.track 300)
@12 pass in on egress inet proto tcp from any to (egress:*) port = 53 flags
S/SA keep state (source-track global, max-src-states 10) rdr-to 127.0.0.1 port
8053
@13 pass in on egress inet proto udp from any to (egress:*) port = 53 keep
state (source-track global, max-src-states 10) rdr-to 127.0.0.1 port 8053
@14 pass in on vether1 inet proto tcp from (vether1:network:*) to (egress:*)
port = 53 flags S/SA
@15 pass in on em1 inet proto tcp from (vether1:network:*) to (egress:*) port =
53 flags S/SA
@16 pass in on em2 inet proto tcp from (vether1:network:*) to (egress:*) port =
53 flags S/SA
@17 pass in on em3 inet proto tcp from (vether1:network:*) to (egress:*) port =
53 flags S/SA
@18 pass in on vether1 inet proto udp from (vether1:network:*) to (egress:*)
port = 53
@19 pass in on em1 inet proto udp from (vether1:network:*) to (egress:*)
Re: kernel rebuild to debug problem
On 20/05/2024 00:03, Kirill A. Korinsky wrote: > On Sun, 19 May 2024 20:52:56 +0100, > Kapetanakis Giannis wrote: >> I'm trying to bisect a bug and compile an older kernel from cvs >> >> cvs checkout -D "2023-01-05" src/sys >> >> and following https://www.openbsd.org/faq/faq5.html#Options >> + make install >> >> New kernel compiles and boots but I get: >> >> # pfctl -f /etc/pf.conf >> pfctl: DIOCADDRULE: Operation not supported by device >> >> # pfctl -sr >> pfctl: Permission denied >> >> # pfctl -si | head -1 >> Status: Enabled for 0 days 00:05:03 Debug: err >> >> any ideas about this? >> > You need to build / use no only old kernel but the whole system. > > The simplest way is to use archived version of snapshots from > https://openbsd.cs.toronto.edu/archive/ or another mirror. Thanks for the reply. I did also build pfctl. Unfortunately the date I'm looking for is older than the archives on toronto or hostserver.de Anyway I'll try a build the whole system. Any hints of what specific is needed would be nice, since building the whole system every time until I find the commit I'm looking for would be a pain. G
Re: kernel rebuild to debug problem
On Sun, 19 May 2024 20:52:56 +0100, Kapetanakis Giannis wrote: > > I'm trying to bisect a bug and compile an older kernel from cvs > > cvs checkout -D "2023-01-05" src/sys > > and following https://www.openbsd.org/faq/faq5.html#Options > + make install > > New kernel compiles and boots but I get: > > # pfctl -f /etc/pf.conf > pfctl: DIOCADDRULE: Operation not supported by device > > # pfctl -sr > pfctl: Permission denied > > # pfctl -si | head -1 > Status: Enabled for 0 days 00:05:03 Debug: err > > any ideas about this? > You need to build / use no only old kernel but the whole system. The simplest way is to use archived version of snapshots from https://openbsd.cs.toronto.edu/archive/ or another mirror. -- wbr, Kirill
kernel rebuild to debug problem
I'm trying to bisect a bug and compile an older kernel from cvs cvs checkout -D "2023-01-05" src/sys and following https://www.openbsd.org/faq/faq5.html#Options + make install New kernel compiles and boots but I get: # pfctl -f /etc/pf.conf pfctl: DIOCADDRULE: Operation not supported by device # pfctl -sr pfctl: Permission denied # pfctl -si | head -1 Status: Enabled for 0 days 00:05:03 Debug: err any ideas about this? G
Re: pf anchors attached to irrelevant states
On 5/19/24 13:37, Stuart Henderson wrote: I can confirm this is a problem, definitely seen in 7.4, I can't remember if 7.3 was affected. 7.2 from Dec 22 seems ok. Yes, 7.3 is affected. It is the same problem reported here: https://marc.info/?l=openbsd-misc=168754952806369
Re: pf anchors attached to irrelevant states
On 19/05/2024 14:37, Stuart Henderson wrote: On 2024-05-19, Kapetanakis Giannis wrote: This is a bit strange. pf works normal, but rules after an enchor an being attached to the anchor (somehow). All states that are created from rules after the anchor, show the anchor (pf rule) number instead of (only) the rule number in pfctl -vv and in pflog. I can confirm this is a problem, definitely seen in 7.4, I can't remember if 7.3 was affected. 7.2 from Dec 22 seems ok. 7.3 release was also affected, just tested on a vm. G
Re: pf anchors attached to irrelevant states
On 2024-05-19, Kapetanakis Giannis wrote: > This is a bit strange. pf works normal, but rules after an enchor an > being attached to the anchor (somehow). > > All states that are created from rules after the anchor, show the anchor > (pf rule) number instead of (only) the rule number in pfctl -vv and in > pflog. I can confirm this is a problem, definitely seen in 7.4, I can't remember if 7.3 was affected. 7.2 from Dec 22 seems ok.
pf anchors attached to irrelevant states
This is a bit strange. pf works normal, but rules after an enchor an
being attached to the anchor (somehow).
All states that are created from rules after the anchor, show the anchor
(pf rule) number instead of (only) the rule number in pfctl -vv and in
pflog.
Here is a quite simple example.
# pfctl -sr -a'*' -vv | egrep -v "Evaluations|Inserted" | head -6
@0 match in all scrub (no-df random-id)
@1 pass in quick on vio0 from to any flags S/SA set (prio 6) keep
state (if-bound, pflow) tag from_external
@2 anchor "test" quick all {
@0 pass out log quick on egress inet proto tcp from any to any port = 2000
flags S/SA keep state (if-bound) rdr-to 127.0.0.1
}
@3 pass out log quick inet proto tcp from any to yy.yy.yy.yy port = 22 flags
S/SA keep state (if-bound, pflow)
Test traffic for anchor rule works fine (xx.xx.xx.xx is my external ip)
# telnet 8.8.8.8 2000
pflog: May 19 13:54:03.427024 rule 2.test.0/(match) pass out on vio0: xx.xx.xx.36179
> 8.8.8.8.2000: S 4080176752:4080176752(0) win 16384 (DF) [tos 0x10]
# pfctl -ss -vv | grep -A3 8.8.8.8
vio0 tcp xx.xx.xx.xx:36179 -> 127.0.0.1:2000 (8.8.8.8:2000)
SYN_SENT:CLOSED
[4080176752 + 2] [0 + 1]
age 00:00:01, expires in 00:01:59, 1:0 pkts, 64:0 bytes, anchor 2, rule 0
<<<--- this rule 0 of anchor which is correct
id: 661391580039aaa3 creatorid: bfd893f9
See what happens if I try to triger rule @3 and ssh to yy.yy.yy.yy
pflog: May 19 13:55:42.386186 rule 2/(match) pass out on vio0: xx.xx.xx.xx.23564 >
yy.yy.yy.yy.22: S 3631867116:3631867116(0) win 16384 (DF) [tos 0x48]
pfctl -ss -vv|grep -A3 yy.yy.yy.yy
vio0 tcp xx.xx.xx.xx:23564 -> yy.yy.yy.yy:22 ESTABLISHED:ESTABLISHED
[3631869502 + 37760] wscale 6 [3744464382 + 16384] wscale 7
age 00:01:10, expires in 23:58:54, 16:19 pkts, 3229:3857 bytes, anchor 2,
rule 3, pflow
id: 661391580039ab07 creatorid: bfd893f9
pflog, logs "rule 2" which is the anchor instead of "rule 3"
pfctl, shows "anchor 2, rule 3" instead of just "rule 3"
Traffic works normally but there is something fishy here.
quick on anchor does not make any difference, although to my understanding it
shouldn't matter either set or not set in this case.
G
Re: smtpd outgoing mail configuration
On Fri, May 17, 2024 at 08:12:27AM +0200, fr...@lilo.org wrote: How to forward outgoing mail to a remote SMTP server with smtpd? I found this page, but it's out of date I think. https://romanzolotarev.com/openbsd/smtpd-forward.html Tks Pascal I have mine setup like this and its working. My /etc/mail/smtpd.conf: --- start file --- # $OpenBSD: smtpd.conf,v 1.14 2019/11/26 20:14:38 gilles Exp $ # This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. table aliases file:/etc/mail/aliases listen on socket # To accept external mail, replace with: listen on all listen on all action "local_mail" mbox alias action "outbound" relay host smtp://" # Uncomment the following to accept external mail for domain match from # any for domain "" action "local_mail" match from local for local action "local_mail" match from local for any action "outbound" --- End file --- # doas rcctl enable smtpd # doas
Re: pax and ext2fs
On Sat May 18 08:50:21 2024 Philip Guenther wrote:
> > So yeah, what's needed is pathconfat(2)** but whether this winding loose
> > end ("That poor yak.") merits that much code and surface is yet to be
> > examined deeply.
>
> The fix for this has now been committed, so it'll be in 7.6 and a near
> future snapshot.
>
And you wrote the library!
Philip Guenther at https://austingroupbugs.net/view.php?id=1831 wrote:
With a fresh cup of coffee, it's 'obvious' the correct action is to
use pathconfat(AT_FDCWD, path, _PC_TIMESTAMP_RESOLUTION,
AT_SYMLINK_NOFOLLOW)
This was touched on in https://austingroupbugs.net/view.php?id=786
[^] (c.f. Geoff's comment 2827 from 2015) so maybe I should just
implement this in OpenBSD and drag everyone else along from there. :)
No yaks for Philip "John Wayne" Guether, only cows. ;-)
>
> Philip Guenther
>
>
Walter
Re: nginx + php = system() not working?
On 2024-05-17, Martijn van Duren wrote: > On Thu, 2024-05-16 at 21:58 -0400, F Bax wrote: >> I think I missed something simple? I installed 7.5 release in a VM. I then >> installed nginx and PHP 8.3.3; with pkg_add. I then ran these two commands: >> # rcctl enable php83_fpm >> # rcctl start php83_fpm >> I found an issue with php system() function; so created this simple script >> which produces "HiThere"; why is the date not presented? >> >> > echo 'Hi'; >> system( 'date' ); >> echo 'There'; >> ?> > > All the advise I've seen is horrible. chroot isn't enabled by default > without a reason (php and security have a history). > My first question would by why you need system() in the first place. > If you need the date/time, just use > https://www.php.net/manual/en/class.datetime.php. If it's just a proof > of concept be more precise in what you want to achieve and see if > there's a PHP library equivalent. If there's no reasonable way to > achieve it (which I highly doubt) I advise to copy the required binary > (and dependencies) into the chroot and make a memo to keep them up to > date. There's some information about this in PHP's pkg-readme file. -- Please keep replies on the mailing list.
Re: pax and ext2fs
On Thu, May 16, 2024 at 12:08 AM Philip Guenther wrote:
> On Wed, May 15, 2024 at 1:14 AM Philip Guenther wrote:
...
>> I think you've managed to hit a spot where the POSIX standard doesn't
>> provide a way for a program to find the information it needs to do its job
>> correctly. I've filed a ticket there
>>https://austingroupbugs.net/view.php?id=1831
>>
>> We'll see if my understanding of pathconf() is incorrect or if someone has a
>> great idea for how to get around this...
>
> So yeah, what's needed is pathconfat(2)** but whether this winding loose end
> ("That poor yak.") merits that much code and surface is yet to be examined
> deeply.
The fix for this has now been committed, so it'll be in 7.6 and a near
future snapshot.
Philip Guenther
Re: nginx + php = system() not working?
"Souji Thenria" wrote: > Another issue might be that nginx is still running as www and doesn't > have access to /home/Testing. As per above suggestion double check that the user by which you run nginx (usually www) has access almost by the group to to the prefix directory declared by the -p flag, and to the subfolders. (and clearly you can't manage to do that on an usr home dir..) Then you should double check your phpfpm user and group by the php-fpm.conf in the following declarations: ; Unix user/group of processes ; Note: The user is mandatory. If the group is not set, the default user's group ; will be used. user = www group = www -dan
Re: nginx + php = system() not working?
On Fri May 17, 2024 at 2:56 PM BST, F Bax wrote:
In /etc/rc.conf.local - I changed nginx_flags="-u -p /home/Testing"
(home directory of a real user).
reboot system and now browser is refused connection
This site can’t be reached 192.168.1.131 refused to connect.
Neither /var/www/logs/{access|error}.log is changed.
What else needs to change?
Can you verify that nginx is running?
You may have an error in your configuration. You can check the nginx
configuration using nginx -t.
Another issue might be that nginx is still running as www and doesn't
have access to /home/Testing.
Regards,
Souji
Re: nginx + php = system() not working?
On Thu, 2024-05-16 at 21:58 -0400, F Bax wrote: > I think I missed something simple? I installed 7.5 release in a VM. I then > installed nginx and PHP 8.3.3; with pkg_add. I then ran these two commands: > # rcctl enable php83_fpm > # rcctl start php83_fpm > I found an issue with php system() function; so created this simple script > which produces "HiThere"; why is the date not presented? > > echo 'Hi'; > system( 'date' ); > echo 'There'; > ?> All the advise I've seen is horrible. chroot isn't enabled by default without a reason (php and security have a history). My first question would by why you need system() in the first place. If you need the date/time, just use https://www.php.net/manual/en/class.datetime.php. If it's just a proof of concept be more precise in what you want to achieve and see if there's a PHP library equivalent. If there's no reasonable way to achieve it (which I highly doubt) I advise to copy the required binary (and dependencies) into the chroot and make a memo to keep them up to date. martijn@
Re: nginx + php = system() not working?
Thanks for the tips and security warnings Mike, Souji and Dan,
In php-fpm.conf - I changed "; chroot = /var/www" to comment.
In /etc/rc.conf.local - I changed nginx_flags="-u -p /home/Testing"
(home directory of a real user).
reboot system and now browser is refused connection
This site can’t be reached 192.168.1.131 refused to connect.
Neither /var/www/logs/{access|error}.log is changed.
/var/log/php-fpm.log show normal startup; then nothing in any /var/log/
files.
[17-May-2024 09:41:59] NOTICE: fpm is running, pid 8072
[17-May-2024 09:41:59] NOTICE: ready to handle connections
What else needs to change?
Re: nginx + php = system() not working?
It can even help to run nginx in "unsecure mode" if you want to stay not chrooted: nginx_flags="-u -p /home/mytests" man nginx ; while php-fpm.conf should remain with the default values ; in this case.. -dan Mike Fischer wrote: > > > Am 17.05.2024 um 03:58 schrieb F Bax : > > > > I think I missed something simple? I installed 7.5 release in a VM. > > I then installed nginx and PHP 8.3.3; with pkg_add. I then ran > > these two commands: # rcctl enable php83_fpm # rcctl start php83_fpm > > I found an issue with php system() function; so created this simple > > script which produces "HiThere"; why is the date not presented? > > > system( 'date' ); > > echo 'There'; > > ?> > > You are probably running the php83_fpm process accessed from nginx in > the default chroot(2) environment?
Re: nginx + php = system() not working?
May 17, 2024 11:30:25 Souji Thenria : > -u By default nginx will chroot(2) to the home > directory of the user running the daemon, typically > "www", or to the home directory of user in > nginx.conf. The -u option disables this behaviour, > and returns nginx to the original "unsecure" > behaviour. > > But it doesn't do it on other systems; I cross-checked with nginx > installed on a FreeBSD, where this option doesn't exist. Indeed take care about this option as I use it every day in my dev environment.. ;-)
Re: nginx + php = system() not working?
On Fri May 17, 2024 at 4:38 AM BST, Mike Fischer wrote: OpenBSD httpd would be a different situation because it runs in a chroot(2) environment by default. You can’t call on a PHP-FPM process that is not also running in the chroot(2) environment. The communication between httpd(8) and PHP-FPM fails due to differing opinions about the root of the filesystem when applied to the paths passed from httpd to PHP-FPM. At least I have not managed to get this to work. But AFAIK nginx does not run chroot(2)ed by default. So PHP-FPM does not need to either. On OpenBSD, nginx chroots its process by default. Here is a snippet from the man page nginx(8). -u By default nginx will chroot(2) to the home directory of the user running the daemon, typically "www", or to the home directory of user in nginx.conf. The -u option disables this behaviour, and returns nginx to the original "unsecure" behaviour. But it doesn't do it on other systems; I cross-checked with nginx installed on a FreeBSD, where this option doesn't exist. Since nginx and php_fpm chroot their processes to the same directory (if not changed), nginx should be able to write to the php_fpm socket.
Re: pax and ext2fs
On Fri May 17 09:50:58 2024 Philip Guenther wrote: > Sounds like you copied with something like 'cp -p' so the copy has a > mtime with zero nsecs part, so now they do compare as equal. This morning I realized that when I copied the symlink from the ext2 drive to my hard disk, cp(1) didn't copy the symlink but the executable itself. Reading cp(1) man page I see that the command I should have used to copy the symlink is 'cp -PR'. In my case caffeine is affecting negatively, it makes me jump to conclusions. Sorry for make you waste your time! > > > > P.S.: I'm curious about the following. After running the stat command > > here and there, I found *many* files showing that lack of mtime > > granularity spread throughout all my system tree (as a side note: this > > doesn't happen with their ctime and atime.) > > The released install tgz files (base75.tgz, etc) use a format where > the contained files all have simple integer mtimes and tar is invoked > with the -p option (required for correct permissions on setuid/gid > files) which makes it also set the mtime on the extracted file to > match what's in the tar file. > > ctime is always set from the local clock when the inode is > allocated/updated, so no reason for it to always have a zero nsecs. > > atime is of course updated from the local clock when you, uh, access them. Thanks for your explanation! > > > Philip Guenther > > Walter
Re: smtpd outgoing mail configuration
Den fre 17 maj 2024 kl 08:56 skrev Pascal Deveaux : > > The command > # chown root:_smtpd /etc/mail/secrets > Return : group smtpd doesn't exist The error message doesn't match the command at all, and the _smtpd group has been in the group file for some 15 years. Look for misspellings somewhere. Or a broken /etc/group file that somehow lacks the _smtpd group. -- May the most significant bit of your life be positive.
Re: pax and ext2fs
On Thu, May 16, 2024 at 5:33 AM Walter Alejandro Iglesias
wrote:
>
> On Thu May 16 09:48:45 2024 Philip Guenther wrote:
> > So yeah, what's needed is pathconfat(2)** but whether this winding loose
> > end ("That poor yak.") merits that much code and surface is yet to be
> > examined deeply.
...
> I read what you posted here:
>
> https://austingroupbugs.net/view.php?id=1831
>
> In the footnote you wrote:
>
> "(This was encountered when trying to fix a pax implementation's
> handling of timestamp comparison for -u when the target filesystem had
> courser resolution that the source filesystem by using
> pathconf(_PC_TIMESTAMP_RESOLUTION) on the target path to handle the
> loss of high-precision time info...but the symlink pointed to a
> location with high-precision timestamps so it couldn't know to round
> the times when doing the comparison...)"
>
>
> I did one more experiment. I removed the offending soft link from my
> hard disk, then I copied the backed-up version of the soft link from the
> ext2 drive back to my system tree.
So you did so and then checked the timestamps on the symlinks using
stat to see how they compared, yes?
> Now pax (with your patches) doesn't
> insist in re-updating the file,
Sounds like you copied with something like 'cp -p' so the copy has a
mtime with zero nsecs part, so now they do compare as equal.
> *even after updating the file with
> touch(1)*.
Why would the symlink needs to be recopied by pax? You didn't update
the symlink's timestamps.
> The soft link *still* points to a location with high-precision
> timestampts, but pax does the right job.
Because the symlinks now have the exact same timestamp, one with zero nsecs.
> Intuitively this suggests me that there is something more that mtime
> precision in this misunderstanding between OpenBSD and ext2 file
> systems.
I think you should check the timestamps on the symlinks at each step
to validate that.
> P.S.: I'm courious about the following. After running the stat command
> here and there, I found *many* files showing that lack of mtime
> granularity spread throughout all my system tree (as a side note: this
> doesn't happen with their ctime and atime.)
The released install tgz files (base75.tgz, etc) use a format where
the contained files all have simple integer mtimes and tar is invoked
with the -p option (required for correct permissions on setuid/gid
files) which makes it also set the mtime on the extracted file to
match what's in the tar file.
ctime is always set from the local clock when the inode is
allocated/updated, so no reason for it to always have a zero nsecs.
atime is of course updated from the local clock when you, uh, access them.
Philip Guenther
Re: smtpd outgoing mail configuration
The command # chown root:_smtpd /etc/mail/secrets Return : group smtpd doesn't exist 17 mai 2024 10:32:19 Otto Moerbeek : > On Fri, May 17, 2024 at 08:12:27AM +0200, fr...@lilo.org wrote: > >> How to forward outgoing mail to a remote SMTP server with smtpd? >> >> I found this page, but it's out of date I think. >> https://romanzolotarev.com/openbsd/smtpd-forward.html >> >> Tks >> Pascal > > man smtpd.conf, first exmaple > > -Otto -- Pascal
Re: smtpd outgoing mail configuration
On 17/05/24 11:42, fr...@lilo.org wrote: How to forward outgoing mail to a remote SMTP server with smtpd? I found this page, but it's out of date I think. https://romanzolotarev.com/openbsd/smtpd-forward.html Tks Pascal The config looks fine, use: man smtpd.conf -James
Re: smtpd outgoing mail configuration
On Fri, May 17, 2024 at 08:12:27AM +0200, fr...@lilo.org wrote: > How to forward outgoing mail to a remote SMTP server with smtpd? > > I found this page, but it's out of date I think. > https://romanzolotarev.com/openbsd/smtpd-forward.html > > Tks > Pascal man smtpd.conf, first exmaple -Otto
smtpd outgoing mail configuration
How to forward outgoing mail to a remote SMTP server with smtpd? I found this page, but it's out of date I think. https://romanzolotarev.com/openbsd/smtpd-forward.html Tks Pascal
Re: nginx + php = system() not working?
> Am 17.05.2024 um 03:58 schrieb F Bax : > > I think I missed something simple? I installed 7.5 release in a VM. I then > installed nginx and PHP 8.3.3; with pkg_add. I then ran these two commands: > # rcctl enable php83_fpm > # rcctl start php83_fpm > I found an issue with php system() function; so created this simple script > which produces "HiThere"; why is the date not presented? >echo 'Hi'; > system( 'date' ); > echo 'There'; > ?> You are probably running the php83_fpm process accessed from nginx in the default chroot(2) environment? If so you need to reconfigure your /etc/php-fpm.conf to not use chroot(2) — comment the line »chroot = /var/www« — or install /bin/date and a /bin/sh into /var/www. (Note that PHP needs a shell to execute shell commands and the date command is not present in the chroot(2) environment by default either.) Your test script works for me in Apache httpd and a php-fpm.conf without chroot(2) (in a non-public setting). So I don’t think this is related to nginx specifically. Could happen with OpenBSD httpd and PHP-FPM as well. Basically in any situation where PHP-FPM is running chroot(2)ed. OpenBSD httpd would be a different situation because it runs in a chroot(2) environment by default. You can’t call on a PHP-FPM process that is not also running in the chroot(2) environment. The communication between httpd(8) and PHP-FPM fails due to differing opinions about the root of the filesystem when applied to the paths passed from httpd to PHP-FPM. At least I have not managed to get this to work. But AFAIK nginx does not run chroot(2)ed by default. So PHP-FPM does not need to either. Note: If you need both you can configure your /etc/php-fpm.conf to spawn both chroot(2)ed and non-chroot(2)ed workers with differing sockets. I’m doing this on a machine running both OpenBSD httpd and Apache httpd with PHP based web pages. HTH Mike PS. Hopefully you are aware that running shell commands from a publicly accessible web server can lead to serious security issues? Be very careful when configuring access restrictions to the affected URLs and when constructing the UNIX commands you plan to execute.
Re: What software to debugging and analyzing C?
But you do realise that adding printf() calls to the code can also
change,
for example, the memory layout that the compiler uses, so certain
memory
allocation bugs might become more or less easily triggerable?
This is a big deal especially debugging code that fails with -O3 but
succeeds
otherwise.
My approach (shamelessly stolen from GSL) is to write a debug callable
and let your debugger perform the "printf", rather than your own code.
$ cat oopsie.c
/* define an extern iff debugging */
#ifdef DEBUG
void oopsie(char *s,...){ }
#endif
And in your main code:
...
/* define an extern iff debugging */
#ifdef DEBUG
void oopsie(char *s,...);
#else
void oopsie(char *s,...){ }
#endif
int main(){
int i, j;
printf("%s\n","this is main.c demonstrating debug printing");
i = rand();
if(i> 5) oopsie("i more than 5");
...
Compile with DEBUG on. Run your code with the debugger, and break on
oopsie.
$ gdb ./main
...
(gdb) break oopsie
(gdb) run
Starting program: /home/jal/debug/a.out
...
this is main.c demonstrating debug printing
...
Breakpoint 1, oopsie (s=0x9766035d548 "i more than 5") at oopsie.c:4
4 void oopsie(char *s,...){ }
Backtrace tells you where oopsie was called.
J
nginx + php = system() not working?
I think I missed something simple? I installed 7.5 release in a VM. I then installed nginx and PHP 8.3.3; with pkg_add. I then ran these two commands: # rcctl enable php83_fpm # rcctl start php83_fpm I found an issue with php system() function; so created this simple script which produces "HiThere"; why is the date not presented?
Re: OpenBSD 7.5: xfce-4.18.1: missing Special Characters utility
Reinstalled manually after the upgrade to 7.5 with: pkg_add gucharmap -dan Dan wrote: > > Hello, > > In my OpenBSD 7.5, xfce-4.18.1 is missing the Characters Map / Special > Characters utility both graphically, in the menu, and on the disk. > > Thanks! > > -dan
Weird nd6_debug:nd6_ns_input logging on neighbor sol
Greetings, we're having at least two different weird ndp/icmp6 related behaviours we would like to share The setup is quite simple, we're trying to ping6 from one OpenBSD 6.8 to another, sometimes, without any clear reason, the host sending the request will start to use its link-local address in the icmp neighbor solicitation See below a NOK scenario with the solicitation dropped by the requested system The OK scenario may come back after a while, where the unicast address is again in use in the the neighbor solicitation source address and is properly answered A simple way we found to reproduce this on demand is to run a ndp delete and then ping in a loop $ while (:); do ndp -d 2a02:6f00:c0:30::10; ping6 -c2 2a02:6f00:c0:30::10; done Usually after 5 to 10 attempts the issue shows up NOK pcap on requested host (trafic goes through but ignored by target/requested host) # 16:50:06.340898 00:15:17:dd:60:fa 33:33:ff:00:00:10 86dd 86: fe80::215:17ff:fedd:60fa > ff02::1:ff00:10: icmp6: neighbor sol: who has 2a02:6f00:c0:30::10 # 16:50:07.356429 00:15:17:dd:60:fa 33:33:ff:00:00:10 86dd 86: fe80::215:17ff:fedd:60fa > ff02::1:ff00:10: icmp6: neighbor sol: who has 2a02:6f00:c0:30::10 OK same pcap with unicast address of source instead of link local, properly answered # 16:57:10.308265 00:15:17:dd:60:fa 33:33:ff:00:00:10 86dd 86: 2a02:6f00:c0:30::3 > ff02::1:ff00:10: icmp6: neighbor sol: who has 2a02:6f00:c0:30::10 # 16:57:10.308512 00:1e:67:15:18:ba 00:15:17:dd:60:fa 86dd 86: 2a02:6f00:c0:30::11 > 2a02:6f00:c0:30::3: icmp6: neighbor adv: tgt is 2a02:6f00:c0:30::10 We then enabled nd6_debug with sysctl net.inet6.icmp6.nd6_debug=1, ran the test again and below are the pcap and dmesg output Notice how the src and dst address in nd6_ns_input are altered We started to wonder what is at fault, the 'NS packet from non-neighbor' check or the ipv6 fields being altered higher up in the chain On the live system - OpenBSD 6.8 (yes, we plan to update) # pcap - correct source link-local address # 13:42:54.252709 00:15:17:dd:60:fa 33:33:ff:00:00:10 86dd 86: fe80::215:17ff:fedd:60fa > ff02::1:ff00:10: icmp6: neighbor sol: who has 2a02:6f00:c0:30::10 # dmesg with altered src and dst addresses (extra 9 on bits 16-31) # May 16 13:42:54 vpn1 /bsd: nd6_ns_input: NS packet from non-neighbor # May 16 13:42:54 vpn1 /bsd: nd6_ns_input: src=fe80:9::215:17ff:fedd:60fa # May 16 13:42:54 vpn1 /bsd: nd6_ns_input: dst=ff02:9::1:ff00:10 # May 16 13:42:54 vpn1 /bsd: nd6_ns_input: tgt=2a02:6f00:c0:30::10 The same attempt on a 7.3 system in a "lab" are working correctly: 1. With a "default" configuration, the src address in the neighbor solicitation is the unicast address and not the link local address, 'NS packet from non-neighbor' is not triggered and ndp works correctly, icmp6 request/reply then goes through without issues 2. When explicitly crafting a packet with the link local address as source, the remote system properly reply to neighbor solicitation and icmp req/rep goes as well too which would indicate that 'NS packet from non-neighbor' allows link local address as source 3. The interesting part is when we explicitly craft a neighbor solicitation packet with a "invalid" source link local address, the 'NS packet from non-neighbor' triggers and shows up in dmesg, but again, with an altered bit - "lab" testing - 7.3 # pcap - crafted incorrect source link-local address # 13:58:42.675849 c8:8a:9a:a3:ea:26 33:33:ff:00:00:01 86dd 86: fe80::9:5054:ff:fe28:e372 > ff02::1:ff00:1: icmp6: neighbor sol: who has 2a02:6f00:::1 # dmesg with altered src and dst address (extra 1 on bits 16-31 on top of crafted 9) # May 16 13:58:42 obsd68 /bsd: nd6_ns_input: NS packet from non-neighbor # May 16 13:58:42 obsd68 /bsd: nd6_ns_input: src=fe80:1:0:9:5054:ff:fe28:e372 # May 16 13:58:42 obsd68 /bsd: nd6_ns_input: dst=ff02:1::1:ff00:1 # May 16 13:58:42 obsd68 /bsd: nd6_ns_input: tgt=2a02:6f00:::1 Questions are then: - why would a system change between link-local and unicast address in the source address of neighbor solicitations it sends (although I'm painfully aware 6.8 is quite old) - is the nd6_ns_input logging having issue with display of dst/src addresses ? - and/or the 'NS packet from non-neighbor' mechanism known to sometime discard perfectly valid neighbor solicitation with link-local address as source ? Thank you, and let me know if you would need more details on this
Re: Bgpd multipath conf
I'm working on something similar right now for bgpd, where any connected /128 ipv6 address will be announced over bgp. For example if the router is connected to an adjacent host that has assigned itself an address through slaac such that the router has an entry for that particular host in the routing table, then the router will announce the host's /128 address. On Thu, May 16, 2024 at 6:24 AM Stuart Henderson wrote: > > On 2024-05-16, Marco Agostani wrote: > > Ok so in the end is there a way to install more then one route in the > > kernel table through bgpd or not ? > > No. That is what "bgpd ... does not handle adding multiple paths for the > same prefix to the FIB" means. (FIB = "forwarding information base" = > kernel route table) > > > And if it's something that could be done in the future ? > > could? sure, if someone were to write the code to support it. > > I don't think it will be a particularly easy thing to do though. > > > -- > Please keep replies on the mailing list. >
Re: pax and ext2fs
On Thu May 16 09:48:45 2024 Philip Guenther wrote:
> So yeah, what's needed is pathconfat(2)** but whether this winding loose
> end ("That poor yak.") merits that much code and surface is yet to be
> examined deeply.
>
> Philip Guenther
>
>
> ** or lpathconf(2), but pathconfat(2) is better
>
I read what you posted here:
https://austingroupbugs.net/view.php?id=1831
In the footnote you wrote:
"(This was encountered when trying to fix a pax implementation's
handling of timestamp comparison for -u when the target filesystem had
courser resolution that the source filesystem by using
pathconf(_PC_TIMESTAMP_RESOLUTION) on the target path to handle the
loss of high-precision time info...but the symlink pointed to a
location with high-precision timestamps so it couldn't know to round
the times when doing the comparison...)"
I did one more experiment. I removed the offending soft link from my
hard disk, then I copied the backed-up version of the soft link from the
ext2 drive back to my system tree. Now pax (with your patches) doesn't
insist in re-updating the file, *even after updating the file with
touch(1)*.
The soft link *still* points to a location with high-precision
timestampts, but pax does the right job.
Intuitively this suggests me that there is something more that mtime
precision in this misunderstanding between OpenBSD and ext2 file
systems. If I copy files using pax from Linux (another *BSD* version of
pax) to that same ext2 drive it works as expected.
Walter
P.S.: I'm courious about the following. After running the stat command
here and there, I found *many* files showing that lack of mtime
granularity spread throughout all my system tree (as a side note: this
doesn't happen with their ctime and atime.)
Re: Bgpd multipath conf
>> Ok so in the end is there a way to install more then one route in the kernel >> table through bgpd or not ? >No. That is what "bgpd ... does not handle adding multiple paths for the same >prefix to the FIB" means. (FIB = "forwarding information base" = kernel route >table) Ok so the only thing is having a 3 routers instead of one. Two speaking ebgp and the third speaking ospf with them or static multipath in order to send load balanced traffic . >> And if it's something that could be done in the future ? >could? sure, if someone were to write the code to support it. >I don't think it will be a particularly easy thing to do though. Yeah, probably not me ☹. Anyway tks a lot for your answer . Cheers Marco -- Please keep replies on the mailing list. Caterpillar: Confidential Green
Re: Bgpd multipath conf
On 2024-05-16, Marco Agostani wrote: > Ok so in the end is there a way to install more then one route in the kernel > table through bgpd or not ? No. That is what "bgpd ... does not handle adding multiple paths for the same prefix to the FIB" means. (FIB = "forwarding information base" = kernel route table) > And if it's something that could be done in the future ? could? sure, if someone were to write the code to support it. I don't think it will be a particularly easy thing to do though. -- Please keep replies on the mailing list.
Re: Bgpd multipath conf
Ok so in the end is there a way to install more then one route in the kernel
table through bgpd or not ?
And if it's something that could be done in the future ?
Cheers
Marco
Caterpillar: Confidential Green
-Original Message-
From: Stuart Henderson
Sent: Wednesday, May 15, 2024 8:26 AM
To: misc@openbsd.org
Subject: Re: Bgpd multipath conf
CAUTION: EXTERNAL EMAIL This is a message from owner-m...@openbsd.org. Use
caution when opening unexpected emails and do not click on links or attachments
from unknown senders. For more resources, visit security.cat.com/phishing.
__
On 2024-05-14, Marco Agostani wrote:
> I try to setup an openbgpd setup involving multipath configuration
> ...with = no success.
...
> neighbor $GW01 {
> descr "bgp#1"
> announce IPv4 unicast
> announce add-path recv yes
> set localpref 110
> }
This just announces the add-path BGP capability.
> #bgpctl sh rib
> https://urldefense.com/v3/__http://172.18.180.0/24__;!!FtR4BK4x7WL3xYs
> !6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG2
> 7Pi4GFOtJXl9T8K7jGv49b5Fz40$
>
> flags: * =3D Valid, > =3D Selected, I =3D via IBGP, A =3D Announced,
>S =3D Stale, E =3D Error
> origin validation state: N =3D not-found, V =3D valid, ! =3D invalid
> aspa validation state: ? =3D unknown, V =3D valid, ! =3D invalid
> origin: i =3D IGP, e =3D EGP, ? =3D Incomplete
>
> flags vs destination gateway lpref med aspath origin
> *>N-?
> https://urldefense.com/v3/__http://172.18.180.0/24__;!!FtR4BK4x7WL3xYs!6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG27Pi4GFOtJXl9T8K7jGv49b5Fz40$
>
> https://urldefense.com/v3/__http://10.0.1.241__;!!FtR4BK4x7WL3xYs!6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG27Pi4GFOtJXl9T8K7jGv41J-7V7w$
> 110 0 14381 i
> *mN-?
> https://urldefense.com/v3/__http://172.18.180.0/24__;!!FtR4BK4x7WL3xYs!6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG27Pi4GFOtJXl9T8K7jGv49b5Fz40$
>
> https://urldefense.com/v3/__http://10.0.1.245__;!!FtR4BK4x7WL3xYs!6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG27Pi4GFOtJXl9T8K7jGv48_2TUKx$
> 110 0 14381 i
>
> Show me two routes one marked with multipath
>
> But in fib I see only one route
>
> #bgpctl sh fib
> https://urldefense.com/v3/__http://172.18.180.0/24__;!!FtR4BK4x7WL3xYs
> !6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG2
> 7Pi4GFOtJXl9T8K7jGv49b5Fz40$
>
> flags: B =3D BGP, C =3D Connected, S =3D Static
>N =3D BGP Nexthop reachable via this route
>r =3D reject route, b =3D blackhole route
> flags prio destination gateway
> B 48
> https://urldefense.com/v3/__http://172.18.180.0/24__;!!FtR4BK4x7WL3xYs!6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG27Pi4GFOtJXl9T8K7jGv49b5Fz40$
>
> https://urldefense.com/v3/__http://10.0.1.241__;!!FtR4BK4x7WL3xYs!6DAqIASIWdakyXeDyLwAsmIOK4cM3WJigBhTBEZwFP2QSx-N8iqQRQKGfW0L4XpCyVGG27Pi4GFOtJXl9T8K7jGv41J-7V7w$
...
> What I miss here ??
bgpd does allow add-path and having multiple paths to a prefix in the RIB (e.g.
perhaps useful on a route-server) but it does not handle adding multiple paths
for the same prefix to the FIB.
Re: pax and ext2fs
On Wed, May 15, 2024 at 1:14 AM Philip Guenther wrote:
> On Tue, May 14, 2024 at 11:59 AM Walter Alejandro Iglesias <
> w...@roquesor.com> wrote:
>
>> Hi Philip,
>>
>> On Tue May 14 19:40:04 2024 Philip Guenther wrote:
>> > If you like, you could try the following patch to pax to more gracefully
>> > handle filesystems with time resolution more granular than nanoseconds.
>>
>> After applying your patch, as I'd done before reporting the issue, I
>> sycronized my home directory to an external ext2fs drive with the
>> command showed by the man page:
>>
>> $ pax -rw -v -Z -Y source target
>>
>> This time only one file stays updating again an again, a soft link I
>> have in my ~/bin folder of /usr/local/bin/prename.
>
>
> I think you've managed to hit a spot where the POSIX standard doesn't
> provide a way for a program to find the information it needs to do its job
> correctly. I've filed a ticket there
>https://austingroupbugs.net/view.php?id=1831
>
> We'll see if my understanding of pathconf() is incorrect or if someone has
> a great idea for how to get around this...
>
So yeah, what's needed is pathconfat(2)** but whether this winding loose
end ("That poor yak.") merits that much code and surface is yet to be
examined deeply.
Philip Guenther
** or lpathconf(2), but pathconfat(2) is better
Re: Errata: OpenBSD 7.5: high temperature spotted different times
Correction: CPU: Intel(R) Core(TM) i7-4500U CPU @ 1.80GHz, 06-45-01, patch 0026 (year 2014) Dan wrote: > Hello, > > In my OpenBSD 7.5 stable temperature incrises timtotime remaining on > 64-65°C; an old quad cores I5 cpu. > > Thanks, > > -dan
OpenBSD 7.5: xfce-4.18.1: missing Special Characters utility
Hello, In my OpenBSD 7.5, xfce-4.18.1 is missing the Characters Map / Special Characters utility both graphically, in the menu, and on the disk. Thanks! -dan
OpenBSD 7.5: high temperature spotted different times
Hello, In my OpenBSD 7.5 stable temperature incrises timtotime remaining on 64-65°C; an old quad cores I5 cpu. Thanks, -dan
subscribe
subscribe please
Re: pax and ext2fs
On Wed May 15 13:04:53 2024 Walter Alejandro Iglesias wrote: > After more testing I realized that I was wrong my modification doesn't > solve the problem. > Yeah, I also realized that what I did was stupid. :-)
Re: pax and ext2fs
On Wed May 15 10:24:32 2024 Walter Alejandro Iglesias wrote: > I get it working but I don't know if what I did is fine. > > As I'd told you the problem was ctime (when using -Y), so I added one > conditional to your diff where it checks only mtime and it works: > > > Index: ar_subs.c > === > RCS file: /cvs/src/bin/pax/ar_subs.c,v > diff -u -p -r1.51 ar_subs.c > [...] After more testing I realized that I was wrong my modification doesn't solve the problem.
Re: pax and ext2fs
On Wed May 15 10:20:04 2024 Philip Guenther wrote:
> I think you've managed to hit a spot where the POSIX standard doesn't
> provide a way for a program to find the information it needs to do its job
> correctly. I've filed a ticket there
>https://austingroupbugs.net/view.php?id=1831
>
> We'll see if my understanding of pathconf() is incorrect or if someone has
> a great idea for how to get around this...
>
>
> Philip Guenther
>
Hi Philip,
I get it working but I don't know if what I did is fine.
As I'd told you the problem was ctime (when using -Y), so I added one
conditional to your diff where it checks only mtime and it works:
Index: ar_subs.c
===
RCS file: /cvs/src/bin/pax/ar_subs.c,v
diff -u -p -r1.51 ar_subs.c
--- ar_subs.c 10 Jul 2023 16:28:33 - 1.51
+++ ar_subs.c 15 May 2024 08:19:08 -
@@ -146,23 +146,61 @@ list(void)
}
static int
-cmp_file_times(int mtime_flag, int ctime_flag, ARCHD *arcn, struct stat *sbp)
+cmp_file_times(int mtime_flag, int ctime_flag, ARCHD *arcn, const char *path)
{
struct stat sb;
+ long res;
- if (sbp == NULL) {
- if (lstat(arcn->name, ) != 0)
- return (0);
- sbp =
+ if (path == NULL)
+ path = arcn->name;
+ if (lstat(path, ) != 0)
+ return (0);
+
+ /*
+* The target (sb) mtime might be rounded down due to the limitations
+* of the FS it's on. If it's strictly greater or we don't care about
+* mtime, then precision doesn't matter, so check those cases first.
+*/
+ if (ctime_flag && mtime_flag) {
+ if (timespeccmp(>sb.st_mtim, _mtim, <=))
+ return timespeccmp(>sb.st_ctim, _ctim, <=);
+ if (!timespeccmp(>sb.st_ctim, _ctim, <=))
+ return 0;
+ /* <= ctim, but >= mtim */
+ } else if (mtime_flag) {
+ return timespeccmp(>sb.st_mtim, _mtim, <=);
+ } else if (ctime_flag)
+ return timespeccmp(>sb.st_ctim, _ctim, <=);
+ else if (timespeccmp(>sb.st_mtim, _mtim, <=))
+ return 1;
+
+ /*
+* If we got here then the target arcn > sb for mtime *and* that's
+* the deciding factor. Check whether they're equal after rounding
+* down the arcn mtime to the precision of the target path.
+*/
+ res = pathconf(path, _PC_TIMESTAMP_RESOLUTION);
+ if (res == -1)
+ return 0;
+
+ /* nanosecond resolution? previous comparisons were accurate */
+ if (res == 1)
+ return 0;
+
+ /* common case: second accuracy */
+ if (res == 10)
+ return arcn->sb.st_mtime <= sb.st_mtime;
+
+ if (res < 10) {
+ struct timespec ts = arcn->sb.st_mtim;
+ ts.tv_nsec = (ts.tv_nsec / res) * res;
+ return timespeccmp(, _mtim, <=);
+ } else {
+ /* not a POSIX compliant FS */
+ res /= 10;
+ return ((arcn->sb.st_mtime / res) * res) <= sb.st_mtime;
+ return arcn->sb.st_mtime <= ((sb.st_mtime / res) * res);
}
-
- if (ctime_flag && mtime_flag)
- return (timespeccmp(>sb.st_mtim, >st_mtim, <=) &&
- timespeccmp(>sb.st_ctim, >st_ctim, <=));
- else if (ctime_flag)
- return (timespeccmp(>sb.st_ctim, >st_ctim, <=));
- else
- return (timespeccmp(>sb.st_mtim, >st_mtim, <=));
}
/*
@@ -842,14 +880,12 @@ copy(void)
/*
* if existing file is same age or newer skip
*/
- res = lstat(dirbuf, );
- *dest_pt = '\0';
-
- if (res == 0) {
+ if (cmp_file_times(uflag, Dflag, arcn, dirbuf)) {
+ *dest_pt = '\0';
ftree_skipped_newer(arcn);
- if (cmp_file_times(uflag, Dflag, arcn, ))
- continue;
+ continue;
}
+ *dest_pt = '\0';
}
/*
Re: pax and ext2fs
On Tue, May 14, 2024 at 11:59 AM Walter Alejandro Iglesias wrote: > Hi Philip, > > On Tue May 14 19:40:04 2024 Philip Guenther wrote: > > If you like, you could try the following patch to pax to more gracefully > > handle filesystems with time resolution more granular than nanoseconds. > > After applying your patch, as I'd done before reporting the issue, I > sycronized my home directory to an external ext2fs drive with the > command showed by the man page: > > $ pax -rw -v -Z -Y source target > > This time only one file stays updating again an again, a soft link I > have in my ~/bin folder of /usr/local/bin/prename. I think you've managed to hit a spot where the POSIX standard doesn't provide a way for a program to find the information it needs to do its job correctly. I've filed a ticket there https://austingroupbugs.net/view.php?id=1831 We'll see if my understanding of pathconf() is incorrect or if someone has a great idea for how to get around this... Philip Guenther
Re: viomb0 unable to allocate256 physmem pages, error 12
viomb is a driver that tries to support OpenBSD, as a VM guest, responding to a request from the VM host to stop using so much physical memory. That log message indicates that the kernel couldn't easily free up that much physical memory, sorry! The VM host is, of course, free to decide to just page out whatever memory it wants instead, possibly resulting in thrashing: running a VM setup oversubscribed for memory is a great way to be frustrated and hate computers. How can you make that message go away? Provision your VM setup with enough memory that it's not over subscribed, or at least so that the OpenBSD guest(s) isn't the one being asked to slim itself (possibly by giving it *less* but _reserved_ memory, so that the VM host never tries to shrink its usage). Philip Guenther On Tue, May 14, 2024 at 4:16 PM F Bax wrote: > I'm not a coder; but I found source for viomb; which > calls uvm_pglistalloc; which calls uvm_pmr_getpages which mentions ENOMEM: > > https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/uvm/uvm_pmemrange.c?rev=1.66=text/plain > There I found this comment: > * fail if any of these conditions is true: > * [1] there really are no free pages, or > * [2] only kernel "reserved" pages remain and > *the UVM_PLA_USERESERVE flag wasn't used. > * [3] only pagedaemon "reserved" pages remain and > *the requestor isn't the pagedaemon nor the syncer. > > Unsure how I might use this information to get rid of the previously > mentioned error message.. > > On Tue, May 14, 2024 at 2:28 PM Peter J. Philipp > wrote: > >> On Tue, May 14, 2024 at 01:58:18PM -0400, F Bax wrote: >> > Recently installed 7.5 amd64 in qemu VM (8G RAM) under proxmox. See this >> > message many times on console and dmesg. >> > >> > viomb0 unable to allocate 256 physmem pages, error 12 >> > >> > What does this mean? How to resolve this issue? >> >> Hi, >> >> When you see "error " it's good to look up the manpage on errno. >> Under number 12 it says: ENOMEM "Cannot Allocate Memory". But look for >> yourself for a deeper explanation. Also if you want to hunt for this >> errno >> in the code you would most likely grep for ENOMEM. >> >> Best Regards, >> -pjp >> >> -- >> ** all info about me: lynx https://callpeter.tel, dig loc >> delphinusdns.org ** >> >>
Re: Bgpd multipath conf
On 2024-05-14, Marco Agostani wrote:
> I try to setup an openbgpd setup involving multipath configuration ...with =
> no success.
...
> neighbor $GW01 {
> descr "bgp#1"
> announce IPv4 unicast
> announce add-path recv yes
> set localpref 110
> }
This just announces the add-path BGP capability.
> #bgpctl sh rib 172.18.180.0/24
>
> flags: * =3D Valid, > =3D Selected, I =3D via IBGP, A =3D Announced,
>S =3D Stale, E =3D Error
> origin validation state: N =3D not-found, V =3D valid, ! =3D invalid
> aspa validation state: ? =3D unknown, V =3D valid, ! =3D invalid
> origin: i =3D IGP, e =3D EGP, ? =3D Incomplete
>
> flags vs destination gateway lpref med aspath origin
> *>N-? 172.18.180.0/24 10.0.1.241110 0 14381 i
> *mN-? 172.18.180.0/24 10.0.1.245110 0 14381 i
>
> Show me two routes one marked with multipath
>
> But in fib I see only one route
>
> #bgpctl sh fib 172.18.180.0/24
>
> flags: B =3D BGP, C =3D Connected, S =3D Static
>N =3D BGP Nexthop reachable via this route
>r =3D reject route, b =3D blackhole route
> flags prio destination gateway
> B 48 172.18.180.0/24 10.0.1.241
...
> What I miss here ??
bgpd does allow add-path and having multiple paths to a prefix in the
RIB (e.g. perhaps useful on a route-server) but it does not handle
adding multiple paths for the same prefix to the FIB.
Bgpd multipath conf
Hello guys,
I try to setup an openbgpd setup involving multipath configuration ...with no
success.
My bgpd.conf is like that
prefix-set privnetworks {
10.55.0.0/16
10.60.0.0/16
172.16.0.0/12
}
log updates
network 10.240.0.0/16
group "eBGP" {
remote-as $AS1
neighbor $GW01 {
descr "bgp#1"
announce IPv4 unicast
announce add-path recv yes
set localpref 110
}
neighbor $GW02 {
descr "bgp#2"
announce IPv4 unicast
announce add-path recv yes
set localpref 110
}
}
match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
deny quick from group eBGP prefix 0.0.0.0/0
Deny out internal route
deny quick from group eBGP prefix 10.240.0.0/16 or-longer
##allow private
allow quick from group eBGP prefix-set privnetworks or-longer set rtlabel
PRIVNET
allow quick to group eBGP prefix 10.240.0.0/16
deny quick from any
#bgpctl sh rib 172.18.180.0/24
flags: * = Valid, > = Selected, I = via IBGP, A = Announced,
S = Stale, E = Error
origin validation state: N = not-found, V = valid, ! = invalid
aspa validation state: ? = unknown, V = valid, ! = invalid
origin: i = IGP, e = EGP, ? = Incomplete
flags vs destination gateway lpref med aspath origin
*>N-? 172.18.180.0/24 10.0.1.241110 0 14381 i
*mN-? 172.18.180.0/24 10.0.1.245110 0 14381 i
Show me two routes one marked with multipath
But in fib I see only one route
#bgpctl sh fib 172.18.180.0/24
flags: B = BGP, C = Connected, S = Static
N = BGP Nexthop reachable via this route
r = reject route, b = blackhole route
flags prio destination gateway
B 48 172.18.180.0/24 10.0.1.241
Confirmed by route
#route -n get 172.18.180.0/24
route to: 172.18.180.0
destination: 172.18.180.0
mask: 255.255.255.0
gateway: 10.0.1.241
interface: sec7130
if address: 10.0.1.242
priority: 48 (bgp)
flags:
label: PRIVNET
use mtuexpire
0 0 0
sockaddrs:
Multipath is enabled
# sysctl net.inet.ip.multipath
net.inet.ip.multipath=1
and static routes with -mpath option are setup correctly
What I miss here ??
Cheers
Marco
Caterpillar: Confidential Green
Re: What software to debugging and analyzing C?
On Tue, May 14, 2024 at 05:19:43AM -0300, Crystal Kolipe wrote: > On Sun, May 12, 2024 at 10:26:55PM +0200, Tomasz Rola wrote: > > I am sure gdb has some merits but for whatever C programs I wrote so > > far, a much more useful debugging technique was putting printf in > > right places and isolate the problem, and after that doing some mental > > work to actually understand why this seemingly correct line does > > something so wrong. > > Exactly. What you describe is likely the best method to fully understand the > code, what it's supposed to do and what it actually does, and by extension [...] Yes, I guess. > > Besides, all debuggers introduce their own perturbation and thus > > certain classes of error will be very hard to catch with them, if > > ever. > > But you do realise that adding printf() calls to the code can also change, > for example, the memory layout that the compiler uses, so certain memory > allocation bugs might become more or less easily triggerable? No, this did not occurred to me, at least not in such explicit way. Albeit somewhere deep I realise that program execution can change, if for example two "not related" lines of code switch places etc. (because of optimisation, for example). Before you pointed it out above I considered printf to be almost non-intrusive way of debugging. Thanks! -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: Could OpenBSD use some compute?
On May 14 12:24:28, romand...@gmail.com wrote: > If someone had spare capacity, (say, in their homelab, ~80% available, > about same amount 10k/mon would buy in AWS spot instances), and wanted to > share it with the open source community in general and OpenBSD devs in > particular, and were willing to do some ops and eat the electricity bill, > how could they go about putting all those to good use? > Hosting mirrors comes to mind, maybe some build/test server? Fuzzy testing > dev branches? "I'm asking for a friend"
Re: viomb0 unable to allocate256 physmem pages, error 12
I'm not a coder; but I found source for viomb; which calls uvm_pglistalloc; which calls uvm_pmr_getpages which mentions ENOMEM: https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/uvm/uvm_pmemrange.c?rev=1.66=text/plain There I found this comment: * fail if any of these conditions is true: * [1] there really are no free pages, or * [2] only kernel "reserved" pages remain and *the UVM_PLA_USERESERVE flag wasn't used. * [3] only pagedaemon "reserved" pages remain and *the requestor isn't the pagedaemon nor the syncer. Unsure how I might use this information to get rid of the previously mentioned error message.. On Tue, May 14, 2024 at 2:28 PM Peter J. Philipp wrote: > On Tue, May 14, 2024 at 01:58:18PM -0400, F Bax wrote: > > Recently installed 7.5 amd64 in qemu VM (8G RAM) under proxmox. See this > > message many times on console and dmesg. > > > > viomb0 unable to allocate 256 physmem pages, error 12 > > > > What does this mean? How to resolve this issue? > > Hi, > > When you see "error " it's good to look up the manpage on errno. > Under number 12 it says: ENOMEM "Cannot Allocate Memory". But look for > yourself for a deeper explanation. Also if you want to hunt for this errno > in the code you would most likely grep for ENOMEM. > > Best Regards, > -pjp > > -- > ** all info about me: lynx https://callpeter.tel, dig loc > delphinusdns.org ** > >
Re: Localnet Hacking
On Tue, May 14, 2024 at 01:54:52AM +0200, Peter J. Philipp wrote: > Hi, > > A few more people responded, I'm falling behind on priorities though because Hi again, https://mainrechner.de/Buecher2024/batch1.png Here is the first batch that will be mailed out on Friday at the latest. I still have to find cartons for these. We have Sweden, Israel, Turkeye, Germany, USA, Canada, Spain, Australia, with some of them double or triple. Thanks to all. If you waould like to be on the second batch which goes out Friday the 24th please start writing me in private starting Saturday the 18th. I have my hands full with this and life in general. Thanks to all that participated. -pjp -- ** all info about me: lynx https://callpeter.tel, dig loc delphinusdns.org **
Re: pax and ext2fs
Hi Philip, On Tue May 14 19:40:04 2024 Philip Guenther wrote: > If you like, you could try the following patch to pax to more gracefully > handle filesystems with time resolution more granular than nanoseconds. After applying your patch, as I'd done before reporting the issue, I sycronized my home directory to an external ext2fs drive with the command showed by the man page: $ pax -rw -v -Z -Y source target This time only one file stays updating again an again, a soft link I have in my ~/bin folder of /usr/local/bin/prename. I tried the command Stuart Henderson taught me in that file: $ stat -f %Fm /usr/local/bin/prename 1713451867.0 ... no sub-second timestamp, like happens when I run the same stat comand with the files in the ext2fs drive. I ran stat with other files under /usr/local, same result, I end noticing that /usr/local is the only partition mounted with the wxallowed option. I wish my guessing info will be useful. :-) Let me know what more I can do to help. > The whitespace will presumably be mauled by gmail so use patch's -l option. Some lines in the diff arrived wrapped but I corrected them and could apply the patch. Gmail has an well hidden option :-), if you open your account from your browser you can configure it to send in plain text. You have to click in Compose, then in the compose window go to the last icon in the bottom right (with a vertical ellipis) hovering with your mouse says "More options", click and you'll see there the "Plain Text" option, select it and the option stays saved. I don't use gmail since a long time, I had to investigate this tired of friends sending me the HTML copy of all their messages. > > Philip Guenther > > -- Walter
Re: viomb0 unable to allocate256 physmem pages, error 12
On Tue, May 14, 2024 at 01:58:18PM -0400, F Bax wrote: > Recently installed 7.5 amd64 in qemu VM (8G RAM) under proxmox. See this > message many times on console and dmesg. > > viomb0 unable to allocate 256 physmem pages, error 12 > > What does this mean? How to resolve this issue? Hi, When you see "error " it's good to look up the manpage on errno. Under number 12 it says: ENOMEM "Cannot Allocate Memory". But look for yourself for a deeper explanation. Also if you want to hunt for this errno in the code you would most likely grep for ENOMEM. Best Regards, -pjp -- ** all info about me: lynx https://callpeter.tel, dig loc delphinusdns.org **
viomb0 unable to allocate256 physmem pages, error 12
Recently installed 7.5 amd64 in qemu VM (8G RAM) under proxmox. See this message many times on console and dmesg. viomb0 unable to allocate 256 physmem pages, error 12 What does this mean? How to resolve this issue?
Re: pax and ext2fs
If you like, you could try the following patch to pax to more gracefully
handle filesystems with time resolution more granular than nanoseconds.
The whitespace will presumably be mauled by gmail so use patch's -l option.
Philip Guenther
Index: ar_subs.c
===
RCS file: /data/src/openbsd/src/bin/pax/ar_subs.c,v
diff -u -p -r1.51 ar_subs.c
--- ar_subs.c 10 Jul 2023 16:28:33 - 1.51
+++ ar_subs.c 14 May 2024 17:19:15 -
@@ -146,23 +146,59 @@ list(void)
}
static int
-cmp_file_times(int mtime_flag, int ctime_flag, ARCHD *arcn, struct stat
*sbp)
+cmp_file_times(int mtime_flag, int ctime_flag, ARCHD *arcn, const char
*path)
{
struct stat sb;
+ long res;
- if (sbp == NULL) {
- if (lstat(arcn->name, ) != 0)
- return (0);
- sbp =
+ if (path == NULL)
+ path = arcn->name;
+ if (lstat(path, ) != 0)
+ return (0);
+
+ /*
+* The target (sb) mtime might be rounded down due to the
limitations
+* of the FS it's on. If it's strictly greater or we don't care
about
+* mtime, then precision doesn't matter, so check those cases first.
+*/
+ if (ctime_flag && mtime_flag) {
+ if (timespeccmp(>sb.st_mtim, _mtim, <=))
+ return timespeccmp(>sb.st_ctim, _ctim,
<=);
+ if (!timespeccmp(>sb.st_ctim, _ctim, <=))
+ return 0;
+ /* <= ctim, but >= mtim */
+ } else if (ctime_flag)
+ return timespeccmp(>sb.st_ctim, _ctim, <=);
+ else if (timespeccmp(>sb.st_mtim, _mtim, <=))
+ return 1;
+
+ /*
+* If we got here then the target arcn > sb for mtime *and* that's
+* the deciding factor. Check whether they're equal after rounding
+* down the arcn mtime to the precision of the target path.
+*/
+ res = pathconf(path, _PC_TIMESTAMP_RESOLUTION);
+ if (res == -1)
+ return 0;
+
+ /* nanosecond resolution? previous comparisons were accurate */
+ if (res == 1)
+ return 0;
+
+ /* common case: second accuracy */
+ if (res == 10)
+ return arcn->sb.st_mtime <= sb.st_mtime;
+
+ if (res < 10) {
+ struct timespec ts = arcn->sb.st_mtim;
+ ts.tv_nsec = (ts.tv_nsec / res) * res;
+ return timespeccmp(, _mtim, <=);
+ } else {
+ /* not a POSIX compliant FS */
+ res /= 10;
+ return ((arcn->sb.st_mtime / res) * res) <= sb.st_mtime;
+ return arcn->sb.st_mtime <= ((sb.st_mtime / res) * res);
}
-
- if (ctime_flag && mtime_flag)
- return (timespeccmp(>sb.st_mtim, >st_mtim, <=) &&
- timespeccmp(>sb.st_ctim, >st_ctim, <=));
- else if (ctime_flag)
- return (timespeccmp(>sb.st_ctim, >st_ctim, <=));
- else
- return (timespeccmp(>sb.st_mtim, >st_mtim, <=));
}
/*
@@ -842,14 +878,12 @@ copy(void)
/*
* if existing file is same age or newer skip
*/
- res = lstat(dirbuf, );
- *dest_pt = '\0';
-
- if (res == 0) {
+ if (cmp_file_times(uflag, Dflag, arcn, dirbuf)) {
+ *dest_pt = '\0';
ftree_skipped_newer(arcn);
- if (cmp_file_times(uflag, Dflag, arcn, ))
- continue;
+ continue;
}
+ *dest_pt = '\0';
}
/*
On Thu, May 2, 2024 at 6:54 AM Walter Alejandro Iglesias
wrote:
> On Thu, 2 May 2024 12:03:10, Stuart Henderson wrote
> > I don't have a suitable filesystem handy to test, but does OpenBSD's
> > implementation of ext2fs support sub-second timestamps?
> >
> > stat -f %Fm $filename
> >
> > If not, that's a probable explanation for the difference in behaviour.
> > You could probably confirm by forcing timestamps with no nanosecond
> > components, e.g. touch -t mmddhhmm.ss $filename, or copy to ext2fs
> > and back again.
>
> $ doas mount -t ext2fs /dev/sd0i /mnt
> $ touch ~/test.txt
> $ cp ~/test.txt /mnt
> $ stat -f %Fm /mnt/test.txt
> 1714657214.0
> $ cp ~/test.txt /mnt
> $ stat -f %Fm /mnt/test.txt
> 1714657409.0
> 癘m
>
Could OpenBSD use some compute?
If someone had spare capacity, (say, in their homelab, ~80% available, about same amount 10k/mon would buy in AWS spot instances), and wanted to share it with the open source community in general and OpenBSD devs in particular, and were willing to do some ops and eat the electricity bill, how could they go about putting all those to good use? Hosting mirrors comes to mind, maybe some build/test server? Fuzzy testing dev branches? If not compute, mb smth else, like storage/GPU/much-ram? Thanks in advance, Roman.
Re: Favorite configuration and system replication tools?
On Tue May 14 18:11:16 2024 Страхиња Радић wrote:
> Antipatterns are bad. I don't mean the ellipsis in `ls -l ...`. I mean
> things like
>
> cat file | grep hello | cat | sed 's/hello/world/g' | cat - > output
>
> for file in `echo `ls *` `; do echo $file; done
>
> ls -l | awk '{ print $5 }' # different things with different ls'es
> # under different locales, and on
> # different systems, with differently
> # named files
>
> which are something a novice will see and adopt, especially when it is
> not even communicated as a "sketch", and a seasoned user of shell will
> just be annoyed with.
This recalls me again the quote of the article you linked:
"..., unless extreme portability is more important..."
Let's make it short. Would you guarantee that your shell scripts work
on any unix-like system?
And here I abandon the discussion, needless to say that I appreciate all
your advices.
Greetings!
Re: Favorite configuration and system replication tools?
Дана 24/05/14 11:52AM, Walter Alejandro Iglesias написа:
> I learned about the convenience of adding the '-r' option in the "while
> read" loop many years ago when I was writing a script to convert roff to
> html, the problem aroused with the backslash in roff comments (.\").
That's more or less the point that answer is trying to make: you should
use the right tool for the job. Shell command language is not fit to
create parsers or interpreters in. Attempts can be made, but some
corner will be hit sooner or later. When that happens, it is time to
rethink if the problem would be better attacked by a full-fledged
program in a general programming language. Parsers, interpreters and
compilers are examples of this.
> Immediately, some "experts" started to point me "holes" in my
> script, when what I posted was just a sketch. In that case disregarding
> shell scripting wasn't useful as an argument since they were proposing
> fail2ban.sh (a clear example of using shell scripting for something
> complicated. ;-))
Antipatterns are bad. I don't mean the ellipsis in `ls -l ...`. I mean
things like
cat file | grep hello | cat | sed 's/hello/world/g' | cat - > output
for file in `echo `ls *` `; do echo $file; done
ls -l | awk '{ print $5 }'# different things with different ls'es
# under different locales, and on
# different systems, with differently
# named files
which are something a novice will see and adopt, especially when it is
not even communicated as a "sketch", and a seasoned user of shell will
just be annoyed with.
I've also seen system shell scripts from major distributions of
GNU/Linux and some mainstream software projects which feature examples
of this kind, so not even their developers are immune to it. More than
necessary amounts of bugs and security holes are then not a surprise at
all.
Re: What software to debugging and analyzing C?
On Tue, May 14, 2024 at 05:19:43AM -0300, Crystal Kolipe wrote: > On Sun, May 12, 2024 at 10:26:55PM +0200, Tomasz Rola wrote: > > I am sure gdb has some merits but for whatever C programs I wrote so > > far, a much more useful debugging technique was putting printf in > > right places and isolate the problem, and after that doing some mental > > work to actually understand why this seemingly correct line does > > something so wrong. > > Exactly. What you describe is likely the best method to fully understand the > code, what it's supposed to do and what it actually does, and by extension > avoid making the same coding mistakes in the future. Finding and fixing a > single error with gdb doesn't have the same educational benefit, nor in > many cases such a guarantee that other nearby bugs have also been noticed. > > > Besides, all debuggers introduce their own perturbation and thus > > certain classes of error will be very hard to catch with them, if > > ever. > > But you do realise that adding printf() calls to the code can also change, > for example, the memory layout that the compiler uses, so certain memory > allocation bugs might become more or less easily triggerable? Yes, I do realize that printf has that flaw. I also program some in Perl. print, warn, die, etc. can sometimes help, but often they don't. Carefully studying or just trying to rewrite a section of code from scratch is the only solution. Many years ago I wrote a trivial Perl script wrong. It very slowly grabbed more and more memory until it crashed the server about every two days. After very carefully watching, I figured out it was my script and I fixed a rather silly bug. I'll never forget that experience. -- Regards, Chris Bennett "Who controls the past controls the future. Who controls the present controls the past." George Orwell - 1984
Re: What software to debugging and analyzing C?
On Tue May 14 11:40:42 2024 Tomasz Rola wrote: > I am sure gdb has some merits but for whatever C programs I wrote so > far, a much more useful debugging technique was putting printf in > right places and isolate the problem, I got used to doing this too. I started doing it intuitively, I'm self-taught (and I'm certainly not an expert).
Re: Favorite configuration and system replication tools?
On Tue May 14 11:11:33 2024 wrote: > When `while ... read ...` idiom is used, it is advisable to clear IFS > to turn off field splitting, and use -r to avoid interpretation of > backslash sequences in the input: > > while IFS= read -r dir; do # ... > > Back to parsing the output of ls(1) (also applicable to parsing the > output of find(1), or globs), there is an indepth analysis of the > problem at [1]. The accepted answer concludes that perhaps shell > command language is not the right tool for the job, and a more > sophisticated language should be used instead. While I don't agree with > the author's choice of Python, any language supporting opendir(3), > readdir(3) or equivalent functions will suffice. > > [1]: > https://unix.stackexchange.com/questions/128985/why-not-parse-ls-and-what-to-do-instead > Let's start for what the first answer in that forum put in bold: Bourne shell is a bad language. It should not be used for anything complicated, unless extreme portability is more important than any other factor (e.g. autoconf). If you analyze that statement, depending on the case, it may have sense or not at all. I learned about the convenience of adding the '-r' option in the "while read" loop many years ago when I was writing a script to convert roff to html, the problem aroused with the backslash in roff comments (.\"). When I post an example of a shell script in some forum or mailing list I post an sketch, assuming others will use it as a example and write themselves their own solution. Who won't be able to overcome issues like the above are those who aren't familiar or trained in that language in particular. For me (as I think it should be for any unix user) shell scripting is mainly the way a "use" the computer, it's not a "programmers" language, something you use to write whole applications of the kind "Push this button and relax, I'll do the job". I have a hundred of dirty shell scripts in my ~/bin directory that if you examine them you'll find many dumb errors, but mainly they do the job. I don't know anything about python (I don't like it), but I bet that if you analyze python, or C or Perl, you'll also find inconsistencies you'll have to workaround as with shell scripting. Nothing is perfect when you see it in detail. Many years ago I posted in some linux forum an example of a shell script to blacklist IPs in a web-mail server. My intention was encouraging users to not follow the MSWin approach, I mean downloading some 3rd party tool instead of learning what the system already has to offer. And I remember myself proposing and giving solutions with rsync in that same forum to someone asking for a mirror capable synchronizing tool. This, and *learning to do things by yourself* (even if your program isn't as good the one you download or isn't good at at all), are the fundamental tendencies I always defend since ARE THE REASON OF EXISTENCE OF FOSS. Immediately, some "experts" started to point me "holes" in my script, when what I posted was just a sketch. In that case disregarding shell scripting wasn't useful as an argument since they were proposing fail2ban.sh (a clear example of using shell scripting for something complicated. ;-)) Summarizing, my motivation was triggered by the topic of the thread and the way it was raised by the OP. -- Walter
Re: What software to debugging and analyzing C?
On Sun, May 12, 2024 at 10:26:55PM +0200, Tomasz Rola wrote: > I am sure gdb has some merits but for whatever C programs I wrote so > far, a much more useful debugging technique was putting printf in > right places and isolate the problem, and after that doing some mental > work to actually understand why this seemingly correct line does > something so wrong. Exactly. What you describe is likely the best method to fully understand the code, what it's supposed to do and what it actually does, and by extension avoid making the same coding mistakes in the future. Finding and fixing a single error with gdb doesn't have the same educational benefit, nor in many cases such a guarantee that other nearby bugs have also been noticed. > Besides, all debuggers introduce their own perturbation and thus > certain classes of error will be very hard to catch with them, if > ever. But you do realise that adding printf() calls to the code can also change, for example, the memory layout that the compiler uses, so certain memory allocation bugs might become more or less easily triggerable?
Re: Localnet Hacking
On Tue, May 14, 2024 at 01:54:52AM +0200, Peter J. Philipp wrote: > Hi, > > A few more people responded, I'm falling behind on priorities though because > I am very close to cracking AES-128 I have reduced it to a complexity of > 2 ^ 64. However I have some old code to get the first 32 bits identified but > I want to find a cleaner way. I'll upload my code to the https://centroid.eu > misc repo tomorrow. Once I have the crib for the first 32 bits in a sureshot > everything falls into place and the complexity falls to 2 * (2 ^ 32). I guess > that's the same a 2 ^ 33. Well my sugar high is over. It was good for a week or two. I spotted the error in my logic. I'll still be working on this tough. I passed rk into gosh() and used it.. I totally oversaw that. Best Regards, -pjp
Re: Localnet Hacking
On Sat, May 11, 2024 at 10:35:38AM +0200, Peter J. Philipp wrote: > On Sat, May 11, 2024 at 08:45:45AM +0200, Peter J. Philipp wrote: > > Contact me privately if you would like a batch with what you like. I'll > > make note on that webpage of what's given away. Offer ends July 1st of this > > year. > > Three books have already been given away. They went to Finland. Look for > a marking of a flag beside the name of the title of the book. > > Also if I may interest some people: The Java book is autographed by Ian F. > Darwin who is also on this list. Also the 4.4BSD book which is quite beaten > up was autographed by 3 of the 4 authors at BSDCon 2000. They were everyone > other than John Quarterman. Maybe I'll run into him one day but then I'll > be missing 3 signatures hehe. > > -pjp Hi, A few more people responded, I'm falling behind on priorities though because I am very close to cracking AES-128 I have reduced it to a complexity of 2 ^ 64. However I have some old code to get the first 32 bits identified but I want to find a cleaner way. I'll upload my code to the https://centroid.eu misc repo tomorrow. Once I have the crib for the first 32 bits in a sureshot everything falls into place and the complexity falls to 2 * (2 ^ 32). I guess that's the same a 2 ^ 33. It's kept me up most of the day and night today as it's exciting work. I promise to send your books by friday as wednesday and thursday are booked for me too. Also there has been close to 10 people now, for any new request I plea you to wait until next week. This is an exciting May. (Are you ready for the non-quantum cryptography apocalypse?, I'm starting to believe we're in a game like tron or something.. let's work together) -pjp -- ** all info about me: lynx https://callpeter.tel, dig loc delphinusdns.org **
Wacom pen tip pressure
Hello,
I tried to use wacom CTL-672, it works. Except pen tip pressure,
As stated in man uwacom.
On linux, pen tip pressure also not working if xf86-input-wacom
not installed.
In /src/sys/dev/hid/hidms.c:
...
case HID_USAGE2(HUP_WACOM | HUP_DIGITIZERS, HUD_TIP_PRESSURE):
DPRINTF(("Stylus usage pressure set\n"));
ms->sc_loc_z = h.loc;
ms->sc_tsscale.minz = h.logical_minimum;
ms->sc_tsscale.maxz = h.logical_maximum;
ms->sc_flags |= HIDMS_Z;
break;
...
Seems like it can read tip pressure, but need more processing,
isn't it? (correct me if i wrong).
So, to make pen tip pressure works, do we need uwacom + X driver
(xf86-input-wacom) to handle advanced process?.
Is xf86-input-wacom need to be ported?.
Or just improve uwacom?.
Or both?
It would be good if pen tip pressure can be used on openbsd.
Re: What software to debugging and analyzing C?
On Mon, May 13, 2024 at 08:24:38AM +0200, Janne Johansson wrote: > pkg_add llvm and run "scan-build" on your code, then you get a quite > thorough analysis on what potential error code paths it detects, with > fancy webpages to go along with the explanations for each found issue: > > http://c66.it.su.se:8080/obsd/scan-build-2019-10-10-202112-79522-1/report-3f2f00.html#EndPath > > It's not 100% perfect of course, but it still is a neat way to point > out where in the code you may need to make an extra effort to cover > corner cases. > > > I also wouldn't mind any other useful tips that might not be software. > > Any help very appreciated. > > Perhaps this fuzzing guide helps a bit getting programs to run better? > https://undeadly.org/cgi?action=article=20150121093259 Thank you and to the others replying. -- Regards, Chris Bennett "Who controls the past controls the future. Who controls the present controls the past." George Orwell - 1984
Re: gmake compile of python3.12 crashes on openBSD 7.5 but not on openBSD 7.4
On 2024-05-12, Sandeep Gupta wrote: > ./Tools/scripts/pydoc3 > build/scripts-3.12/pydoc3.12 > Illegal instruction (core dumped) > > I am unable to find a proper debugger into which to load the python.core > generated after core dump, so can't provide any useful debug info. pkg_add gdb and use the 'egdb' command.
Re: https://twitter.com/openbsd
On Sunday, May 12th, 2024 at 11:04 AM, T.J. Townsend > Wasting everyone's time by complaining on a mailing list that > we didn't post a tweet seems a little petty too. Anyway I just > blocked you from the OpenBSD account, so that should make life > easier for everyone going forward. Funny how the others have started rambling on about learning to type and you haven't had another one of your petty power trips about everyone's time being wasted. Are you feeling OK? Do you need a pettiness power-up? I bet you do! I did note I forgot to format the width for my last few emails, I forgot about protonmail simply sending it holus-bolus. This one should be OK. Oh, get on-topic!!! I learnt to type long enough ago that I don't remember actually learning, then one day I just stopped looking at the keyboard and never noticed the transition. Those kids will get bored of those typewriters in days, weeks at most. tux2bsd
Re: What software to debugging and analyzing C?
> I found a YouTube channel LowLevelLearning that covers various > programming languages in a manner that I find particularly helpful and > clear. For example comparing C and assembly on the same code is superb. > > In a short, he recommended valgrind to help finding memory leaks. > Other than splint and gdb, what other software is useful for working > with C? pkg_add llvm and run "scan-build" on your code, then you get a quite thorough analysis on what potential error code paths it detects, with fancy webpages to go along with the explanations for each found issue: http://c66.it.su.se:8080/obsd/scan-build-2019-10-10-202112-79522-1/report-3f2f00.html#EndPath It's not 100% perfect of course, but it still is a neat way to point out where in the code you may need to make an extra effort to cover corner cases. > I also wouldn't mind any other useful tips that might not be software. > Any help very appreciated. Perhaps this fuzzing guide helps a bit getting programs to run better? https://undeadly.org/cgi?action=article=20150121093259 -- May the most significant bit of your life be positive.
Re: Favorite configuration and system replication tools?
Andreas Kähäri writes:
i'm not sure why you're addressing this to me, as i'm not the
OP.
It's addressed to the thread in general.
Your response quoted me, then made use of the word 'you'. Which
you - and yes, i mean you, Andreas, specifically - have again done
below:
That said, yes, minimising the extent to which certain
non-'word'
characters
(i.e. roughly the POSIX 'alnum' class as described in
re_format(7))
_can_
make it easier to programatically do certain tasks which are
restricted by
the long and messy history of C and Unix development. Given
that
i've been
using computers for a few decades, i still instinctively don't
use
spaces in
filenames, even though they're very much allowed. But of
course,
that's not
what most of the world does, and this is an example of trying
to
work out
what the best tradeoffs might be when dealing with the
messiness of
the real
world.
Alexis.
With rsync(1):
rsync -n -aim --delete-excluded \
--include-from=list \
--include='*/' \
--exclude='*' \
source/ target
This would read your inclusion patterns from the file "list" (it
is
assumed that directories are entered as "dirname/***", which
matches the
name "directory" and all its content), include any directory,
and then
finaly exclude anything not already included. The matched names
would
be synchonised from beneath "source" to "target", and excluded
names
would be deleted from the target. With "-m", we don't keep
directories
at the target that ends up being empty.
You - by which i mean, you, Andreas, specifically - have quoted me
_at length_, in an email with _my email address_ in the To header,
before immediately making remarks not related to the text of mine
you quoted (which was instead related to the `-r` option to `read`
in the context of processing filenames, and which Страхиња has
addressed to my satisfaction).
If you - by which i mean, you, Andreas, specifically - want to use
'you' in the general sense of 'one' ("This would read one's
inclusions patterns ..."), which is certainly fair enough, then
please don't quote unrelated text _from me_ when doing so. If, for
some bizarre reason, your mail client prevents the removal of
unrelated quotes from others, use a better mail client.
In any case, from this point forward, please do not include my
email address in the delivery-related headers of any further
replies to this thread.
Alexis.
Re: What software to debugging and analyzing C?
On Sun, May 12, 2024 at 11:51:32AM -0700, Chris Bennett wrote: > I found a YouTube channel LowLevelLearning that covers various > programming languages in a manner that I find particularly helpful and > clear. For example comparing C and assembly on the same code is superb. > > In a short, he recommended valgrind to help finding memory leaks. > Other than splint and gdb, what other software is useful for working > with C? > I also wouldn't mind any other useful tips that might not be software. > Any help very appreciated. I am sure gdb has some merits but for whatever C programs I wrote so far, a much more useful debugging technique was putting printf in right places and isolate the problem, and after that doing some mental work to actually understand why this seemingly correct line does something so wrong. This approach does not look sexy enough to show it on y-t, so I guess there will not be a movie showing it. Besides, all debuggers introduce their own perturbation and thus certain classes of error will be very hard to catch with them, if ever. It also sometimes happened to me, that debugger pointed to wrong place, where the error supposedly happened. Very wrong place - like GUI code when in fact the bug was in database communication. I think it all becomes even more funky if you start playing with multithreaded apps and languages which come with threads built-in, under the hood. All of those things happened many years ago - perhaps debuggers improved, I have no idea. So, I suggest that you do: man tee and after that: ./yourcode 2>&1 | tee log.txt and: less log.txt -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_r...@bigfoot.com **
Re: https://twitter.com/openbsd
On 13/5/24 04:40, Chris Bennett wrote: I saw a news bit yesterday that in one town, all of the school children are buying old fashioned typewriters to break their link to computers and do things the old fashioned way. +1 to them. I prefer real text on paper myself. I learn things much better that way. Makes a lot more sense for teaching typing skills actually. In my day, my school was using i586-class machines running Windows 95 and Office 97. Seemed like a reasonable choice, but Word 97 was overkill for the job, and actually had a few anti-features which were problematic in exams: it only took an arsehole student a brief moment to stab F7 on your keyboard just as a teacher came around the corner to get someone disqualified from the typing exam… since use of the spell checker was forbidden in that context. Typewriters had no such luxuries. Mind you, neither did "simpler" word processors and text editors, so chalk one up for educator tunnel vision. I applaud the kids there for thinking outside the box. -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.
Re: Favorite configuration and system replication tools?
On Sun May 12 21:50:12 2024 Martin Schröder wrote: > > If a line begins with "- " (dash, space) or "+ " (plus, space), > then the type of rule is being explicitly specified as an exclude > or an include (respectively). Any rules without such a prefix are > taken to be an include. I'd read the man page. What I understand from this paragraph is that you have to include in the list *all* the files. How convenient! :-) > > > Coming back to the topic of this thread: Yeah, because I'm talking about football. > > Best > Martin > > >
Re: Why /var/www/run instead of /var/run for web services
> I suspect that it is because a web service might change its root > directory to /var/www using chroot(2), > Can anyone confirm or deny my assumption? right, www is chrooted. -Dan
Re: What software to debugging and analyzing C?
On May 12 11:51:32, cpb_m...@bennettconstruction.us wrote: > In a short, he recommended valgrind to help finding memory leaks. man malloc
Re: What software to debugging and analyzing C?
Otto Moerbeek thought me this: First compile your program with debug symbols (and, conveniently, without optimization settings.) $ DEBUG="-g -O0" make Then: $ MALLOC_OPTIONS=D ktrace -tu $ kdump -u malloc kdump will though you lines like this: 0x34f10a4b153 20480 1 20480 addr2line -e /usr/lib/libc.so.97.1 0x4d153 0x34f10a96470 410576 25 16423 addr2line -e /usr/src/usr.bin/ 0x98470 If you compiled your program with debugging symbols and your program has some leak, the name of your program will appear in some of those lines. Then you run that addr2line command and it'll show you in which file and line the leak is produced. It will show your errors that valgrind won't.
Re: Favorite configuration and system replication tools?
Am So., 12. Mai 2024 um 21:18 Uhr schrieb Walter Alejandro Iglesias : > On Sun May 12 20:58:43 2024 Andreas Kähäri wrote > > With rsync(1): > > > > rsync -n -aim --delete-excluded \ > > --include-from=list \ > > --include='*/' \ > > --exclude='*' \ > > source/ target > > > > I don't understand what your command does exactly. And this is surely man rsync --include=PATTERN This option is a simplified form of the --filter option that specifies an include rule and does not allow the full rule-parsing syntax of normal filter rules. This is equivalent to specifying -f'+ PATTERN'. See the FILTER RULES section for detailed information on this option. --include-from=FILE This option is related to the --include option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file are ignored, as are whole-line comments that start with ';' or '#' (filename rules that contain those characters are unaffected). If a line begins with "- " (dash, space) or "+ " (plus, space), then the type of rule is being explicitly specified as an exclude or an include (respectively). Any rules without such a prefix are taken to be an include. If a line consists of just "!", then the current filter rules are cleared before adding any further rules. If FILE is '-', the list will be read from standard input. Coming back to the topic of this thread: I'm curious that nobody has mentioned ansible/puppet/salt/... yet. Best Martin
Re: Favorite configuration and system replication tools?
On Sun May 12 20:58:43 2024 Andreas Kähäri wrote > With rsync(1): > > rsync -n -aim --delete-excluded \ > --include-from=list \ > --include='*/' \ > --exclude='*' \ > source/ target > I don't understand what your command does exactly. And this is surely of everyone interest since, like me, everyone has many files and directories in $HOME which are not worth to save, as ~/.cache for example. If you take a second look to my examples, the one using pax and the other using rsync inside a loop, both are thought to synchronize *only* what I put in the list and, as you see, in the list there are files and directories. I appreciate that you or any other rsync expert here show me how to accomplish that (in case it's possible) with rsync without resorting to a loop as I did. (Not challenging, I'm asking this as favor.) I use a simliar solution to syncronize my $HOME directory to other machines. Related to the idea of synchronizing a selection of files and directories saved in a list, I attempted to lern how rdist(1) works, as Robert B. Carleton advised me, but I couldn't see much, rdistd(1) core dumped.
What software to debugging and analyzing C?
I found a YouTube channel LowLevelLearning that covers various programming languages in a manner that I find particularly helpful and clear. For example comparing C and assembly on the same code is superb. In a short, he recommended valgrind to help finding memory leaks. Other than splint and gdb, what other software is useful for working with C? I also wouldn't mind any other useful tips that might not be software. Any help very appreciated. -- Regards, Chris Bennett "Who controls the past controls the future. Who controls the present controls the past." George Orwell - 1984
Re: https://twitter.com/openbsd
On Sun, May 12, 2024 at 09:53:00AM +, Rubén Llorente wrote: > > I think it is worth mentioning I know of a number of small operations that > have announced their complete withdrawal from social media - Twitter, > Facebook, Instagram, the Fediverse - because the benefit they get from > social media presence is not worth the labor time required to sustain social > media presence. > > That said, when those operations ceased social media activity, they took > care of making it widely known among their audience rather than just let > their social media accounts rot... > I saw a news bit yesterday that in one town, all of the school children are buying old fashioned typewriters to break their link to computers and do things the old fashioned way. +1 to them. I prefer real text on paper myself. I learn things much better that way. -- Regards, Chris Bennett "Who controls the past controls the future. Who controls the present controls the past." George Orwell - 1984
Re: Favorite configuration and system replication tools?
On Sun, May 12, 2024 at 01:40:25PM +0200, Walter Alejandro Iglesias wrote: > > Unix development. Given that i've been using computers for a few > > decades, i still instinctively don't use spaces in filenames, even > > though they're very much allowed. But of course, that's not what > > most of the world does, and this is an example of trying to work > > out what the best tradeoffs might be when dealing with the > > messiness of the real world. > > I overlooked this in my example because I *never* use spaces, UTF-8 or > any special characters to name my file names. Lately, I finally > persuaded my wife to use Linux, after decades of having to use Windows. > Even when I educated her in this matter she has clients who send her > files named with any kind of crap, so taking care of this issue is still > convenient. > I download a lot of files with a hideous mess of characters. I wrote a small script to substitute in acceptable characters. I can enter a regex, select to just use a directory or go down recursively. Also I can select to only change filenames or directories or both. After reading this thread I see I need to update the script. -- Regards, Chris Bennett "Who controls the past controls the future. Who controls the present controls the past." George Orwell - 1984
Why /var/www/run instead of /var/run for web services
Hi everyone, I hope all of you had a great weekend so far! I was wondering why OpenBSD web services like httpd write their PID file to /var/www/run instead of /var/run. I suspect that it is because a web service might change its root directory to /var/www using chroot(2), making everything outside of this directory inaccessible during runtime (this would probably not affect httpd but maybe other web-related services), and that /var/www is the home directory of the www user. However, I couldn't find anything to confirm this. Can anyone confirm or deny my assumption? Regards, Souji
Re: Favorite configuration and system replication tools?
Andreas Kähäri writes: The external env(1) utility will only ever list environment variables. The IFS variable does not need to be exported as an environment variable as it's only ever used by the current shell (and any new shell would reset it). To list all variables in a shell, use the built-in set utility without any arguments. $ (unset -v IFS; ksh -c 'set' | grep -A 1 IFS) IFS=' ' $ (unset -v IFS; ksh -c 'printf "%s" "$IFS" | hexdump -C') 20 09 0a | ..| 0003 i stand corrected. Alexis.
Re: Favorite configuration and system replication tools?
Страхиња Радић writes: Дана 24/05/12 07:31PM, Alexis написа: Omitting -r as a parameter to read would make it interpret backscape sequences, which would make the directory name in the filesystem different than the one command/script operates on, which is most likely undesired (unless the intention is to exploit some bug). Yes, i understood that omitting `-r` would make it interpret escape sequences, hence me asking: about the possibility of someone having consciously put e.g. a \t in a directory name because they were assuming that it _would_ get interpreted when required? So i take your answer as, in reference a comment in the other subthread: yes, there are in fact 'inappropriate' characters, in the sense that certain representations of certain characters aren't allowed. Which seems very reasonable to me. Thanks for explaining. Alexis.
Re: Favorite configuration and system replication tools?
Дана 24/05/12 07:31PM, Alexis написа:
> i wondered about that in this context. If people putting odd / inappropriate
> things in directory names are a concern ("weird characters", as you wrote
> upthread), what do we do about the possibility of someone having consciously
> put e.g. a \t in a directory name because they were assuming that it _would_
> get interpreted when required?
Omitting -r as a parameter to read would make it interpret backscape
sequences, which would make the directory name in the filesystem
different than the one command/script operates on, which is most
likely undesired (unless the intention is to exploit some bug).
Consider
$ dir=$'helloe[1mworlde[0m'; echo $dir | while read dir; do
echo $dir; mkdir $dir; done
helloe[1mworlde[0m
$ ls -ldq hello*
drwxr-xr-x 2 user user 512 May 12 14:13 helloe[1mworlde[0m/
$ ls -ld $(echo $dir)
ls: hello\e[1mworld\e[0m: No such file or directory
$ rmdir $(echo $dir)
rmdir: hello\e[1mworld\e[0m: No such file or directory
$ rmdir helloe\[1mworlde\[0m/
-- expansion by Tab key
vs
$ dir=$'helloe[1mworlde[0m'; echo $dir | while read -r dir; do
echo $dir; mkdir $dir; done
helloworld
^-- bold attribute on
$ ls -ldq hello*
drwxr-xr-x 2 user user 512 May 12 14:13 hello\e[1mworld\e[0m/
$ ls -ld $(echo $dir)
drwxr-xr-x 2 user user 512 May 12 14:13 hello\e[1mworld\e[0m/
$ rmdir $(echo $dir)
Re: Favorite configuration and system replication tools?
On Sun, May 12, 2024 at 08:08:17PM +1000, Alexis wrote: > Andreas Kähäri writes: > > > Well, that's one way to control this trainwreck of a script; just say > > that any name containing "inappropriate" characters aren't allowed! > > > > May I ask why you don't simply use rsync(1) (or even openrsync(1) from > > the OpenBSD base system)? > > i'm not sure why you're addressing this to me, as i'm not the OP. It's addressed to the thread in general. > > That said, yes, minimising the extent to which certain non-'word' characters > (i.e. roughly the POSIX 'alnum' class as described in re_format(7)) _can_ > make it easier to programatically do certain tasks which are restricted by > the long and messy history of C and Unix development. Given that i've been > using computers for a few decades, i still instinctively don't use spaces in > filenames, even though they're very much allowed. But of course, that's not > what most of the world does, and this is an example of trying to work out > what the best tradeoffs might be when dealing with the messiness of the real > world. > > > Alexis. With rsync(1): rsync -n -aim --delete-excluded \ --include-from=list \ --include='*/' \ --exclude='*' \ source/ target This would read your inclusion patterns from the file "list" (it is assumed that directories are entered as "dirname/***", which matches the name "directory" and all its content), include any directory, and then finaly exclude anything not already included. The matched names would be synchonised from beneath "source" to "target", and excluded names would be deleted from the target. With "-m", we don't keep directories at the target that ends up being empty. -- Andreas (Kusalananda) Kähäri SciLifeLab, NBIS, ICM Uppsala University, Sweden .
Re: Favorite configuration and system replication tools?
On Sun, May 12, 2024 at 07:56:55PM +1000, Alexis wrote: > Andreas Kähäri writes: > > > The ksh(1) shell sets IFS by default to a space, tab and a newline > > character. > > Those are the defaults used when IFS is not set _as a variable_. If you log > in, and run env(1), in the absence of any manual setting of IFS in .kshrc or > whatever, you'll see that IFS is not listed, because it's not 'set' in the > shell variable sense. When it's not set, the shell assumes that IFS has the > value you listed. > > (Additionally, a shell variable not being set is _not_ the same as that > variable being set to the empty string.) > > > Alexis. The external env(1) utility will only ever list environment variables. The IFS variable does not need to be exported as an environment variable as it's only ever used by the current shell (and any new shell would reset it). To list all variables in a shell, use the built-in set utility without any arguments. $ (unset -v IFS; ksh -c 'set' | grep -A 1 IFS) IFS=' ' $ (unset -v IFS; ksh -c 'printf "%s" "$IFS" | hexdump -C') 20 09 0a | ..| 0003 -- Andreas (Kusalananda) Kähäri SciLifeLab, NBIS, ICM Uppsala University, Sweden .
Re: Favorite configuration and system replication tools?
On Sun May 12 13:22:13 2024 Alexis wrote: > Andreas Kähäri writes: > > Well, that's one way to control this trainwreck of a script; > > just say > > that any name containing "inappropriate" characters aren't > > allowed! > > > > May I ask why you don't simply use rsync(1) (or even > > openrsync(1) from > > the OpenBSD base system)? > > i'm not sure why you're addressing this to me, as i'm not the OP. I guess it's me who Andreas should address this question to, right? I gave a dirty example to someone who mentioned pax to the OP. Just playing and lerning. :-) I've been using rsync since ever, but, first, I don't think rsync is bulletproof either and, second, making a backup with pax is faster and in some cases simpler. If you don't want to delete files on the target you don't need to do scripting at all. Let's take the example I put in my first message. With rsync, you'll have to do something like this: ~/backup_list # backup_list .Xdefaults .kshrc .nexrc .profile .calendar/ .config/feh/ .config/fontconfig/ .config/gtk-3.0/gtk.css Documents/ Pictures/ [...] --- files=$(egrep -v "^$|^#" ~/backup_list) for i in $files ; do rsync -av --delete --mkpath $HOME/$i $device/$user/$i done But openrsync doesn't have a '--mkpath' option, I let Andreas to think the solution. :-) > > That said, yes, minimising the extent to which certain non-'word' > characters (i.e. roughly the POSIX 'alnum' class as described in > re_format(7)) _can_ make it easier to programatically do certain > tasks which are restricted by the long and messy history of C and > Unix development. Given that i've been using computers for a few > decades, i still instinctively don't use spaces in filenames, even > though they're very much allowed. But of course, that's not what > most of the world does, and this is an example of trying to work > out what the best tradeoffs might be when dealing with the > messiness of the real world. I overlooked this in my example because I *never* use spaces, UTF-8 or any special characters to name my file names. Lately, I finally persuaded my wife to use Linux, after decades of having to use Windows. Even when I educated her in this matter she has clients who send her files named with any kind of crap, so taking care of this issue is still convenient. > > > Alexis. > > > -- Walter
Re: Favorite configuration and system replication tools?
On Sun, May 12, 2024 at 07:31:41PM +1000, Alexis wrote:
> Страхиња Радић writes:
>
> > When `while ... read ...` idiom is used, it is advisable to clear IFS to
> > turn off field splitting
>
> *nod* Fair point; it's not set by default, so i didn't think to note that
> any manual setting of it should be overridden for this.
The ksh(1) shell sets IFS by default to a space, tab and a newline
character.
>
> > and use -r to avoid interpretation of backslash sequences in the input:
>
> i wondered about that in this context. If people putting odd / inappropriate
> things in directory names are a concern ("weird characters", as you wrote
> upthread), what do we do about the possibility of someone having consciously
> put e.g. a \t in a directory name because they were assuming that it _would_
> get interpreted when required?
>
>
> Alexis.
Well, that's one way to control this trainwreck of a script; just say
that any name containing "inappropriate" characters aren't allowed!
May I ask why you don't simply use rsync(1) (or even openrsync(1) from
the OpenBSD base system)?
--
Andreas (Kusalananda) Kähäri
Uppsala, Sweden
.
Re: Favorite configuration and system replication tools?
On Sun May 12 11:40:05 2024 tux2bsd wrote > Hi Walter > > mktemp makes temporary unique filenames like this: > > delete_list=$(mktemp) > source_list=$(mktemp) > target_list=$(mktemp) > # Do your code. If you want to keep something you do > # that appropriately then: > rm $delete_list $source_list $target_list > > This version can deal with files with special characters and spaces: # Remove files from target directory delete_list=$(mktemp -t delete.XX) || exit 1 source_list=$(mktemp -t source.XX) || exit 1 target_list=$(mktemp -t target.XX) || exit 1 dirs=$(echo "$files" | grep '/$') cd && find $dirs | sort | uniq > $source_list cd "$target" && find $dirs | sort | uniq > $target_list diff $source_list $target_list | grep '^> ' | sed 's#^> #'$target'/#' > $delete_list cd && while read line; do echo "delete $line" rm "$line" done < $delete_list # Clean rm $source_list $target_list $delete_list
Re: Favorite configuration and system replication tools?
Andreas Kähäri writes: Well, that's one way to control this trainwreck of a script; just say that any name containing "inappropriate" characters aren't allowed! May I ask why you don't simply use rsync(1) (or even openrsync(1) from the OpenBSD base system)? i'm not sure why you're addressing this to me, as i'm not the OP. That said, yes, minimising the extent to which certain non-'word' characters (i.e. roughly the POSIX 'alnum' class as described in re_format(7)) _can_ make it easier to programatically do certain tasks which are restricted by the long and messy history of C and Unix development. Given that i've been using computers for a few decades, i still instinctively don't use spaces in filenames, even though they're very much allowed. But of course, that's not what most of the world does, and this is an example of trying to work out what the best tradeoffs might be when dealing with the messiness of the real world. Alexis.
Re: Favorite configuration and system replication tools?
Andreas Kähäri writes: The ksh(1) shell sets IFS by default to a space, tab and a newline character. Those are the defaults used when IFS is not set _as a variable_. If you log in, and run env(1), in the absence of any manual setting of IFS in .kshrc or whatever, you'll see that IFS is not listed, because it's not 'set' in the shell variable sense. When it's not set, the shell assumes that IFS has the value you listed. (Additionally, a shell variable not being set is _not_ the same as that variable being set to the empty string.) Alexis.
Re: https://twitter.com/openbsd
Stuart Longland wrote: It's also dead because how how things are being run there. It's a site for misinformation. "OpenBSD 7.5 is released" isn't misinformation, it's fact, so has no place on twitter.com or x.com. It's also news about an open-source free-software project, something that also is off-topic for twitter.com and x.com. Traditional social media has been on the decline since 2015, and it has nothing to do with the reliability of the content it hosts. I would argue TV stations and newspapers generate a comparable amount of garbage and nobody complains. The main issue is that it is getting harder to reach people from social media, which is the main goal of corporate and commercial users of social media. So called "organic growth" (ie. passively building an audience just because they find you on social media) has been descending since 2015. The main issue seems to be that mainstream social media generates so much stuff for people to read that the probability of people finding you instead of finding yet another silly video has decreased significantly. I think it is worth mentioning I know of a number of small operations that have announced their complete withdrawal from social media - Twitter, Facebook, Instagram, the Fediverse - because the benefit they get from social media presence is not worth the labor time required to sustain social media presence. That said, when those operations ceased social media activity, they took care of making it widely known among their audience rather than just let their social media accounts rot...
Re: Favorite configuration and system replication tools?
Страхиња Радић writes:
When `while ... read ...` idiom is used, it is advisable to
clear IFS
to turn off field splitting
*nod* Fair point; it's not set by default, so i didn't think to
note that any manual setting of it should be overridden for this.
and use -r to avoid interpretation of backslash sequences in the
input:
i wondered about that in this context. If people putting odd /
inappropriate things in directory names are a concern ("weird
characters", as you wrote upthread), what do we do about the
possibility of someone having consciously put e.g. a \t in a
directory name because they were assuming that it _would_ get
interpreted when required?
Alexis.
Re: Favorite configuration and system replication tools?
> What about the following, better? > > - > # Remove files from target directory > date=$(date +%H%M%S) > delete_list=/tmp/delete_$date > source_list=/tmp/source_$date > target_list=/tmp/target_$date Hi Walter mktemp makes temporary unique filenames like this: delete_list=$(mktemp) source_list=$(mktemp) target_list=$(mktemp) # Do your code. If you want to keep something you do # that appropriately then: rm $delete_list $source_list $target_list tux2bsd
Re: Favorite configuration and system replication tools?
Дана 24/05/12 06:17PM, Alexis написа: > To deal with spaces etc., one could possibly use something along the lines > of the following kludge; it assumes that \n is relatively unlikely to be > found in a directory name, and that the directories in $dirs can be > separated by \n. > > cd "$target" && >echo "$(echo $dirs | while read dir >do > find $dir >done)\n" | sort | uniq > "$target_list" When `while ... read ...` idiom is used, it is advisable to clear IFS to turn off field splitting, and use -r to avoid interpretation of backslash sequences in the input: while IFS= read -r dir; do # ... Back to parsing the output of ls(1) (also applicable to parsing the output of find(1), or globs), there is an indepth analysis of the problem at [1]. The accepted answer concludes that perhaps shell command language is not the right tool for the job, and a more sophisticated language should be used instead. While I don't agree with the author's choice of Python, any language supporting opendir(3), readdir(3) or equivalent functions will suffice. [1]: https://unix.stackexchange.com/questions/128985/why-not-parse-ls-and-what-to-do-instead
Re: Favorite configuration and system replication tools?
On Sun May 12 10:07:30 2024 Страхиња Радић wrote:
> A few notes:
>
> - You don't need a backslash after a pipe (|) or a list operator (||
> and &&) - a line ending with a pipe is an incomplete pipeline. So
> (with added quoting):
>
> diff "$source_list" "$target_list" |
> awk '/^> / { print "'"$target"'/" $NF }' > "$delete_list"
I know, just fingers habit. :-)
>
> As an example for a list operator, the second line beginning with cd
> could also be written as:
>
> cd "$target" &&
> find "$dirs" | sort | uniq > "$target_list"
>
> This works even when entering commands interactively from the command
> line.
>
> - Before the `rm -rf` line, a useless use of cat[1]:
>
> sed 's/^/delete /' "$delete_list"
>
> - The xargs is unnecessary in `rm -rf $(cat $delete_list | xargs)`;
> BTW, that line is vulnerable to weird pathnames (for example,
> those including spaces, line feeds and special characters).
>
What about the following, better?
-
# Remove files from target directory
date=$(date +%H%M%S)
delete_list=/tmp/delete_$date
source_list=/tmp/source_$date
target_list=/tmp/target_$date
dirs=$(echo "$files" | grep '/$')
cd && find $dirs | sort | uniq > $source_list
cd "$target" && find $dirs | sort | uniq > $target_list
diff $source_list $target_list |
awk '/^> / { print "'$target'/" $NF }' > $delete_list
cd &&
if [ -s $delete_list ]; then
echo "Deleting on ${target}:"
rm -vrf $(cat $delete_list)
fi
# Clean
rm $source_list $target_list $delete_list
-
Thanks for your recomendations!
--
Walter
Re: Favorite configuration and system replication tools?
Страхиња Радић writes: Lapsus: the variable dirs should not be quoted here if it contains more than one directory to be passed to find. It is vulnerable to directory names containing spaces and weird characters, however. So: cd "$target" && find $dirs | sort | uniq > "$target_list" To deal with spaces etc., one could possibly use something along the lines of the following kludge; it assumes that \n is relatively unlikely to be found in a directory name, and that the directories in $dirs can be separated by \n. cd "$target" && echo "$(echo $dirs | while read dir do find $dir done)\n" | sort | uniq > "$target_list" Alexis.
