Re: EACCES of UDP packet
On Tue, Jun 22, 2021 at 04:48:26PM +0800, Siegfried Levin wrote: > > Why have you chosen to hide information that may be useful in debugging > > your problem? > > I’m truly sorry for the inconvenience but I do have some concerns of security > and privacy. I confirm it is not a broadcast address because it is the public > IP of the server and this issue has a probability of 1% to happen. The > address cannot just be a broadcast address at 1% of the time while not at the > rest of 99%. I also double checked it by SSHing to the address I copied from > the kdump, if it makes sense. > > > So, since the manpage mentions blocking pf, I suggest the hypothesis "it > > returns EACCES because pf is blocking your packets". I can think of > > several ways to test that; what testing have you performed to confirm or > > rule out that possibility? "doas pfctl -d; run test; doas pfctl -e”? > > This issue is really hard to reproduce because the application works at most > of the time, but I think you are right. I’ll be watching the pf log in next > weeks. > Also check the various counters of netstat -s and especially pfctl -si (or systat pf). In the pfctl output especially check memory, congestion or state errors. -- :wq Claudio
Re: EACCES of UDP packet
> Why have you chosen to hide information that may be useful in debugging your
> problem?
I’m truly sorry for the inconvenience but I do have some concerns of security
and privacy. I confirm it is not a broadcast address because it is the public
IP of the server and this issue has a probability of 1% to happen. The address
cannot just be a broadcast address at 1% of the time while not at the rest of
99%. I also double checked it by SSHing to the address I copied from the kdump,
if it makes sense.
> So, since the manpage mentions blocking pf, I suggest the hypothesis "it
> returns EACCES because pf is blocking your packets". I can think of several
> ways to test that; what testing have you performed to confirm or rule out
> that possibility? "doas pfctl -d; run test; doas pfctl -e”?
This issue is really hard to reproduce because the application works at most of
the time, but I think you are right. I’ll be watching the pf log in next weeks.
> Alternatively: what's different about *that* call? Does every sento() call
> on that socket fail? What is special about that socket? If other sendto()
> calls succeed, what is different about that call? Earlier setsockopt() calls?
The call failed once after a few days of running. I have no idea of how to
present this but it seems the second and third arguments of sendto call are
always changing?
3058 myapp CALL
sendto(5,0x1689f5f6100,0x5d,0x400,0x7f7f1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:y }
3058 myapp GIO fd 5 wrote 93 bytes
"\M^Ai9\M^N\M^U\M-@9
\M-7\M-B\M-m\M^C.\M-ut\M^E~)\M^H'w\M-A\M-p|\M-+DK\M-V\M-8v\M-~c\240\M-=y\M-E\M-*\M-+.\M-I\M-m\M^[\M-s\M-&\\\M^GwP\M-I\r\M^[\M-C\^]B\M-%\M-<\M-q\
\M-rk\M-_\M-g\M-9Y\^B\M-+Y\M-:K\M^X\^E/*\M-4_\M-x\at\M-{\M-$\M-[\^NU\\\M^HPk\M-i\\\M-M\M-H8t
"
3058 myapp RET sendto 93/0x5d
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ,
flags=0x61, fflags=0<>, data=73, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7f0af0,0x7d0)
3058 myapp GIO fd 4 read 73 bytes
"\0\0\0\^BE\0\0E\M^_?\0\0@\^Q\M-9\M-:\M-@\M-(_\^D\^A\^A\^A\^A\M-x\M-U\0005\0001\M^U\M-Q\M-B\M-U\^A\0\0\^A\0\0\0\0\0\^A\^Bbt\^Ehenbt\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\
\0\M^@\0\0\0"
3058 myapp RET read 73/0x49
3058 myapp CALL read(4,0x7f7f0af0,0x7d0)
3058 myapp GIO fd 4 read 79 bytes
"\0\0\0\^BE\0\0K\M-i\M^E\0\0@\^Qpo\M-@\M-(_\^D\^A\0\0\^A\M-Sy\0005\0007\M-8
\M-dB\^A\0\0\^A\0\0\0\0\0\^A\atracker\^Fcity9x\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\
\0"
3058 myapp RET read 79/0x4f
3058 myapp CALL read(4,0x7f7f0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7f07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u"
3058 myapp RET read 12/0xc
3058 myapp CALL
sendto(5,0x1690776aa00,0x61,0x400,0x7f7f1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:y }
3058 myapp GIO fd 5 wrote 97 bytes
"\M-)\M-,\M-r\M-b\^R\M-Jb\M-@\M-b\M-{\M^G"\M-Z\v\M-v\M-L\^^\M-BI\M-X\M-S\M-L]\M-&\^R%\M-?\M-\\M-kN\M^G:\M-NVI\M-@s\0#\M-V\M-}\M-h\M-T\M^C\^R\M-W\M-"\^?\M^Q[:u\M-=\
\M-Q%\M-c\M-+\M-xZ\^B\M-$\M-N\M-1\^Zy\M^D 9#\^Q\M--\M^Jb\M^T(
\M^@\M^L;~\M-<\M-}0\M-G\M-`4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u"
3058 myapp RET sendto 97/0x61
3058 myapp CALL read(7,0x7f7f07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D"
3058 myapp RET read 12/0xc
3058 myapp CALL
sendto(5,0x1690776a600,0x67,0x400,0x7f7f1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:y }
3058 myapp GIO fd 5 wrote 103 bytes
"\^W\^\\M-B\M^E\M-~\M-c\M-3\M-f\M-X\M^Y\M-Z\^V\M-G\M-p\M^L\M^G\M-b\M-T\M^I\M-SWu\M-]\M^Jq\M^Q\M-Z\^Z\M^[\bW>`/fW\b\M^K\M-*\^]:E\M-T\M-[\M^X\M-5M\M^J\M-j\M-v\^^Z@:\
\M-QfQ\a-\M-\)\M-O$[\M-x]\b\M-Wh5\M^F\M^B|I>
\M^CY\M-%\M-6'\M^?g9F?y\^T\M-}\^E\M-g\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D"
3058 myapp RET sendto 103/0x67
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ,
flags=0x61, fflags=0<>, data=75, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7f0af0,0x7d0)
3058 myapp GIO fd 4 read 75 bytes
"\0\0\0\^BE\0\0GS\M^L\0\0@\^Q\^Fm\M-@\M-(_\^D\^A\0\0\^A\M-n\M-)\0005\0003m\M-_N\M-#\^A\0\0\^A\0\0\0\0\0\^A\^Cbtx\aanifilm\^Btv\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\0"
3058 myapp RET read 75/0x4b
3058 myapp CALL read(4,0x7f7f0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7f07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M^T\M-|\M-e\M-Fk-\M^P\M-u}\M-j\M^U="
3058 myapp RET read 12/0xc
3058 myapp CALL
Re: EACCES of UDP packet
On Mon, Jun 21, 2021 at 9:07 PM Siegfried Levin
wrote:
> Thanks a lot for the hint. Unfortunately I’m still not able to see why
> sendto failed with 13 Permission denied. The AF_INET address masked is the
> correct one of my server, not a broadcast address. A sendto before this one
> to the same address just worked.
>
> 3058 myapp CALL
> sendto(5,0x1689f5f6500,0x5d,0x400,0x7f7f1144,0x10)
> 3058 myapp STRU struct sockaddr { AF_INET, xxx.xxx.xxx.xxx: }
>
Why have you chosen to hide information that may be useful in debugging
your problem?
"Hi, I'm asking for help but I have to hide addresses because...this
application is insecure if anyone else has its IP+port? Because I've never
heard of shodan and don't believe that people are constantly scanning the
Internet? And while I don't know why it's failing I'm 1000% sure that
there's no information to be gained from seeing the IP, so if it later
turns out my understanding of 'broadcast address' is incorrect, the time
I've wasted for myself and others will be...a total loss?"
3058 myapp RET sendto -1 errno 13 Permission denied
> 3058 myapp CALL close(5)
> 3058 myapp RET close 0
>
The dump file is like 600MB. I can provide more trace log if it is
> necessary for locating the root cause.
>
Use the scientific method:
* make a testable hypothesis
* devise a test for that
* perform the test
* determine whether the hypothesis has been ruled out or confirmed
So, since the manpage mentions blocking pf, I suggest the hypothesis "it
returns EACCES because pf is blocking your packets". I can think of
several ways to test that; what testing have you performed to confirm or
rule out that possibility? "doas pfctl -d; run test; doas pfctl -e"?
Alternatively: what's different about *that* call? Does every sento() call
on that socket fail? What is special about that socket? If other sendto()
calls succeed, what is different about that call? Earlier setsockopt()
calls?
You say "I can confirm the packet was not sent to a broadcast address":
*how* have you confirmed that your understanding of 'broadcast address'
matches the kernel's understanding? It ain't just 255.255.255.255
Philip Guenther
>
>
> Siegfried
> siegfried.le...@gmail.com
>
>
>
>
> > On Jun 15, 2021, at 8:50 PM, Theo de Raadt wrote:
> >
> > use ktrace
> >
> > Siegfried Levin wrote:
> >
> >> Hi,
> >>
> >> I have a application run by a normal user communicating with the server
> with UDP. It crashes very occasionally, like once per week, due to EACCES
> when sending a UDP packet. According to the manpage (
> https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES), the reason might
> be either being blocked by PF or sending to a broadcast address. I can
> confirm the packet was not sent to a broadcast address. However, I cannot
> figure out what rule could block the connection occasionally either. The
> application can be brought back online without changing any configuration.
> Does anyone know what might fix this? I can also rewrite the code to make
> it ignore the error and keep trying but that might not be a proper
> solution. Running it as root might not be a good idea, too.
> >>
> >> It happens since OpenBSD 6.8. Now I’m running it on 6.9. The
> application is written in Rust.
> >>
> >> Siegfried
> >> siegfried.le...@gmail.com
> >>
> >>
> >>
> >>
>
>
Re: EACCES of UDP packet
Thanks a lot for the hint. Unfortunately I’m still not able to see why sendto
failed with 13 Permission denied. The AF_INET address masked is the correct one
of my server, not a broadcast address. A sendto before this one to the same
address just worked.
3058 myapp CALL
sendto(5,0x1689f5f6500,0x5d,0x400,0x7f7f1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xxx.xxx.xxx: }
3058 myapp RET sendto -1 errno 13 Permission denied
3058 myapp CALL close(5)
3058 myapp RET close 0
The dump file is like 600MB. I can provide more trace log if it is necessary
for locating the root cause.
Siegfried
siegfried.le...@gmail.com
> On Jun 15, 2021, at 8:50 PM, Theo de Raadt wrote:
>
> use ktrace
>
> Siegfried Levin wrote:
>
>> Hi,
>>
>> I have a application run by a normal user communicating with the server with
>> UDP. It crashes very occasionally, like once per week, due to EACCES when
>> sending a UDP packet. According to the manpage
>> (https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES), the reason might be
>> either being blocked by PF or sending to a broadcast address. I can confirm
>> the packet was not sent to a broadcast address. However, I cannot figure out
>> what rule could block the connection occasionally either. The application
>> can be brought back online without changing any configuration. Does anyone
>> know what might fix this? I can also rewrite the code to make it ignore the
>> error and keep trying but that might not be a proper solution. Running it as
>> root might not be a good idea, too.
>>
>> It happens since OpenBSD 6.8. Now I’m running it on 6.9. The application is
>> written in Rust.
>>
>> Siegfried
>> siegfried.le...@gmail.com
>>
>>
>>
>>
Re: EACCES of UDP packet
use ktrace Siegfried Levin wrote: > Hi, > > I have a application run by a normal user communicating with the server with > UDP. It crashes very occasionally, like once per week, due to EACCES when > sending a UDP packet. According to the manpage > (https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES), the reason might be > either being blocked by PF or sending to a broadcast address. I can confirm > the packet was not sent to a broadcast address. However, I cannot figure out > what rule could block the connection occasionally either. The application can > be brought back online without changing any configuration. Does anyone know > what might fix this? I can also rewrite the code to make it ignore the error > and keep trying but that might not be a proper solution. Running it as root > might not be a good idea, too. > > It happens since OpenBSD 6.8. Now I’m running it on 6.9. The application is > written in Rust. > > Siegfried > siegfried.le...@gmail.com > > > >
EACCES of UDP packet
Hi, I have a application run by a normal user communicating with the server with UDP. It crashes very occasionally, like once per week, due to EACCES when sending a UDP packet. According to the manpage (https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES), the reason might be either being blocked by PF or sending to a broadcast address. I can confirm the packet was not sent to a broadcast address. However, I cannot figure out what rule could block the connection occasionally either. The application can be brought back online without changing any configuration. Does anyone know what might fix this? I can also rewrite the code to make it ignore the error and keep trying but that might not be a proper solution. Running it as root might not be a good idea, too. It happens since OpenBSD 6.8. Now I’m running it on 6.9. The application is written in Rust. Siegfried siegfried.le...@gmail.com
