Re: [capnproto] Server process stops accepting connections

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Sorry for the slow reply, I've been on leave.

In general, KJ async I/O objects are tied to the thread / event loop where
they were created. This means you cannot pass an AsyncIoStream between
threads. You can, however, pass a file descriptor. If you are careful, you
could tear down the AsyncIoStream wrapping your socket, pass the socket
file descriptor to another thread, and construct a new AsyncIoStream around
it there.

-Kenton

On Wed, Nov 9, 2022 at 12:19 PM Jens Alfke  wrote:

>
> On Nov 9, 2022, at 6:47 AM, 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
> Have you tested whether your server is able to accept concurrent
> connections normally? E.g. if you open a connection with telnet or
> something without sending any bytes, leave that open, and then try to use
> your server, does it work?
>
>
> *Thanks! That was indeed the problem.* The telnet test showed my server
> would accept multiple connections at once, but would only authenticate one
> of them at a time, so a client that didn’t send the right handshake would
> block others.
>
> (I’m using custom code, largely copied from rpc.c++ and tls.c++, but
> substituting the SecretHandshake protocol (from Scuttlebutt) for TLS. When
> I adapted TlsConnectionReceiver::acceptLoop, I got things in the wrong
> order, so the tail-recursion back to acceptLoop didn’t occur until the
> secure handshake finished.)
>
> I’ve fixed it now. Next question: *if I want to run client connections on
> multiple threads*, I presume that after the accept-and-handshake
> finishes, I would create a new thread with an event loop, then use an
> Executor to pass the AsyncIoStream to it, and then do the magic RPC setup
> on that thread. On disconnect I’d either exit the thread or return it to a
> pool for reuse. Most of this seems like boilerplate … are there any working
> examples I could steal from?
>
> (Why would I want multiple threads when CapnP is async? Because my actual
> RPC methods do SQLite queries, which are blocking and sometimes slow, and
> I’d rather not create async wrappers for my entire complicated database
> API.)
>
> —Jens
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DZfFUQYcFzKfUtJ57KuXH5Jpbd2HqX4Hbh8wLGdW8-Hw%40mail.gmail.com.

Re: [capnproto] Server process stops accepting connections

[Jens Alfke]

> On Nov 9, 2022, at 6:47 AM, 'Kenton Varda' via Cap'n Proto 
>  wrote:
> 
> Have you tested whether your server is able to accept concurrent connections 
> normally? E.g. if you open a connection with telnet or something without 
> sending any bytes, leave that open, and then try to use your server, does it 
> work?

Thanks! That was indeed the problem. The telnet test showed my server would 
accept multiple connections at once, but would only authenticate one of them at 
a time, so a client that didn’t send the right handshake would block others. 

(I’m using custom code, largely copied from rpc.c++ and tls.c++, but 
substituting the SecretHandshake protocol (from Scuttlebutt) for TLS. When I 
adapted TlsConnectionReceiver::acceptLoop, I got things in the wrong order, so 
the tail-recursion back to acceptLoop didn’t occur until the secure handshake 
finished.)

I’ve fixed it now. Next question: if I want to run client connections on 
multiple threads, I presume that after the accept-and-handshake finishes, I 
would create a new thread with an event loop, then use an Executor to pass the 
AsyncIoStream to it, and then do the magic RPC setup on that thread. On 
disconnect I’d either exit the thread or return it to a pool for reuse. Most of 
this seems like boilerplate … are there any working examples I could steal from?

(Why would I want multiple threads when CapnP is async? Because my actual RPC 
methods do SQLite queries, which are blocking and sometimes slow, and I’d 
rather not create async wrappers for my entire complicated database API.)

—Jens

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/8A947B79-0997-4DEB-9A9A-3EF7B0ACB024%40mooseyard.com.

Re: [capnproto] Server process stops accepting connections

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Have you tested whether your server is able to accept concurrent
connections normally? E.g. if you open a connection with telnet or
something without sending any bytes, leave that open, and then try to use
your server, does it work?

What does your listen loop look like?

-Kenton

On Tue, Nov 8, 2022 at 1:33 PM Jens Alfke  wrote:

> I’ve got a CapnP-based server program running on a Raspberry Pi 4. It’s
> written in C++, using CapnP 0.9.1. It’s single-threaded; the main()
> function just sets up a listener, calls kj::NEVER_DONE.wait, and stays
> there.
>
> It works fine, except that every few weeks it stops accepting connections;
> it either doesn’t accept the incoming socket at all or never reads from it,
> I can’t tell which; all I see is that the client times out and gives up
> after 15 seconds. Then I have to kill and relaunch the server, after which
> it works fine again for a while.
>
> The server process itself isn’t hung; it’s in epoll waiting for events. I
> attached gdb and got this backtrace:
>
> #0  0xb6b8563c in epoll_wait (epfd=7, events=0xbecb4210, maxevents=16,
>  timeout=-1)
> #1  0x0052edcc in kj::UnixEventPort::doEpollWait (this=0x1fed420,
> timeout=-1)
> #2  0x0052e58c in kj::UnixEventPort::wait (this=0x1fed420)
> #3  0x00487744 in kj::EventLoop::wait (this=0x1fed4e0)
> #4  0x00488460 in kj::_::waitImpl (node=..., result=..., waitScope=…)
> #5  0x00488bd8 in kj::_::NeverDone::wait (this=0x72d3c0 ,
>  waitScope=…)
> …
>
>
> The last line in the log shows that it received a connection from some
> unknown IPv4 address which geolocates to Russia (this happens a few times a
> day; the server’s on my home LAN but exposes a public port and I assume
> these are random hackers looking for vulns.) This connection never did
> anything, not surprisingly since I doubt hackers are expecting anything
> other than HTTP, but the logs show the socket never closed. And sure
> enough, if I run `lsof` I see an open TCP port from that address. (Which
> has been open for at least 24 hours, strangely; doesn’t the disconnect idle
> TCP connections after 90 minutes?)
>
> I’m not sure what to do about this. I assume CapnP is capable of handling
> multiple incoming connections, so this idle socket won’t block others from
> connecting, right? But if not, then why isn’t it accepting any connections?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/008DEC56-3B4D-4CF7-BFC3-5BECC13BE502%40mooseyard.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D6xH_pyvRGBnRpTma6_K2mU8Ogn%2BptcQ9RytARhAUrAw%40mail.gmail.com.

[capnproto] Server process stops accepting connections

[Jens Alfke]

I’ve got a CapnP-based server program running on a Raspberry Pi 4. It’s written 
in C++, using CapnP 0.9.1. It’s single-threaded; the main() function just sets 
up a listener, calls kj::NEVER_DONE.wait, and stays there.

It works fine, except that every few weeks it stops accepting connections; it 
either doesn’t accept the incoming socket at all or never reads from it, I 
can’t tell which; all I see is that the client times out and gives up after 15 
seconds. Then I have to kill and relaunch the server, after which it works fine 
again for a while.

The server process itself isn’t hung; it’s in epoll waiting for events. I 
attached gdb and got this backtrace:
#0  0xb6b8563c in epoll_wait (epfd=7, events=0xbecb4210, maxevents=16,  
timeout=-1)
#1  0x0052edcc in kj::UnixEventPort::doEpollWait (this=0x1fed420, timeout=-1)
#2  0x0052e58c in kj::UnixEventPort::wait (this=0x1fed420)
#3  0x00487744 in kj::EventLoop::wait (this=0x1fed4e0)
#4  0x00488460 in kj::_::waitImpl (node=..., result=..., waitScope=…)
#5  0x00488bd8 in kj::_::NeverDone::wait (this=0x72d3c0 ,  
waitScope=…)
…

The last line in the log shows that it received a connection from some unknown 
IPv4 address which geolocates to Russia (this happens a few times a day; the 
server’s on my home LAN but exposes a public port and I assume these are random 
hackers looking for vulns.) This connection never did anything, not 
surprisingly since I doubt hackers are expecting anything other than HTTP, but 
the logs show the socket never closed. And sure enough, if I run `lsof` I see 
an open TCP port from that address. (Which has been open for at least 24 hours, 
strangely; doesn’t the disconnect idle TCP connections after 90 minutes?)

I’m not sure what to do about this. I assume CapnP is capable of handling 
multiple incoming connections, so this idle socket won’t block others from 
connecting, right? But if not, then why isn’t it accepting any connections?

—Jens

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/008DEC56-3B4D-4CF7-BFC3-5BECC13BE502%40mooseyard.com.