Hi:
I think it is safe to perform connection acceptance and epoll() oriented
TCP socket
transactions in separate thread/tasks if you adhere to the following rules.
My server follows these rules and works seamlessly with the following
exception. If it uses
Linux close() TCP socket descriptors, after uv_poll_stop() and uv_close()
have succeeded for a
particular uv_poll_t connection handle, internal Libuv structures are
corrupted
intermittently after a long period of successfully making and releasing
many TCP connections.
This implies that Libuv automatically closes TCP socket descriptors because
my Server
never runs out of them no matter how long it runs.
RULES
-----
* There is separate uv_loop_t Loop structure for accepting incoming
connections and epoll() I/O events.
In my case they are declared as follows:
uv_loop_t Connect_Loop; // Libuv loop for incoming connections.
static uv_loop_t Poll_Loop; // Libuv loop for reading incoming
network data.
* The Loop structures are owned by separate tasks and only accessed by the
task that owns the Loop structure.
In my case the main() task/thread owns Connect_Loop and the IO_Task()
thread/task owns Poll_Loop.
* Only the task/thread which owns the incoming connection acceptance Loop
calls connection management
related Libuv API routines, such uv_accept() and uv_close(), and access
uv_tcp_t connection
handles allocated during connection acceptance.
In my case the main() task/thread follows this rule and owns Connect_Loop.
* Only the task/thread which owns the epoll() I/O event Loop calls polling
related Libuv API routines,
such as uv_poll_start() and uv_poll_stop(), and access uv_poll_t
connection handles allocated
during polling initiation.
In my case the the IO_Task() thread/task follows this rule and owns the
Poll_Loop.
Best Regards,
Paul R.
On Tuesday, May 11, 2021 at 6:45:59 AM UTC-7 pa...@rcom-software.com wrote:
> Hi:
>
> Perhaps it would be useful to know that all Libuv epoll() operations occur
> in the IO_Task() which
> is dedicated to reception of networrk data. This provides nice modularity
> and minimizes the
> possibiliy of deadlocks due to interactions with other kinds of
> functionality.
>
> Best Regards,
>
> Paul R.
>
> On Sunday, May 9, 2021 at 12:14:28 PM UTC-7 pa...@rcom-software.com wrote:
>
>> Hi:
>>
>> I get the FD immediately after accepting an incomig connection with
>> uv_fileno().
>> Then subsequently it is used as follows:
>>
>> * IO_Task(): When the poll callback executes correctly, it is used with
>> Linux recv() to
>> read incoming network data in non-blocking mode. (All incoming data and
>> polling
>> is done in this thread/task.)
>>
>> * Transmit_Task(): It is used with Linux send() to send data across the
>> network in non-blocking
>> mode. This thread/ task is driven by message reception outside Libuv
>> mechanisms.
>>
>> That's it other than the final close() we discussed. I never observed any
>> problems unrelated
>> to Libuv connection acceptance and release, and starting and stoping
>> Libuv polling.
>>
>> Best Regards,
>>
>> Paul R.
>>
>>
>> On 05/09/2021 11:58 AM, Jameson Nash wrote:
>>
>> Are you extracting a fd from a libuv object then calling close or uv_poll
>> on it? Either would cause problems, and shouldn't be done.
>>
>> On Sun, May 9, 2021 at 2:53 PM Jameson Nash <vtj...@gmail.com> wrote:
>>
>>> uv_poll_stop and uv_accept must happen on the same thread, so how are
>>> they racing? Once uv_poll_stop is called, it should be safe to notify
>>> another thread to call close, even if that then happens concurrently, as
>>> the kernel should be thread-safe (though weird things may happen in
>>> userspace if you close a fd that is concurrently in use on another thread
>>> or event queue).
>>>
>>> On Sun, May 9, 2021 at 1:24 PM Paul Romero <pa...@rcom-software.com>
>>> wrote:
>>>
>>>> Hi:
>>>>
>>>> Using TSAN did turn up one very signifcant problem. The root cause of
>>>> the TCP socket descriptor corrpution
>>>> is that accept4() executes concurrently in the main() task, with
>>>> close() in the Protocol_Task().
>>>>
>>>> As an interim measure I avoid the problem by simply not calling
>>>> close(). So far this has worked
>>>> seemlesly and Libuv appears to automatically free socket descriptors.
>>>> The Libuv documentation
>>>> about this is somewhat ambiguous. It indicates that after calling
>>>> uv_poll_stop() or uv_close(),
>>>> for a particular uv_poll_t poll handle, the socket descriptor is
>>>> returned to the user per the Libuv
>>>> contract. I am not sure what that means, and in particular, if it means
>>>> the socket descriptor is freed.
>>>> Can you clarify this ?
>>>>
>>>> A tagential issue is whether Linux accept4() and close() are thread
>>>> safe. I believe they
>>>> are and the crucial data is protected in the kernel. Is it possible
>>>> Libuv is not handling the accept4()
>>>> return codes correctly ? The Linux accept4() man page details how
>>>> errors should be handled and is it
>>>> somewhat fussy. The Linux close() man page also details error handling
>>>> but it is straight forward.
>>>>
>>>> Also, I haven't been able to make the program compiled with TSAN dump
>>>> core. Do you have
>>>> any suggestions ? Incidentally, I had to use clang rather than the
>>>> usual gcc to get
>>>> TSAN to work on my system.
>>>>
>>>> Best Regards,
>>>>
>>>> Paul R.
>>>>
>>>> On 05/08/2021 08:51 AM, Jameson Nash wrote:
>>>>
>>>> Are you still accessing libuv (sans explicitly thread-safe functions
>>>> such as uv_async_send) from multiple threads, as you mentioned earlier? If
>>>> so, I'd suggest fixing that first. In conjunction, I recommend running
>>>> TSan
>>>> and making sure it runs cleanly before checking for library or logic
>>>> problems. Then, if it is still a rare failure, I recommend debugging under
>>>> `rr` as you'll be able to run forward to the problem, then walk backwards
>>>> through the code to see what happened to your state and file descriptors.
>>>>
>>>>
>>>> On Sat, May 8, 2021 at 11:27 AM pa...@rcom-software.com <
>>>> pa...@rcom-software.com> wrote:
>>>>
>>>>> Hi:
>>>>>
>>>>> Addition to my last message. When uv__nonblock() fails it is
>>>>> indicative of a Linux FIONBIO ioctl() failure. What would cause
>>>>> setting non-blocking mode to fail ?
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Paul R.
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "libuv" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to libuv+un...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/libuv/CADnnjUW6KL3OQw5C54aNfKh95z1OpQiq2bgVtXya8z_BeqMS9w%40mail.gmail.com
>>>> .
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> Paul Romero
>>>> -----------
>>>> RCOM Communications Software
>>>> EMAIL: pa...@rcom-software.com
>>>> PHONE: (510)482-2769 <(510)%20482-2769>
>>>>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "libuv" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to libuv+un...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/libuv/60981AE7.3000809%40rcom-software.com
>>>>
>>>> <https://groups.google.com/d/msgid/libuv/60981AE7.3000809%40rcom-software.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "libuv" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to libuv+un...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/libuv/CADnnjUWfszx9K8FdGPwGD%3D_5C0s1QVrDkaJD2ML2%3DEygefFD_A%40mail.gmail.com
>>
>> <https://groups.google.com/d/msgid/libuv/CADnnjUWfszx9K8FdGPwGD%3D_5C0s1QVrDkaJD2ML2%3DEygefFD_A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>> --
>>
>>
>> Paul Romero
>> -----------
>> RCOM Communications Software
>> EMAIL: pa...@rcom-software.com
>> PHONE: (510)482-2769 <(510)%20482-2769>
>>
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"libuv" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to libuv+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/libuv/843aca48-bc3a-4c4d-8260-241e6c56e0d5n%40googlegroups.com.
