Hi guys Felix and I discussed for a while at the OIC members meeting and we came to the following suggestion on how to achieve thread-safety for the context:
1) the context is reference-counted, so we will provide functions like: * ITVTContext_context_create() (ref counts up) * ITVTContext_context_unref() * ITVTContext_context_ref(ITVTContext *) On platforms that support so, the refcounter will be atomic (x86, ARMv7 and up, etc.). I have the code for this for GCC, ICC and Clang at http://code.qt.io/cgit/qt/qtbase.git/tree/src/3rdparty/forkfd/forkfd_gcc.h?h=5.5 (see ffd_atomic_add_fetch). It's BSD-licensed, which is compatible with Apache2, but since it's copyright by Intel, Intel is allowed to relicense. 2) on multithreaded systems, we'll do a simple producer-consumer solution for handling callbacks. The thread handling the socket parses the incoming CoAP message and determines what callback to activate. It then ref counts up the context and posts the callback into a thread-safe queue. A thread or a thread pool will read from the queue and dispatch the callback. This way, we ensure that the context cannot get freed until the callbacks have been handled and we ensure that the user callback taking a long time will not block socket handling. Additionally, since no locks were held prior to calling the user code, there should be no deadlocks in reentering the ITVTContext API. This solution requires memory allocation for the queue and a semaphore. 3) on thread-incapable systems, we will still use a queue but there won't be a thread or thread pool to consume the queue. Instead, shortly before going back to poll(2)ing the socket, the main socket loop will empty the queue. If memory is an issue, this code can also be modified to use a fixed-size and pre-allocated queue, so it will not pop and parse messages from the socket if the queue is full. 4) the main ITVTContext API needs to be protected by a mutex on systems that are thread-capable. On Linux, pthread_mutex is extremely lightweight. If need be, I can write code to use futex(2) directly, at the expense of not being able to use wait conditions. We discussed whether we should use a RW Lock instead and came to the conclusion that most calls into the API are expected to be mutating ones, so there isn't much value in having the more complex RW Lock. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center
