Hi Folks:
I worked out the kinks in my multi-task threaded Libuv oriented
application and an
overview of its salient features follows. It is now highly reliable and
I think this
overview closes the relevant Libuv issues and provides insight about
using Libuv
in a threaded environment.
My application includes two Libuv oriented tasks. Their primary
functions are performed
with standard Libuv mechanisms. Each task has its own uv_loop_t poll
loop structure which
is accessed only by that task and is used to perform related Libuv
operations. The tasks
can also perform Libuv operations on behalf of other tasks upon receipt
of Libuv async.
messages from other tasks. An overview of the Libuv task functionality
is as follows.
Libuv Task/Threads
------------------
* main(): Handles incoming TCP network connection establishment and
Linux/Unix signals.
(SSL connectivity is also handled by extension.)
* IO_Task(): Handles incoming network data.
The relevant files, which are attached for your convenience, are as follows.
Relevant Files
----------------
* main.c. The main() task. The make_incoming_connection() routine handles
incoming network connections and the connect_proxy() routine handles
async.
messages from other tasks.
* network_io.c: The IO_Task() task. The poll_callback() routine handles
incoming network data and the poll_proxy() routine handles async.
messages
from other tasks.
* async.c: The send_async_msg() routine is used to send messages to the
Libuv oriented tasks.
* ssl.c: Performs SSL related operations. This file is included to
resolve ambiguity in
the details of the make_incoming_connection() routine.
An overview of the other message oriented application tasks is as follows.
Message Oriented Task/Threads
---------------------------------
* Protocol_Task(): Implements the data transfer state machine used for
data transfer
between the Server and a Client on a per TCP connection basis.
* DB_Task(): Performs database insertion operations.
* Scram_Task(): Performs slow SCRAM authentication proof computation
operations.
* Transmit_Task(): Transmits data from the Server to the Client on a per
TCP connection basis.
* Timer_Task(): Manages protocol and application timers.
Best Regards,
Paul R.
On 05/14/2021 09:37 AM, pa...@rcom-software.com wrote:
Hi:
I think it is safe to perform connection acceptance and epoll()
oriented TCP socket
transactions in separate thread/tasks if you adhere to the following
rules.
My server follows these rules and works seamlessly with the following
exception. If it uses
Linux close() TCP socket descriptors, after uv_poll_stop() and
uv_close() have succeeded for a
particular uv_poll_t connection handle, internal Libuv structures are
corrupted
intermittently after a long period of successfully making and
releasing many TCP connections.
This implies that Libuv automatically closes TCP socket descriptors
because my Server
never runs out of them no matter how long it runs.
RULES
-----
* There is separate uv_loop_t Loop structure for accepting incoming
connections and epoll() I/O events.
In my case they are declared as follows:
uv_loop_t Connect_Loop; // Libuv loop for incoming connections.
static uv_loop_t Poll_Loop; // Libuv loop for reading incoming
network data.
* The Loop structures are owned by separate tasks and only accessed by
the task that owns the Loop structure.
In my case the main() task/thread owns Connect_Loop and the
IO_Task() thread/task owns Poll_Loop.
* Only the task/thread which owns the incoming connection acceptance
Loop calls connection management
related Libuv API routines, such uv_accept() and uv_close(), and
access uv_tcp_t connection
handles allocated during connection acceptance.
In my case the main() task/thread follows this rule and owns
Connect_Loop.
* Only the task/thread which owns the epoll() I/O event Loop calls
polling related Libuv API routines,
such as uv_poll_start() and uv_poll_stop(), and access uv_poll_t
connection handles allocated
during polling initiation.
In my case the the IO_Task() thread/task follows this rule and owns
the Poll_Loop.
Best Regards,
Paul R.
On Tuesday, May 11, 2021 at 6:45:59 AM UTC-7 pa...@rcom-software.com
wrote:
Hi:
Perhaps it would be useful to know that all Libuv epoll()
operations occur in the IO_Task() which
is dedicated to reception of networrk data. This provides nice
modularity and minimizes the
possibiliy of deadlocks due to interactions with other kinds of
functionality.
Best Regards,
Paul R.
On Sunday, May 9, 2021 at 12:14:28 PM UTC-7
pa...@rcom-software.com wrote:
Hi:
I get the FD immediately after accepting an incomig connection
with uv_fileno().
Then subsequently it is used as follows:
* IO_Task(): When the poll callback executes correctly, it is
used with Linux recv() to
read incoming network data in non-blocking mode. (All
incoming data and polling
is done in this thread/task.)
* Transmit_Task(): It is used with Linux send() to send data
across the network in non-blocking
mode. This thread/ task is driven by message reception
outside Libuv mechanisms.
That's it other than the final close() we discussed. I never
observed any problems unrelated
to Libuv connection acceptance and release, and starting and
stoping Libuv polling.
Best Regards,
Paul R.
On 05/09/2021 11:58 AM, Jameson Nash wrote:
Are you extracting a fd from a libuv object then calling
close or uv_poll on it? Either would cause problems, and
shouldn't be done.
On Sun, May 9, 2021 at 2:53 PM Jameson Nash
<vtj...@gmail.com> wrote:
uv_poll_stop and uv_accept must happen on the same
thread, so how are they racing? Once uv_poll_stop is
called, it should be safe to notify another thread to
call close, even if that then happens concurrently, as
the kernel should be thread-safe (though weird things may
happen in userspace if you close a fd that is
concurrently in use on another thread or event queue).
On Sun, May 9, 2021 at 1:24 PM Paul Romero
<pa...@rcom-software.com> wrote:
Hi:
Using TSAN did turn up one very signifcant problem.
The root cause of the TCP socket descriptor corrpution
is that accept4() executes concurrently in the main()
task, with close() in the Protocol_Task().
As an interim measure I avoid the problem by simply
not calling close(). So far this has worked
seemlesly and Libuv appears to automatically free
socket descriptors. The Libuv documentation
about this is somewhat ambiguous. It indicates that
after calling uv_poll_stop() or uv_close(),
for a particular uv_poll_t poll handle, the socket
descriptor is returned to the user per the Libuv
contract. I am not sure what that means, and in
particular, if it means the socket descriptor is freed.
Can you clarify this ?
A tagential issue is whether Linux accept4() and
close() are thread safe. I believe they
are and the crucial data is protected in the kernel.
Is it possible Libuv is not handling the accept4()
return codes correctly ? The Linux accept4() man page
details how errors should be handled and is it
somewhat fussy. The Linux close() man page also
details error handling but it is straight forward.
Also, I haven't been able to make the program
compiled with TSAN dump core. Do you have
any suggestions ? Incidentally, I had to use clang
rather than the usual gcc to get
TSAN to work on my system.
Best Regards,
Paul R.
On 05/08/2021 08:51 AM, Jameson Nash wrote:
Are you still accessing libuv (sans explicitly
thread-safe functions such as uv_async_send) from
multiple threads, as you mentioned earlier? If so,
I'd suggest fixing that first. In conjunction, I
recommend running TSan and making sure it runs
cleanly before checking for library or logic
problems. Then, if it is still a rare failure, I
recommend debugging under `rr` as you'll be able to
run forward to the problem, then walk backwards
through the code to see what happened to your state
and file descriptors.
On Sat, May 8, 2021 at 11:27 AM
pa...@rcom-software.com <pa...@rcom-software.com> wrote:
Hi:
Addition to my last message. When uv__nonblock()
fails it is indicative of a Linux FIONBIO
ioctl() failure. What would cause
setting non-blocking mode to fail ?
Best Regards,
Paul R.
--
You received this message because you are subscribed
to the Google Groups "libuv" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
libuv+un...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/libuv/CADnnjUW6KL3OQw5C54aNfKh95z1OpQiq2bgVtXya8z_BeqMS9w%40mail.gmail.com.
--
Paul Romero
-----------
RCOM Communications Software
EMAIL:pa...@rcom-software.com
PHONE:(510)482-2769 <tel:%28510%29%20482-2769>
--
You received this message because you are subscribed
to the Google Groups "libuv" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
libuv+un...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/libuv/60981AE7.3000809%40rcom-software.com.
--
You received this message because you are subscribed to the
Google Groups "libuv" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to libuv+un...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/libuv/CADnnjUWfszx9K8FdGPwGD%3D_5C0s1QVrDkaJD2ML2%3DEygefFD_A%40mail.gmail.com.
--
Paul Romero
-----------
RCOM Communications Software
EMAIL:pa...@rcom-software.com
PHONE:(510)482-2769 <tel:%28510%29%20482-2769>
--
You received this message because you are subscribed to a topic in the
Google Groups "libuv" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/libuv/_4ClQoaVPCg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
libuv+unsubscr...@googlegroups.com
<mailto:libuv+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/libuv/843aca48-bc3a-4c4d-8260-241e6c56e0d5n%40googlegroups.com
<https://groups.google.com/d/msgid/libuv/843aca48-bc3a-4c4d-8260-241e6c56e0d5n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Paul Romero
-----------
RCOM Communications Software
EMAIL: pa...@rcom-software.com
PHONE: (510)482-2769
--
You received this message because you are subscribed to the Google Groups
"libuv" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to libuv+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/libuv/60B4F7E7.1050901%40rcom-software.com.
#include "os_def.h"
#include "basic_def.h"
#include <uv.h>
#include "scram.h"
#include "framework.h"
#include <sys/types.h>
#include <sys/socket.h>
//
// Use this definition once debugging is complete.
//
// #define PRIVATE static
//
#define PRIVATE static
//
// ***********************************************************************************************
// *************************************** DATA **************************************************
// ***********************************************************************************************
//
//
// ***********************************************************************************************
// ************************************* PROTOTYPES *********************************************
// ***********************************************************************************************
//
void send_async_msg(CONN_DESC *, const ASYNC_DATA *);
void async_close_callback(uv_handle_t *);
void SendProtoMsg(CONN_DESC *, MSG *);
void insert_msg(MSG_FIFO *, MSG *);
MSG *MSG_ALLOC(int, BOOL);
void VFREE(UCHAR *);
//
// Send a message to the target loop task.
//
//
// FUNCTION - send_async_task(): Sends an async. message to the specified task.
//
// ARGUMENTS
// -----------
// cdesc: The connection descriptor corresponding to the connection from which message is sent.
// parm: Specifies the task to which the message will be sent, and the relevant task attributes.
//
// RETURN VALUE
// --------------
// None
//
// NOTES
// -------
// Care must be taken to specify the following attributes correctly.
//
// * type: The message type.
// * async_handle: The Libuv async. handle used to convey the message.
// * async_q: The message input queue of the task.
// * async_mutex: The Mutex that protects the queue.
// * object_handle: If this field is not NULL, it contains the Libuv handle necessary
// for performing the operation conveyed by the message type.
//
// The message is handled by a proxy routine which already must be specified via
// a earlier call to uv_async_init().
//
ROUTINE void send_async_msg(CONN_DESC *cdesc, const ASYNC_DATA *parm)
{
MSG *msg;
uv_handle_t **ppdata;
msg = MSG_ALLOC( sizeof(uv_handle_t *), FALSE );
msg->class = C_ASYNC;
msg->type = parm->type;
msg->conn_desc = (void *) cdesc;
//
// Store the Libuv handle if it is specified.
//
if(parm->object_handle)
{
ppdata = (uv_handle_t **) &msg->buffer[0];
*ppdata = (uv_handle_t *) parm->object_handle;
}
ENTER_MUTEX(parm->async_mutex);
insert_msg(parm->async_q, msg);
EXIT_MUTEX(parm->async_mutex);
uv_async_send(parm->async_handle);
return;
}
//
// FUNCTION - async_close_callback(): Frees a Libuv handle and notifies the task
// that initiated an async. operation that it is complete.
//
// ARGUMENTS
// -----------
// handle: The Libuv handle.
//
// RETURN VALUE
// --------------
// None
//
// NOTES
// -------
// This is a Libuv specific callback routine whose execution is specified by a call to uv_close().
//
ROUTINE void async_close_callback(uv_handle_t *handle)
{
MSG *msg;
CONN_DESC *cdesc = (CONN_DESC *) handle->data;
int type = handle->type;
VFREE((UCHAR *) handle);
//
// Send a notification message to the Protocol_Task.
//
msg = MSG_ALLOC(0, FALSE);
msg->class = C_NOTIFY;
msg->type = T_DISMANTLE_RSP;
msg->info = 0;
SendProtoMsg(cdesc, msg);
//
// Signal that the poll handle is released.
//
if(type == UV_POLL)
syslog(LOG_INFO, "IO_Task: CONN[%d] Poll Release Complete", cdesc->index);
return;
}
#include "os_def.h"
#include "basic_def.h"
#include <uv.h>
#include "scram.h"
#include "framework.h"
//
// Use this definition once debugging is complete.
//
// #define PRIVATE static
//
#define PRIVATE static
//
// *************************************************************************************
// ************************************ DATA *******************************************
// *************************************************************************************
//
//
// **************** Data Shared by multple Tasks ******************************
//
//
// Protects network connectivity data shared by the main() task and Protocol_Task()
//
uv_mutex_t Network_Conn_Mutex;
int N_Sockets; // The current number of connections accepted.
//
// Protects protocol data shared by or accessible to multiple tasks.
//
uv_mutex_t Shared_Data_Mutex;
int N_Tasks; // The number of initialized thread tasks.
//
// Connection Descriptor Table.
//------------------------------------
// Using a statically allocated table like this is only appropriate for
// an implementation with a small maximum number of connections.
//
// TBD: In the case of large maximum number of connections this table
// should be replaced by one that holds pointers to dynamically allocated
// connection descriptors.
//
CONN_DESC Conn_Desc_Table[MAX_CONN_DESC];
//
// **************************** main() Process/Task Data *********************
//
//
// Loop for detecting incoming connections and handling Linux signals.
// (Owned by the main() task.)
//
PRIVATE uv_loop_t Connect_Loop;
//
// TRUE when the connection accept callback is executing.
//
PRIVATE BOOL Connect_Accept_Busy = FALSE;
//
// TRUE when the connection plumbing proxy is executing.
//
PRIVATE BOOL Connect_Proxy_Busy = FALSE;
//
// The main() task handles incoming connections and Linux signals.
//
TASK_DATA Main_Tdata;
//
// **************************** IO_Task() Data *****************************
//
//
// The IO_Task() handles epoll() read I/O events with a poll loop of type uv_poll_t.
// It is dedicated to network data reception.
//
void IO_Task(void *arg);
TASK_DATA IO_Tdata;
//
// **************************** DB_Task() Data *************************
//
//
// The DB_Task performs slow database insertions.
//
void DB_Task(void *);
TASK_DATA DB_Tdata;
#ifdef USE_SCRAM
//
// ***************************** Scram_Task() Data *****************************
//
#ifndef DYNAMIC_SCRAM_THREAD
//
// The SCRAM task handles authentication with the remote Mobile.
//
void Scram_Task(void *);
TASK_DATA Scram_Tdata;
#endif // DYNAMIC_SCRAM_THREAD
#endif // USE_SCRAM
//
// ***************************** Timer_Task() Data **********************
//
//
// The Time_Task() implements protocol timers and handlles ticks every 1/10 of second in the Timer_Loop.
//
void Timer_Task(void *);
TASK_DATA Timer_Tdata;
//
// ************************* Trasnsit_Task() Data ******************************
//
//
// The tranmission task handle.
//
void Transmit_Task(void *);
TASK_DATA Transmit_Tdata;
//
// ************************* Protocol_Task() Data *****************************
//
//
// The Service Queue is doubly linked Circular queue containing
// only Connection Descriptors with SDUs on their Task Input Queue.
// It is mutex protected by the Task mutex.
//
CONN_DESC *Service_Q; // The head of the queue.
//
// The Protocol_Task() handles protocol messages from a remote Mobile.
//
void Protocol_Task(void *arg);
TASK_DATA Service_Tdata;
//
// ******************************************************************************************
// ********************************** PROTOTYPES *********************************************
// ******************************************************************************************
//
void send_async_msg(CONN_DESC *, const ASYNC_DATA *);
void async_close_callback(uv_handle_t *);
PRIVATE void make_incoming_connection(uv_stream_t *, int ); // This is a callback routine.
PRIVATE void connect_proxy(uv_async_t * ); // This callback routine executes in the main() process.
PRIVATE void forced_close_callback(uv_handle_t *);
char *host_to_ip(const char *, char [], int );
#ifdef USE_SCRAM
#ifdef DYNAMIC_SCRAM_THREAD
void scram_init_task_pool();
#endif // DYNAMIC_SCRAM_THREAD
#endif // USE_SCRAM
#ifdef USE_SSL
SSL *Ssl_Accept(CONN_DESC *, int );
void Ssl_ReleaseConnect(CONN_DESC *, uv_mutex_t * );
void Ssl_Init();
#ifdef USE_CRYPTO_LOCK
ROUTINE void Ssl_InitLock(int );
#endif // USE_CRYPTO_LOCK
#endif // USE_SSL
void sig_hup_callback(uv_signal_t *, int );
void sig_term_callback(uv_signal_t *, int );
void sig_kill_callback(uv_signal_t *, int );
MSG *remove_msg(MSG_FIFO *);
CONN_DESC *ALLOC_CONN_DESC(int );
BOOL DELETE_CONN(CONN_DESC *);
void MSG_FREE(MSG *);
void Msg_Task_Init(TASK_DATA *);
void Async_Task_Init(TASK_DATA *, uv_loop_t *, uv_async_cb );
void IO_Task_Init(TASK_DATA *);
void Timer_Task_Init(TASK_DATA *);
UCHAR *VALLOC(int n);
void VFREE(UCHAR *);
void VMEM_INIT();
#ifdef NO_DUP_USER
void duser_init();
#endif // NO_DUP_USER
#ifdef LIBUV_DETACH
int uv_thread_detached_create(uv_thread_t *, void (* )(void * ), void * );
#else // LIBUV_DETACH
int uv_thread_detach(uv_thread_t *);
#endif // LIBUV_DETACH
BOOL ip_address(char *, const char *, int );
void fatal_error(int , const char *, ...);
int main()
{
static char Server[NI_MAXHOST];
uv_tcp_t listen_handle;
uv_signal_t sig_term_handle, sig_hup_handle, sig_int_handle;
struct sockaddr_in addr;
int r, k;
//
// Open the debug log.
//
openlog("pexd", LOG_PID, LOG_LOCAL1);
syslog(LOG_ALERT, "PEXD DAEMON: VERSION %s STARTING", VERSION);
syslog(LOG_INFO, "DAEMON: Initializing System");
//
// Initalize memory managment.
//
VMEM_INIT();
#ifdef NO_DUP_USER
//
// Prevent multiple concurrent connections by the same database user.
//
duser_init();
#endif // NO_DUP_USER
//
// Initilize global data.
//
N_Tasks = 0;
N_Sockets = 0;
bzero((void *) Conn_Desc_Table, sizeof(Conn_Desc_Table));
for(k = 0; k < MAX_CONN_DESC; k++)
{
Conn_Desc_Table[k].index = k;
Conn_Desc_Table[k].fd = -1;
Conn_Desc_Table[k].busy = FALSE;
Conn_Desc_Table[k].decoder.buf_ptr = Conn_Desc_Table[k].decoder.buffer;
}
//
// Intialize the network data mutex.
//
uv_mutex_init(&Network_Conn_Mutex);
//
// Initialize the protocol data Mutex.
//
uv_mutex_init(&Shared_Data_Mutex);
//
// Intialize main() task data.
//
Connect_Accept_Busy = FALSE;
Connect_Proxy_Busy = FALSE;
Async_Task_Init(&Main_Tdata, &Connect_Loop, connect_proxy);
//
// Initialize IO_Task() data.
//
IO_Task_Init(&IO_Tdata);
//
// Initialize Protocol_Task() data.
//
// Initialize the Service_Q, used in the Protocol_Task() and
// its Mutex and condition variable.
//
Service_Q = NULL;
Msg_Task_Init(&Service_Tdata);
//
// Initialize DB_Task() data.
//
Msg_Task_Init(&DB_Tdata);
//
// Initialize Scram_Task() data.
//
Msg_Task_Init(&Scram_Tdata);
//
// Initialize Timer_Task() data.
//
Timer_Task_Init(&Timer_Tdata);
//
// Initialize Transmit_Task() data.
//
Msg_Task_Init(&Transmit_Tdata);
#ifdef USE_SSL
//
// Initialize SSL connection infrastructure.
//
Ssl_Init();
#ifdef USE_CRYPTO_LOCK
//
// Protect internal crypto. operations.
//
Ssl_InitLock(MAIN_TASK);
#endif // USE_CRYPTO_LOCK
#endif // USE_SSL
//
// Launch the Timer_Task(), IO_Task(), DB_Task(), Protocol_Task(), Transmit_Task() and Scram_Task().
//
// TBD: When a newer version of Libuv is available use uv_thread_detach() instead
// of uv_thread_detached_create(). Note that uv_thread_detached_create() is a custom
// routine I added to the src/unix/thread.c code and not part of the official Libuv
// package.
//
syslog(LOG_INFO, "DAEMON: Spawning Tasks");
#ifdef DYNAMIC_SCRAM_THREAD
scram_init_task_pool();
#endif // DYNAMIC_SCRAM_THREAD
#ifdef LIBUV_DETACH
uv_thread_detached_create(&IO_Tdata.handle, IO_Task, NULL);
uv_thread_detached_create(&Service_Tdata.handle, Protocol_Task, NULL);
uv_thread_detached_create(&DB_Tdata.handle, DB_Task, NULL);
#ifdef USE_SCRAM
#ifndef DYNAMIC_SCRAM_THREAD
uv_thread_detached_create(&Scram_Tdata.handle, Scram_Task, &Scram_Tdata);
#endif // DYNAMIC_SCRAM_THREAD
#endif // USE_SCRAM
uv_thread_detached_create(&Transmit_Tdata.handle, Transmit_Task, NULL);
uv_thread_detached_create(&Timer_Tdata.handle, Timer_Task, NULL);
#else // LIBUV_DETACH
tid = uv_thread_create(&IO_Tdata.handle, IO_Task, NULL);
uv_thread_detach(IO_Tdata.handle);
tid = uv_thread_create(&Service_Tdata.handle, Protocol_Task, NULL);
uv_thread_detach(Service_Tdata.handle);
tid = uv_thread_create(&DB_Tdata.handle, DB_Task, NULL);
uv_thread_detach(DB_Tdata.handle);
#ifdef USE_SCRAM
tid = uv_thread_create(&Scram_Tdata.handle, Scram_Task, Scram_Tdata);
uv_thread_detach(&Scram_Tdata.handle);
#endif // USE_SCRAM
tid = uv_thread_create(&Transmit_Tdata.handle, Transmit_Task, NULL);
uv_thread_detach(Transmit_Tdata.handle);
tid = uv_thread_create(&Timer_Tdata.handle, Timer_Task, NULL);
uv_thread_detach(Timer_Tdata.handle);
#endif // LIBUV_DETACH
//
// Initialize Linux signal handling in this task.
//
uv_signal_init(&Connect_Loop, &sig_term_handle);
uv_signal_start(&sig_term_handle, sig_term_callback, SIGTERM);
uv_signal_init(&Connect_Loop, &sig_hup_handle);
uv_signal_start(&sig_hup_handle, sig_hup_callback, SIGHUP);
uv_signal_init(&Connect_Loop, &sig_int_handle);
uv_signal_start(&sig_int_handle, sig_kill_callback, SIGKILL);
//
// TBD: This signal should be disabled in real life.
//
uv_signal_init(&Connect_Loop, &sig_int_handle);
uv_signal_start(&sig_int_handle, sig_kill_callback, SIGINT);
//
// Initialize the incoming connection handle.
//
uv_tcp_init(&Connect_Loop, &listen_handle);
//
// Bind to a listen address.
//
if(ip_address(Server, SERVER_DEVICE, IP_VERSION) != TRUE)
{
fatal_error(LOG_ALERT, "PEXD DAEMON: NO IP ADDRESS for DEV %s IP Version %d\n",
SERVER_DEVICE, IP_VERSION);
}
uv_ip4_addr(Server, SERVER_PORT, &addr);
if(uv_tcp_bind(&listen_handle, (const struct sockaddr *) &addr, 0))
{
fatal_error(LOG_ALERT, "PEXD DAEMON: Can't bind to address %s:%d. Error %d:%s\n",
Server, SERVER_PORT, errno, strerror(errno));
}
//
// Wait for all the thread tasks to be initialized.
//
for(;;)
{
ENTER_MUTEX(&Shared_Data_Mutex);
#ifdef DYNAMIC_SCRAM_THREAD
if(N_Tasks == 5)
#else // DYNAMIC_SCRAM_THREAD
if(N_Tasks == 6)
#endif // DYNAMIC_SCRAM_THREAD
{
EXIT_MUTEX(&Shared_Data_Mutex);
break;
}
EXIT_MUTEX(&Shared_Data_Mutex);
pthread_yield();
}
syslog(LOG_INFO, "MAIN_TASK: Started. ID %ld\n", uv_thread_self());
syslog(LOG_INFO, "DAEMON: All Spawned Tasks Running");
//
// Configure the listen socket.
//
if( (r = uv_listen((uv_stream_t*) &listen_handle, MAX_CONN_DESC, make_incoming_connection)) )
fatal_error(LOG_ERR, "MAIN_TASK: Can't Configure Listening Socket. Error %d: %s", r, uv_err_name(r));
syslog(LOG_ALERT, "PEXD DAEMON: RUNNING");
//
// Wait for a connection.
//
for(;;)
{
r = uv_run(&Connect_Loop, UV_RUN_DEFAULT);
if(r)
{
// fprintf(stderr, "MAIN: Run Error %d: %s\n", uv_err_name(r));
r = 0;
}
}
return(r);
}
#ifdef LIBUV_DETACH
//
// FUNCTION - uv_thread_detach(): Create a detached thread task and put into execution.
//
// ARGUMENTS
// -----------
// tid: Points to memory for holding the Livuv thread ID.
// entry: The thread task.
// arg: Points to data to be passed to the task when it starts.
//
// RETURN VALUE
// --------------
// Returns 0 if successful and a Libuv error code otherwise.
//
// NOTES
// -------
// This routine is kludge which supplies the same functionality as the uv_thread_detach() API.
// It is necessary because some version of Libuv don't include this functionality.
//
ROUTINE int uv_thread_detached_create(uv_thread_t *tid, void (*entry )(void * ), void *arg )
{
int rval;
rval = uv_thread_create(tid, entry, arg);
if(rval != 0)
rval = pthread_detach(*tid);
return(rval);
}
#endif // LIBUV_DETACH
//
// FUNCTION - make_incoming_connection(): A callback routine that executes when an incoming connection event occurs.
//
// ARGUMENTS
// -----------
// listen_handle: The TCP listen socket handle.
// status: The value is 0 when a connection detected without errors.
//
// RETURN VALUE
// --------------
// None
//
// NOTES
// -----
// The execution of this routine must be preconfigured with Libuv uv_listen().
//
ROUTINE PRIVATE void make_incoming_connection(uv_stream_t *listen_handle, int status)
{
ASYNC_DATA poll_data;
CONN_DESC *cdesc;
uv_tcp_t *conn_handle;
int r;
BOOL reject = FALSE;
if (status == -1)
fatal_error(LOG_ERR, "MAIN_TASK: Invalid Incoming Connection Status");
//
// If this routine is called before the work from the last invocation is finished,
// we have no choice but to reject the connection.
//
if(Connect_Accept_Busy)
reject = TRUE;
else
Connect_Accept_Busy = TRUE;
syslog(LOG_INFO, "MAIN_TASK: CONNECTION ATTEMPT: N Sockets = %d", N_Sockets);
//
// Initialize the connection handle.
//
conn_handle = (uv_tcp_t *) VALLOC(sizeof(uv_tcp_t));
uv_tcp_init(&Connect_Loop, conn_handle);
if((r = uv_accept(listen_handle, (uv_stream_t *) conn_handle)) == 0)
{
int nsock;
//
// A new connection occured.
//
uv_os_fd_t fd;
ENTER_MUTEX(&Network_Conn_Mutex);
nsock = N_Sockets++;
EXIT_MUTEX(&Network_Conn_Mutex);
if(reject || nsock >= MAX_CONN_DESC)
{
syslog(LOG_DEBUG, "MAIN_TASK: CONNECTION REJECTION - Transient, N Sockets = %d", nsock);
ENTER_MUTEX(&Network_Conn_Mutex);
N_Sockets--;
EXIT_MUTEX(&Network_Conn_Mutex);
conn_handle->data = (void *) NULL;
uv_close((uv_handle_t *) conn_handle, forced_close_callback);
return;
}
//
// Fetch the socket descriptor from the connection handle.
//
uv_fileno((const uv_handle_t*) conn_handle, &fd);
//
// Allocate the connection descriptor.
//
cdesc = ALLOC_CONN_DESC(fd);
if( !cdesc )
{
syslog(LOG_DEBUG, "MAIN_TASK: CONNECTION REJECTION - No Connection Descriptors, N = %d", nsock);
ENTER_MUTEX(&Network_Conn_Mutex);
N_Sockets--;
EXIT_MUTEX(&Network_Conn_Mutex);
cdesc->conn_handle = (uv_tcp_t *) conn_handle;
conn_handle->data = (void *) cdesc;
uv_close((uv_handle_t *) conn_handle, forced_close_callback);
return;
}
#ifdef USE_SSL
if( !Ssl_Accept(cdesc, fd) )
{
ENTER_MUTEX(&Network_Conn_Mutex);
N_Sockets--;
cdesc->busy = FALSE;
ENTER_MUTEX(&Service_Tdata.mutex);
DELETE_CONN(cdesc);
EXIT_MUTEX(&Service_Tdata.mutex);
EXIT_MUTEX(&Network_Conn_Mutex);
cdesc->conn_handle = (uv_tcp_t *) conn_handle;
conn_handle->data = (void *) cdesc;
uv_close((uv_handle_t *) conn_handle, forced_close_callback);
return;
}
#endif // USE_SSL
//
// Save the connection handle and start polling.
//
cdesc->conn_handle = (uv_tcp_t *) conn_handle;
syslog(LOG_INFO, "MAIN_TASK: NEW CONNECTION ESTABLISHED - CONN %d FD %d", cdesc->index, cdesc->fd);
//
// Set up epoll() plumbing by sending a message to IO_Task();
//
bzero((void *) &poll_data, sizeof(ASYNC_DATA));
poll_data.type = T_POLL_CONFIG_REQ;
poll_data.async_handle = &IO_Tdata.async_handle;
poll_data.async_q = &IO_Tdata.q;
poll_data.async_mutex = &IO_Tdata.mutex;
poll_data.object_handle = NULL;
//
// The IO_Task() sends the T_POLL_CONFIG_RSP message to the Protocol_Task(), which
// is in S_IDLE state, once polling has been initiated.
//
send_async_msg(cdesc, &poll_data);
}
else
fatal_error(LOG_ERR, "MAIN_TASK: CONNECTION REJECTION - Libuv Accept Failure. Error %d: %s", r, uv_err_name(r));
Connect_Accept_Busy = FALSE;
return;
}
//
// FUNCTION - connect_proxy(): Dismantles and releases a Libuv connection handle on behalf of another task.
// The proxy routine executes in the main() process/task bound to the Connect_Loop.
//
// ARGUMENTS
//-----------
// handle: The async. channel handle. (Connect_Task_Async_Handle)
//
// RETURN VALUE
// -----------
// None
//
// NOTES
// --------
// Execution of the routine is triggered by a call to async_send() in the Protocol_Task().
//
ROUTINE PRIVATE void connect_proxy(uv_async_t *handle)
{
CONN_DESC *cdesc;
MSG *msg;
uv_tcp_t *conn_handle;
uv_handle_t **ppdata;
BOOL done;
syslog(LOG_DEBUG, "MAIN_TASK: CONNECT PROXY\n");
//
// Avoid clobbering a previous invocation of this routine.
//
if(Connect_Proxy_Busy)
return;
Connect_Proxy_Busy = TRUE;
//
// Handle all messages from the Protocol_Task()
//
done = FALSE;
while(done != TRUE)
{
ENTER_MUTEX(&Main_Tdata.mutex);
msg = remove_msg(&Main_Tdata.q);
EXIT_MUTEX(&Main_Tdata.mutex);
if(msg)
{
cdesc = (CONN_DESC *) msg->conn_desc;
syslog(LOG_DEBUG, "MAIN_TASK(Proxy): CONN[%d] Type %d\n", cdesc->index, msg->type);
switch(msg->type)
{
case T_CONN_DISMANTLE_REQ:
//
// Release a uv_tcp_t connection handle.
// The protocol task is notified by async_close_callback()
// when the operation is complete.
//
ppdata = (uv_handle_t **) &msg->buffer[0];
conn_handle = (uv_tcp_t *) *ppdata;
conn_handle->data = (void *) cdesc;
uv_close((uv_handle_t *) conn_handle, async_close_callback);
break;
default:
fatal_error(LOG_EMERG, "MAIN_TASK: CONNECT_PROXY - Invalid Message = %d", msg->type);
}
MSG_FREE(msg);
}
else
done = TRUE;
}
Connect_Proxy_Busy = FALSE;
return;
}
//
// FUNCTION - force_close_callback(): Deallocate a Libuv handle.
//
// ARGUMENTS
//-----------
// handle: The handle.
//
// NOTES
// ------
// Releases a uv_tcp_t handle in the case it was allocated before an incoming connection was rejected.
//
ROUTINE PRIVATE void forced_close_callback(uv_handle_t *handle)
{
CONN_DESC *cdesc = (CONN_DESC *) handle->data;
VFREE((UCHAR *) handle);
if(cdesc)
{
close(cdesc->fd);
cdesc->conn_handle = NULL;
}
// printf("ASYNC[X]: REJECT COMPLETE\n");
// fflush(stdout);
//
// Setting this variable indicates the handle has been freed.
//
Connect_Accept_Busy = FALSE;
return;
}
ROUTINE void Msg_Task_Init(TASK_DATA *ptask)
{
bzero((void *) ptask, sizeof(TASK_DATA));
ptask->q.head = ptask->q.tail = NULL;
uv_mutex_init(&ptask->mutex);
uv_cond_init(&ptask->cond);
return;
}
ROUTINE void Async_Task_Init(TASK_DATA *ptask, uv_loop_t *ploop, uv_async_cb proxy_routine)
{
bzero((void *) ptask, sizeof(TASK_DATA));
ptask->q.head = ptask->q.tail = NULL;
uv_mutex_init(&ptask->mutex);
uv_cond_init(&ptask->cond);
uv_loop_init(ploop);
uv_async_init(ploop, &ptask->async_handle, proxy_routine);
return;
}
#include "os_def.h"
#include "basic_def.h"
#include <uv.h>
#include "scram.h"
#include "framework.h"
//
// Use this definition once debugging is complete.
//
// #define PRIVATE static
//
#define PRIVATE static
//
// **********************************************************
// ******************** DATA ***********************************
// **************************************************************
//
//
// This data is used to indicate when the task is initialized.
//
extern uv_mutex_t Shared_Data_Mutex;
extern int N_Tasks;
//
// Task data.
//
extern TASK_DATA IO_Tdata;
//
// Loop for handling epoll() read I/O events.
//
PRIVATE uv_loop_t Poll_Loop;
//
// TRUE when the Poll proxy routine is executing.
//
PRIVATE BOOL Poll_Proxy_Busy;
//
// *******************************************************
// *********************** PROTOTYPES *******************
// *****************************************************
//
PRIVATE void poll_callback(uv_poll_t *, int , int );
PRIVATE void poll_proxy(uv_async_t *);
void async_close_callback(uv_handle_t *);
#ifdef USE_SSL
int Ssl_RxData(SSL *, UCHAR *, int );
#endif // USE_SSL
DECODE_RESULT rx_sdu(CONN_DESC *, int);
void SendProtoMsg(CONN_DESC *, MSG *);
void IO_Task_Init(TASK_DATA *);
void Async_Task_Init(TASK_DATA *, uv_loop_t *, uv_async_cb );
void insert_msg(MSG_FIFO *, MSG *);
MSG *remove_msg(MSG_FIFO *);
MSG *MSG_ALLOC(int, BOOL);
void MSG_FREE(MSG *);
UCHAR *VALLOC(int n);
void VFREE(UCHAR *);
void fatal_error(int , const char *, ...);
// void verify_watch(void *, uv_loop_t *);
//
// Monitor connections for incoming data with epoll()
//
TASK void IO_Task(void *arg)
{
int r;
ENTER_MUTEX(&Shared_Data_Mutex);
N_Tasks++;
EXIT_MUTEX(&Shared_Data_Mutex);
syslog(LOG_INFO, "IO_TASK: Started. ID %ld\n", uv_thread_self());
for(;;)
{
r = uv_run(&Poll_Loop, UV_RUN_DEFAULT);
if(r)
{
syslog(LOG_ERR, "IO_TASK: Run Error %d", r);
r = 0;
}
}
return;
}
//
// FUNCTION - poll_callback(): A callback routine that executes when incoming data is available.
// and reads the data.
//
// ARGUMENTS
// ---------------
// poll_handle: The Libuv poll handle.
// status: The value is 0 if execution occurs due to incoming data.
// events: The value should always be UV_READABLE, which means incoming
// data is available, if status is 0.
//
// RETURNS
//--------
// None
//
// NOTES
// ------
// This routine is invoked by the Libuv epoll() mechanism.
//
ROUTINE PRIVATE void poll_callback(uv_poll_t *poll_handle, int status, int events)
{
if(status == 0)
{
if(events & UV_READABLE)
{
//
// Since the right connection descriptor is known, all you
// neeed to to is read data into the connection descriptor buffer.
//
int n;
MSG *sdu;
CONN_DESC *cdesc = (CONN_DESC *) poll_handle->data;
#ifdef USE_SSL
n = Ssl_RxData(cdesc->ssl_socket, cdesc->rx_buf, RX_BUF_SIZE);
#else // USE_SSL
n = recv(cdesc->fd, cdesc->rx_buf, RX_BUF_SIZE, MSG_DONTWAIT);
#endif // USE_SSL
if(n > 0)
{
DECODE_RESULT r;
//
// Recognize the SDU in the usual way.
// If the SDU is complete, send the it to the Protocol_Task().
//
r = rx_sdu(cdesc, n);
switch(r)
{
case RX_PARTIAL:
case RX_COMPLETE:
case RX_EMPTY:
break;
case RX_OVERFLOW:
case RX_NOMEMORY:
//
// There is no choice but to send an abort, reinitialize everything,
// and release the connection.
//
syslog(LOG_ERR, "IO_TASK: Irrecoverable SDU Reception Error %d", r);
sdu = MSG_ALLOC(0, FALSE);
sdu->class = C_NOTIFY;
sdu->type = T_FAULT;
sdu->info = (int) r;
SendProtoMsg(cdesc, sdu);
break;
default:
fatal_error(LOG_EMERG, "IO_TASK: Invalid SDU Recognition Code, %d", r);
}
}
else
if(n < 0)
{
#ifdef USE_SSL
MSG *msg;
n = 0;
syslog(LOG_INFO, "IO_TASK: RX DISCONNECT %d - %s !", errno, strerror(errno) );
msg = MSG_ALLOC(0, FALSE);
msg->class = C_NOTIFY;
msg->type = T_DISCONNECT;
msg->info = 0;
SendProtoMsg(cdesc, msg);
#else // USE_SSL
n = 0;
if( !(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) )
{
MSG *msg;
syslog(LOG_INFO, "IO_TASK: RX DISCONNECT %d - %s !", errno, strerror(errno) );
msg = MSG_ALLOC(0, FALSE);
msg->class = C_NOTIFY;
msg->type = T_DISCONNECT;
msg->info = 0;
SendProtoMsg(cdesc, msg);
}
#endif // USE_SSL
}
}
}
else
syslog(LOG_ERR, "IO_TASK: poll_callback - Bad Status %d", status);
return;
}
//
// FUNCTION - poll_proxy(): The proxy routine executes in the IO_Task() Poll Loop when
// it has unserviced async. messages from other tasks.
//
// ARGUMENTS
// -----------
// handle: A Libuv async. operation handle.
//
// RETURNS
// -------
// Nothing
//
// NOTES
// -----
// The main() and Protocol_Tasks() use the async. handled to send messages to the IO_Task().
// They do so by inserting messages in the IO_Task() IO_Tdata.input_q which is in shared memory.
//
ROUTINE PRIVATE void poll_proxy(uv_async_t *handle)
{
CONN_DESC *cdesc;
MSG *msg, *rsp;
uv_poll_t *poll_handle;
uv_handle_t **ppdata;
int r;
BOOL done;
syslog(LOG_DEBUG, "IO_TASK: POLL PROXY");
//
// Do nothing if execution is already in progress.
//
if(Poll_Proxy_Busy)
return;
Poll_Proxy_Busy = TRUE;
//
// Handle messages from other tasks.
//
done = FALSE;
while(done != TRUE)
{
ENTER_MUTEX(&IO_Tdata.mutex);
msg = remove_msg(&IO_Tdata.q);
EXIT_MUTEX(&IO_Tdata.mutex);
if(msg)
{
cdesc = (CONN_DESC *) msg->conn_desc;
syslog(LOG_DEBUG, "IO_TASK(Proxy): RX EVENT - CONN[%d] Type %d\n", cdesc->index, msg->type);
switch(msg->type)
{
case T_POLL_CONFIG_REQ:
//
// Main process request to start Libuv/epoll() operation.
//
poll_handle = (uv_poll_t *) VALLOC(sizeof(uv_poll_t));
if( (r = uv_poll_init(&Poll_Loop, poll_handle, cdesc->fd)) != 0)
{
fatal_error(LOG_CRIT, "IO_TASK: POLL_PROXY - Polling Initialization Error %d, %s",
r, uv_err_name(r));
}
poll_handle->data = (void *) cdesc;
if((r = uv_poll_start(poll_handle, UV_READABLE, poll_callback)) < 0)
{
fatal_error(LOG_CRIT, "IO_TASK: POLL_PROXY - Polling Initiation Error %d, %s",
r, uv_err_name(r));
}
//
// Notify the Protocol_Task(), that polling has started.
//
rsp = MSG_ALLOC(0, FALSE);
rsp->class = C_NOTIFY;
rsp->type = T_POLL_CONFIG_RSP;
rsp->info = 0;
rsp->conn_desc = (void *) poll_handle;
SendProtoMsg(cdesc, rsp);
break;
case T_POLL_DISMANTLE_REQ:
//
// Protocol_Task() request to cease Libuv/epoll() operation and
// release the poll handle and its resources.
//
ppdata = (uv_handle_t **) &msg->buffer[0];
poll_handle = (uv_poll_t *) *ppdata;
if( (r = uv_poll_stop(poll_handle)) )
{
fatal_error(LOG_CRIT, "IO_TASK: POLL_PROXY - Poll Stop Error %d, %s",
r, uv_err_name(r) );
}
//
// The callback routine notifies the Protocol_Task() when everything is done.
//
poll_handle->data = (void *) cdesc;
uv_close((uv_handle_t *) poll_handle, async_close_callback);
break;
default:
fatal_error(LOG_EMERG, "IO_TASK: POLL_PROXY - Invalid Message = %d", msg->type);
}
if(msg)
MSG_FREE(msg);
}
else
{
done = TRUE;
}
}
Poll_Proxy_Busy = FALSE;
return;
}
ROUTINE void IO_Task_Init(TASK_DATA *pdata)
{
Poll_Proxy_Busy = FALSE;
Async_Task_Init(pdata, &Poll_Loop, poll_proxy);
return;
}
#include "os_def.h"
#include "basic_def.h"
#include <uv.h>
#include "scram.h"
#include "framework.h"
#ifdef USE_SSL
//
// Use this definition once debugging is complete.
//
// #define PRIVATE static
//
#define PRIVATE static
// #define VTRACE
//
// **************************************************************************************
// ************************************ DATA *******************************************
// **************************************************************************************
//
//
// Protects the Service_Q and its condition variable.
//
extern uv_mutex_t Network_Conn_Mutex;
#ifdef USE_CRYPTO_LOCK
//
// An array of Mutexes used by the cryptography software and libraries to
// prevent corruption of shared memory by the main task/thread.
//
PRIVATE uv_mutex_t *Main_Ssl_Mutex_List;
#endif // USE_CRYPTO_LOCK
//
// *****************************************************************************************
// *********************************** PROTOTYPES *******************************************
// *****************************************************************************************
//
SSL *Ssl_Accept(CONN_DESC *, int );
void Ssl_ReleaseConnect(CONN_DESC *, uv_mutex_t * );
int Ssl_RxData(SSL *, UCHAR *, int );
int Ssl_TxData(SSL *, const UCHAR *, int );
void Ssl_Init();
PRIVATE void Ssl_GetCredentials(SSL_CTX * );
#ifdef MOBILE_AUTH_SSL
PRIVATE BOOL Ssl_ConfigTrust(SSL_CTX *, X509_DESC * );
void Ssl_ReleaseAuth(CONN_DESC * );
#ifdef USE_CRYPTO_LOCK
void Ssl_InitLock(int );
PRIVATE void Ssl_TaskID(CRYPTO_THREADID * );
PRIVATE void Ssl_MainCryptoLock(int , int , const char *, int );
#endif // USE_CRYPTO_LOCK
#endif // MOBILE_AUTH_SSL
#ifdef VTRACE
void PRIVATE ShowCerts(SSL *);
#endif // VTRACE
void fatal_error(int , const char *, ...);
UCHAR *VALLOC(int n);
void VFREE(UCHAR *);
//
// FUNCTION - Ssl_Accept(): Establishes an incoming SSL connection, including insuring
// assessing the client authentication handshake result.
//
// ARGUMENTS
// -----------
// cdesc: A Connection Descriptor.
// sock_fd: The TCP socket descriptor obtained during incoming TCP connection establishment.
//
// RETURN VALUE
// --------------
// Returns an SSL operation socket handle if successful and NULL otherwise.
//
// NOTES
// -------
// If MOBILE_AUTH_SSL is defined, the Server validates the Client.
// The authentication handshake is just configured and initiated in the Server,
// and the handshake protocol exchanges occur within the SSL layer.
//
ROUTINE SSL *Ssl_Accept(CONN_DESC *cdesc, int sock_fd)
{
#ifdef MOBILE_AUTH_SSL
X509_DESC *x509_data;
#endif // MOBILE_AUTH_SSL
SSL_CTX *ctx;
SSL_METHOD *method;
SSL *socket;
int flags, r;
BOOL error;
// printf("TRY\n");
syslog(LOG_INFO, "MAIN_TASK: ATTEMPT SSL CONNECTION - CONN %d FD %d", cdesc->index, cdesc->fd);
ENTER_MUTEX(&Network_Conn_Mutex);
cdesc->ssl_ctx = NULL;
cdesc->ssl_socket = NULL;
EXIT_MUTEX(&Network_Conn_Mutex);
#ifdef MOBILE_AUTH_SSL
x509_data = &cdesc->x509_data;
bzero((void *) x509_data, sizeof(X509_DESC));
#endif // MOBILE_AUTH_SSL
//
// Allocate an SSL/TLS V1.2 Context.
//
method = (SSL_METHOD *) TLSv1_2_server_method();
//method = (SSL_METHOD *) SSLv3_server_method();
if( (ctx = SSL_CTX_new(method)) == NULL)
{
syslog(LOG_ERR, "MAIN_TASK: SSL Error - Context Allocation Failure");
ERR_print_errors_fp(stderr);
abort();
}
else
{
ENTER_MUTEX(&Network_Conn_Mutex);
cdesc->ssl_ctx = ctx;
EXIT_MUTEX(&Network_Conn_Mutex);
SSL_CTX_set_options(ctx,
(SSL_OP_TLS_BLOCK_PADDING_BUG | SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_TLSv1));
// (SSL_OP_TLS_BLOCK_PADDING_BUG | SSL_OP_NO_SSLv2 | SSL_OP_NO_TLSv1));
}
Ssl_GetCredentials(ctx);
//
// Allocate a new SSL socket operation handle.
//
if( !(socket = SSL_new(ctx)) )
{
syslog(LOG_INFO, "MAIN_TASK: CONN[%d] SSL Error - Context Allocation Failure", cdesc->index);
ERR_print_errors_fp(stderr);
ENTER_MUTEX(&Network_Conn_Mutex);
cdesc->ssl_ctx = NULL;
EXIT_MUTEX(&Network_Conn_Mutex);
SSL_CTX_free(ctx);
}
else
{
ENTER_MUTEX(&Network_Conn_Mutex);
cdesc->ssl_socket = socket;
EXIT_MUTEX(&Network_Conn_Mutex);
//
// Configure the SSL authentication characteristics.
//
SSL_set_accept_state(socket);
#ifdef MOBILE_AUTH_SSL
//
// The Server will authenticate the Client.
//
SSL_set_verify(socket, (SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT), NULL);
SSL_set_verify_depth(socket, 1);
//
// Configure the Server as a trusted CA to enable authentication of the Client.
//
if(Ssl_ConfigTrust(ctx, x509_data) != TRUE)
{
Ssl_ReleaseConnect(cdesc, &Network_Conn_Mutex);
socket = NULL;
ctx = NULL;
}
#else // MOBILE_AUTH_SSL
//
// No authentication.
//
SSL_set_verify(socket, SSL_VERIFY_NONE, NULL);
#endif // MOBILE_AUTH_SSL
if(socket)
{
// Establish the SSL connection.
//
// Set up SSL socket I/O
//
if( (r = SSL_set_fd(socket, sock_fd)) != 1)
{
syslog(LOG_ERR, "MAIN_TASK: SSL Error - Can't Set Socket Descriptor: Return %d Error %d",
r, SSL_get_error(socket, r));
ERR_print_errors_fp(stderr);
abort();
}
//
// Accept the SSL connection when the authentication handshake is complete.
//
error = FALSE;
while ((r = SSL_accept(socket)) != 1 ) /* do SSL-protocol accept */
{
r = SSL_get_error(socket, r);
switch(r)
{
case SSL_ERROR_WANT_READ:
case SSL_ERROR_WANT_WRITE:
pthread_yield();
break;
default:
error = TRUE;
}
if(error)
break;
}
if(error)
{
printf("SNAG %d: %s\n", errno, strerror(errno));
syslog(LOG_INFO, "MAIN_TASK: CONN[%d] SSL CONNECTION REJECTED: Return %d Error %d",
cdesc->index, r, SSL_get_error(socket, r));
ERR_print_errors_fp(stderr);
Ssl_ReleaseConnect(cdesc, &Network_Conn_Mutex);
socket = NULL;
ctx = NULL;
}
else
{
syslog(LOG_INFO, "MAIN_TASK: CONN[%d] SSL CONNECTION ESTABLISHED", cdesc->index);
// printf("OK\n");
#ifdef VTRACE
//
// Display certficates from client.
//
ShowCerts(socket);
#endif // VTRACE
//
// Configure the socket for non-blocking operation.
// Simple experimentation confirms this does not break Libuv and works.
//
flags = fcntl(sock_fd, F_GETFL, 0);
fcntl(sock_fd, F_SETFL, (flags | O_NONBLOCK));
}
}
}
return(socket);
}
#ifdef MOBILE_AUTH_SSL
//
// FUNCTION - Ssl_ConfigTrust(): Configures Server authentication of the Client.
//
// ARGUMENTS
// -----------
// ctx: An SSL context.
// x509_data: Points a caller allocated structure for holding the essential X509 authentication items.
//
// RETURN VALUE
// --------------
// Returns TRUE if successful and FALSE otherwise.
//
// NOTES
// -------
// None
//
ROUTINE PRIVATE BOOL Ssl_ConfigTrust(SSL_CTX *ctx, X509_DESC *x509_data)
{
X509_STORE *pstore = NULL;
X509_STORE_CTX *storeCtx = NULL;
BOOL rval = TRUE;
int r = 0;
syslog(LOG_DEBUG, "MAIN_TASK: Configure SSL Authentication");
//
// We no longer load the client's public certificate into memory
// because the underlying bugs making it necessary are fixed.
//
// Now allocate a store context
//
if( !(storeCtx = X509_STORE_CTX_new()) )
{
rval = FALSE;
syslog(LOG_ERR, "MAIN_TASK: X509 Error - Store Context Allocation Failure");
}
else
//
// First, fabricate a temporary trust store and
// add the CA Certificate to the new trust store.
//
if( !(pstore = X509_STORE_new()) )
{
rval = FALSE;
syslog(LOG_INFO, "MAIN_TASK: X509 Error - Store Allocation Failure");
}
else
if( (r = X509_STORE_load_locations(pstore, SSL_CA_CERT_FILE, NULL)) != 1)
syslog(LOG_ERR, "MAIN_TASK: X509 Error - Store Certificate Addition Failure");
else
if( (r = X509_STORE_set_default_paths(pstore)) != 1)
syslog(LOG_ERR, "MAIN_TASK: X509 Error - Store Path Configuration Failure");
else
//
// Now configure the authentication characteristics.
//
if( (r = X509_STORE_CTX_init(storeCtx, pstore, NULL, NULL)) != 1)
syslog(LOG_ERR, "MAIN_TASK: X509 Error - Store Initialization Failure");
else
{
//
// Insert the new store settings in the SSL context.
//
// Finally set the authentication characteristics.
// This flag is just enhance debugging on most SSL versions and
// should be set.
//
SSL_CTX_set_cert_store(ctx, pstore);
X509_STORE_CTX_set_flags(storeCtx, ( X509_V_FLAG_CB_ISSUER_CHECK ) );
}
if(r != 1)
{
syslog(LOG_ERR, "MAIN_TASK: SSL Authentication Configuration Error");
ERR_print_errors_fp(stderr);
abort();
}
if(rval)
{
syslog(LOG_DEBUG, "MAIN_TASK: Authentication Configuration Succeeded");
x509_data->storeCtx = storeCtx;
x509_data->store = pstore;
}
else
{
syslog(LOG_INFO, "MAIN_TASK: Transient SSL Authentication Configuration Failure");
if(pstore)
X509_STORE_free(pstore);
if(storeCtx)
{
X509_STORE_CTX_cleanup(storeCtx);
X509_STORE_CTX_free(storeCtx);
}
}
return(rval);
}
#endif // MOBILE_AUTH_SSL
//
// FUNCTION - Ssl_TxData(): Writes data to the SSL network socket.
//
// ARGUMENTS
// ---------
// socket: The SSL socket.
// buf: The buffer with the data.
// nbytes: The number of bytes to be written.
//
// RETURNS
// -------
// Returns the number of bytes written if successful and -1 if an irrecoverable error occurs.
//
// NOTES
// -----
// If the return value is not -1, it is possible to write more data on the socket.
// Otherwise, for logical purposes, the socket connection is lost.
//
// A threaded environment is assumed.
//
ROUTINE int Ssl_TxData(SSL *socket, const UCHAR *buf, int nbytes)
{
int n, err;
int rval;
if(nbytes <= 0)
{
if(nbytes < 0)
rval = -1;
return(rval);
}
n = SSL_write(socket, buf, nbytes);
if(n > 0)
rval = n;
else
{
err = SSL_get_error(socket, n);
switch(err)
{
case SSL_ERROR_NONE:
case SSL_ERROR_WANT_WRITE:
case SSL_ERROR_WANT_READ:
//
// Retry
//
pthread_yield();
break;
case SSL_ERROR_SYSCALL:
case SSL_ERROR_ZERO_RETURN:
//
// The write failed and we have to asuume to connection is lost.
// If errno is ECONNRESET, the connection is lost due to a remote disconnect.
//
ERR_print_errors_fp(stderr);
rval = -1;
break;
case SSL_ERROR_WANT_CONNECT:
case SSL_ERROR_WANT_ACCEPT:
case SSL_ERROR_WANT_X509_LOOKUP:
//
// We should never get this return value because, the X509 authentication
// should be complete when this routine is called, the client cannot
// initiate renegotriation, and the server does not currently initiate
// renegotiaon.
//
syslog(LOG_EMERG, "TRANSMIT_TASK: SSL Error - Unexpected Negotiation");
ERR_print_errors_fp(stderr);
abort();
break;
case SSL_ERROR_SSL:
syslog(LOG_EMERG, "TRANSMIT_TASK: SSL Error - Libary Failure");
ERR_print_errors_fp(stderr);
abort();
break;
default:
fatal_error(LOG_EMERG, "BUG - Invalid SSL Write Return = %d", err);
}
}
return(rval);
}
//
// FUNCTION - Ssl_RxData(): Reads incoming network data from an SSL socket.
//
// ARGUMENTS
// ---------
// socket: The SSL socket.
// buf: The buffer for the data.
// size: Ths size of the buffer.
//
// RETURNS
// -------
// Returns the postive number of bytes read if successful, 0 if no data is read, and
// -1 if an irrecoverable error occurs.
//
// NOTES
// -----
// If the return value is not -1, it is possible more data may arrive on the socket.
// Otherwise, for logical purposes, the socket connection is lost.
//
// A threaded environment is assumed.
//
ROUTINE int Ssl_RxData(SSL *socket, UCHAR *buf, int size)
{
int n, err;
// printf("START READ\n");
n = SSL_read(socket, buf, size);
if(n <= 0)
{
err = SSL_get_error(socket, n);
// printf("ERR %d\n", err);
switch(err)
{
case SSL_ERROR_NONE:
//
// No data.
//
break;
case SSL_ERROR_WANT_READ:
case SSL_ERROR_WANT_WRITE:
//
// Retry later.
//
if(n <= 0)
pthread_yield();
break;
case SSL_ERROR_SYSCALL:
if(n && errno != EINTR)
{
printf("Bad Call ! E %d\n", errno);
//
// Assume the connection is lost in all cases.
// If errno is ECONNRESET, it is lost due to a remote disconnect.
//
syslog(LOG_ERR, "IO_TASK: SSL Error - Read Failure. Errno %d:%s",
errno, strerror(errno));
ERR_print_errors_fp(stderr);
if(n <= 0)
n = -1;
}
else
{
//
// No data
//
// TBD: It isn't clear if there is still a connection at this point.
//
if(errno == ECONNRESET)
{
syslog(LOG_ERR, "IO_TASK: SSL Error - Connection Loss on Read");
ERR_print_errors_fp(stderr);
if(n <= 0)
n = -1;
}
break;
}
break;
case SSL_ERROR_WANT_CONNECT:
case SSL_ERROR_WANT_ACCEPT:
case SSL_ERROR_WANT_X509_LOOKUP:
//
// We should never get this return value because, the X509 authentication
// should be complete when this routine is called, the client cannot
// initiate renegotriation, and the server does not currently initiate
// renegotiaon.
//
syslog(LOG_EMERG, "TRANSMIT_TASK: SSL Error - Unexpected Negotiation");
ERR_print_errors_fp(stderr);
abort();
break;
case SSL_ERROR_ZERO_RETURN:
syslog(LOG_ERR, "IO_TASK: SSL Error - Unexpected Connection Loss");
if(n <= 0)
n = -1;
break;
case SSL_ERROR_SSL:
syslog(LOG_EMERG, "IO_TASK: SSL Error - Library Failure");
ERR_print_errors_fp(stderr);
abort();
break;
default:
fatal_error(LOG_EMERG, "IO_TASK: BUG - Invalid SSL Read Return = %d", err);
}
}
// printf("RX DONE: %d\n", n);
return(n);
}
#ifdef VTRACE
//
// Display certificates sent by the Client.
//
ROUTINE PRIVATE void ShowCerts(SSL* ssl)
{
X509 *cert;
char *line;
cert = SSL_get_peer_certificate(ssl); /* Get certificates (if available) */
if ( cert != NULL )
{
syslog(LOG_DEBUG, "MAIN_TASK: *** Client Certificate ***");
line = X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
syslog(LOG_DEBUG, "Subject: %s", line);
free(line);
line = X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
syslog(LOG_DEBUG, "Issuer: %s", line);
free(line);
X509_free(cert);
}
else
syslog(LOG_DEBUG, "MAIN_TASK: *** No Client Certificate ***");
return;
}
#endif // VTRACE
//
// FUNCTION - Ssl_GetCreditails(): Configure the SSL context with the certificate files
// required for SSL connection establishment.
//
// ARGUMENTS
// -----------
// ctx: An SSL context.
//
// RETURN VALUE
// --------------
// None
//
// NOTES
// -------
// None
//
ROUTINE PRIVATE void Ssl_GetCredentials(SSL_CTX* ctx)
{
int r = 0;
if( (r = SSL_CTX_use_certificate_file(ctx, SSL_SERVER_CERT_FILE, SSL_FILETYPE_PEM)) != 1)
syslog(LOG_ERR, "MAIN_TASK: SSL Error - Invalid Server Certificate File = %s", SSL_SERVER_CERT_FILE);
else
if( (r = SSL_CTX_use_PrivateKey_file(ctx, SSL_SERVER_KEY_FILE, SSL_FILETYPE_PEM)) != 1)
syslog(LOG_ERR, "MAIN_TASK: SSL Error - Invalid Client Key File = %s", SSL_SERVER_KEY_FILE);
else
if ( (r = SSL_CTX_check_private_key(ctx)) != 1)
syslog(LOG_ERR, "MAIN_TASK: SSL Error -Private Key Public Certificate Mismatch");
if(r != 1)
{
ERR_print_errors_fp(stderr);
abort();
}
return;
}
//
// FUNCTION - ReleaseConnect(): the Releases resources during an SSL encryption session.
//
// ARGUMENTS
// -----------
// cdesc: A connection descriptor.
//
// RETURN VALUE
// --------------
// None
//
// NOTES
// -------
// None
//
ROUTINE void Ssl_ReleaseConnect(CONN_DESC *cdesc, uv_mutex_t *mutex)
{
SSL_CTX *ctx;
SSL *sock;
syslog(LOG_INFO, "PROTO_TASK: CONN[%d] SSL Release Resources", cdesc->index);
if(mutex)
ENTER_MUTEX(mutex);
if(cdesc->ssl_socket)
{
ctx = cdesc->ssl_ctx;
sock = cdesc->ssl_socket;
cdesc->ssl_socket = NULL;
cdesc->ssl_ctx = NULL;
if(mutex)
EXIT_MUTEX(mutex);
SSL_free(sock);
SSL_CTX_sess_set_remove_cb(ctx, NULL);
SSL_CTX_free(ctx);
// printf("FREED SSL CTX %p\n", cdesc->ssl_ctx);
// printf("FREED SSL SOCKET %p\n", cdesc->ssl_socket);
}
else
{
if(mutex)
EXIT_MUTEX(mutex);
}
return;
}
//
// Initialize SSL operation a startup time.
//
ROUTINE void Ssl_Init()
{
//
// Initialize the basic SSL data.
//
ERR_load_crypto_strings();
SSL_library_init();
OpenSSL_add_all_algorithms();
OPENSSL_config(NULL);
SSL_load_error_strings();
CRYPTO_malloc_debug_init();
CRYPTO_set_mem_debug_options(V_CRYPTO_MDEBUG_ALL);
CRYPTO_mem_ctrl(CRYPTO_MEM_CHECK_ON);
return;
}
#ifdef USE_CRYPTO_LOCK
//
// A callback function used to set the ID of the executing task.
//
ROUTINE PRIVATE void Ssl_TaskID(CRYPTO_THREADID *id)
{
ULONG tid = uv_thread_self();
CRYPTO_THREADID_set_numeric(id, tid);
return;
}
//
// A callback fuction for aquiring and releasing a Mutex.
//
ROUTINE PRIVATE void Ssl_MainCryptoLock(int mode, int type, const char *file, int line)
{
if(mode & CRYPTO_LOCK)
ENTER_MUTEX(&Main_Ssl_Mutex_List[type]);
else
EXIT_MUTEX(&Main_Ssl_Mutex_List[type]);
return;
}
//
// FUNCTION - Ssl_InitLock(): Intialize the cryptograpy software locking mechanism.
//
// ARGUMENTS
// -----------
// task: The static ID of the main task/thread.
//
// RETURN VALUE
// --------------
// None
//
// NOTES
// -------
// This must be called within the main() thread/task at startup time.
// A set of Mutexes is used by the cryptography libraries and software to
// to prevent data corruption in a threaded environment. This mechanism
// is not used in recent versions of the OpenSSL library.
//
ROUTINE void Ssl_InitLock(int task)
{
int nlocks, k;
//
// Determine if the SSL library is compiled with thread support.
//
// #if defined(OPENSSL_THREADS)
// printf("THREADS OK\n");
// #else
// printf("THREADS FUCKED\n");
// #endif
//
//
// Just make sure the right task is calling this routine.
//
switch(task)
{
case MAIN_TASK:
// fprintf(stderr, "MAIN_TASK: SSL/Crypto Lock Init\n");
break;
default:
fatal_error(LOG_EMERG, "BUG: Can't Lock Non SSL Task: %d", task);
}
//
// Allocate and intialize the Mutexes.
//
nlocks = CRYPTO_num_locks();
Main_Ssl_Mutex_List = (uv_mutex_t *) VALLOC( (nlocks * sizeof(uv_mutex_t)) );
for(k = 0; k < nlocks; k++)
uv_mutex_init(&Main_Ssl_Mutex_List[k]);
//
// Configure the callback functions. Ssl_TaskID sets the thread/task ID of
// the executing task, and SSl_MainCryptoLock() invokes the appropriate Mutex
// as required.
//
CRYPTO_THREADID_set_callback((void (*) (CRYPTO_THREADID * )) Ssl_TaskID);
CRYPTO_set_locking_callback((void (*)(int, int, const char*, int)) Ssl_MainCryptoLock);
syslog(LOG_INFO, "Task %d: SSL/Crypto Lock Configured", task);
return;
}
#endif // USE_CRYPTO_LOCK
#endif // USE_SSL
