Sorry, i've sent the message without finishing it.
Hello to @ll.
I'm not sure if this is the correct list to post this question, but maybe
i'm dealing with a bug.
I have develop an event logging mechanism where application processes
connect to event loggers (using MPI_Lookup, MPI_open_port,
MPI_Comm_Connect, MPI_Comm_Accept, etc) that are part of another MPI
application.
Well, i have develop my own vprotocol where once a process receive a
message try to establish a connection with an event logger which is a
thread that belongs to another mpi application.
The event logger application consists in one mpi process per node with
multiple threads waiting for connections of MPI processes from the main
application.
I'm suspecting that there is a problem with the critical regions when
processes try to connect with the threads of the event logger.
I'm attaching two short examples that i have written in order to show the
problem. First, i launch the event-logger application:
mpirun -n 2 --machinefile machinefile2-th --report-uri URIFILE ./test-thread
Then i launch the example as this:
mpirun -n 16 --machinefile machine16 --ompi-server file:URIFILE
./thread_logger_connect
I have obtained this output:
*Published: radic_eventlog[1,6], ret=0*
*[clus2:16104] [[39125,1],1] ORTE_ERROR_LOG: Data unpack would read past
end of buffer in file dpm_orte.c at line 315*
*[clus2:16104] [[39125,1],1] ORTE_ERROR_LOG: Data unpack would read past
end of buffer in file dpm_orte.c at line 315*
*[clus2:16104] *** An error occurred in MPI_Comm_accept*
*[clus2:16104] *** on communicator MPI_COMM_SELF*
*[clus2:16104] *** MPI_ERR_UNKNOWN: unknown error*
*[clus2:16104] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)*
*--------------------------------------------------------------------------*
*mpirun has exited due to process rank 1 with PID 16104 on*
*node clus2 exiting improperly. There are two reasons this could occur:*
*
*
*1. this process did not call "init" before exiting, but others in*
*the job did. This can cause a job to hang indefinitely while it waits*
*for all processes to call "init". By rule, if one process calls "init",*
*then ALL processes must call "init" prior to termination.*
*
*
*2. this process called "init", but exited without calling "finalize".*
*By rule, all processes that call "init" MUST call "finalize" prior to*
*exiting or it will be considered an "abnormal termination"*
*
*
*This may have caused other processes in the application to be*
*terminated by signals sent by mpirun (as reported here).*
If i use mutex in order to serialized the access to MPI_Comm_Accept, the
behavior is ok, but shoudn't the MPI_comm_accept be thread safe?
Best regards.
Hugo Meyer
P.d.: This occurs with openmpi1.5.1 and also with also with an old version
of the trunk (1.7).
2013/5/6 Hugo Daniel Meyer <meyer.h...@gmail.com>
> Hello to @ll.
>
> I'm not sure if this is the correct list to post this question, but maybe
> i'm dealing with a bug.
>
> I have develop an event logging mechanism where application processes
> connect to event loggers (using MPI_Lookup, MPI_open_port,
> MPI_Comm_Connect, MPI_Comm_Accept, etc) that are part of another MPI
> application.
>
> Well, i have develop my own vprotocol where once a process receive a
> message try to establish a connection with an event logger which is a
> thread that belongs to another mpi application.
>
> The event logger application consists in one mpi process per node with
> multiple threads waiting for connections of MPI processes from the main
> application.
>
> I'm suspecting that there is a problem with the critical regions when
> processes try to connect with the threads of the event logger.
>
> I'm attaching two short examples that i have written in order to show the
> problem. First, i launch the event-logger application:
>
>
>
> If i use mutex in order to serialized the access to MPI_Comm_Accept,
>
>
>
>
#include "mpi.h"
#include <pthread.h>
#include <stdlib.h>
#include <stdio.h>
void open_port(void *args);
int r1 = 0, r2 = 0;
#define THREADS_NUMBER 8
struct thread_args
{
int thread_number;
int rank;
};
typedef struct thread_args thread_args;
pthread_mutex_t mymutex;
//tener en cuenta que los threads comunican serializados
int main(int argc, char **argv){
int rank, size, i, provided, claimed;
thread_args *argumentos = (thread_args*) malloc( THREADS_NUMBER*sizeof(thread_args));
pthread_t *threads;
threads=(pthread_t *)malloc(THREADS_NUMBER*sizeof(*threads));
//MPI_Init (&argc, &argv);
MPI_Init_thread( &argc, &argv, MPI_THREAD_FUNNELED, &provided );
printf("after init\n");
MPI_Comm_rank (MPI_COMM_WORLD, &rank);
MPI_Comm_size (MPI_COMM_WORLD, &size);
//pthread_mutex_init(&mymutex, NULL);
for (i=0; i<THREADS_NUMBER; i++) {
argumentos[i].rank=rank;
argumentos[i].thread_number=i;
pthread_create(&threads[i], NULL, (void *) open_port, (void *) &argumentos[i]);
}
for (i=0; i<THREADS_NUMBER; i++) {
pthread_join(threads[i], NULL);
}
free(threads);
free(argumentos);
MPI_Finalize();
}
void open_port(void *args){
thread_args *t_data = (struct thread_args *)args;
int ret;
char port_name[MPI_MAX_PORT_NAME];
char name[MPI_MAX_PORT_NAME];
MPI_Comm client;
MPI_Open_port(MPI_INFO_NULL, port_name);
printf("server [%d,%d] available at %s\n", t_data->rank, t_data->thread_number, port_name);
snprintf(name, MPI_MAX_PORT_NAME, "radic_eventlog[%d,%d]", t_data->rank, t_data->thread_number);
ret = MPI_Publish_name(name, MPI_INFO_NULL, port_name);
printf("Published: %s, ret=%d\n", name, ret);
//pthread_mutex_lock(&mymutex);
MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_SELF, &client);
//pthread_mutex_unlock(&mymutex);
MPI_Close_port(port_name);
printf("Accepted Conexion [%d,%d]\n",t_data->rank, t_data->thread_number);
sleep(5);
MPI_Comm_disconnect(&client);
}
#include "mpi.h"
#include <stdlib.h>
#include <stdio.h>
struct thread_args
{
int thread_number;
int rank;
};
typedef struct thread_args thread_args;
#define THREADS_NUMBER 8
int main(int argc, char **argv){
int rank, size, el_rank, el_thread;
char port[MPI_MAX_PORT_NAME];
MPI_Comm client;
char name[MPI_MAX_PORT_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
el_rank= rank/THREADS_NUMBER;
el_thread= rank % THREADS_NUMBER;
snprintf(name, MPI_MAX_PORT_NAME, "radic_eventlog[%d,%d]",el_rank,el_thread);
printf("Before lookup %s\n ", name);
int err = MPI_Lookup_name( name, MPI_INFO_NULL,port);
printf("Found - trying to connect\n ");
MPI_Comm_connect(port, MPI_INFO_NULL, 0, MPI_COMM_SELF, &client);
printf("Conexión aceptada\n");
sleep(5);
MPI_Comm_disconnect(&client);
MPI_Finalize();
}
