Re: [Pvfs2-developers] parallel state machine code
OK, I think I fixed the small-io problem and the mkdir problem. That only leaves the mounting problem. I've never attempted to build the kernel interface or mount the file system (being the old goat that I am) so that might take a bit. I'll commit the changes I made and you can run them in the next nightly to see if anything new pops up. Walt Robert Latham wrote: On Tue, Aug 29, 2006 at 04:55:06PM -0400, Walter B. Ligon III wrote: So, I would appreciate some help running some tests on the branch, while I start documenting, and let me know when you think I should start merging it back with the trunk. Or I'm open to whatever other suggestions ... OK, walt, we're getting close. I committed a couple small fixes to get pvfs2-client-core building. Here's what's not working so well right now: - mounting pvfs2 fails with a timeout - many MPI-IO workloads pass, but the noncontig test triggered a segfault in small_io_cleanup, where it cleans up various fields in the sm_p structure. In particular, 'sm_p->msgarray = NULL' caused a core dump, and when I look at that core file in gdb, sm_p->msgarray_count is really high (135950228). Looks like maybe the sm_p wasn't properly allocated? I dunno, I'm just the messenger. - pvfs2-cp dies with a segfault when using a very small blocksize (-b 128). here's where gdb says the fault lies: --- #0 0x0806d3d8 in small_io_completion_fn (user_args=0x80f0da8, resp_p=0xbfffb42c, index=0) at sys-small-io.sm:242 242 fdata.server_nr = sm_p->u.io.datafile_index_array[index]; (gdb) p sm_p->u.io $8 = {io_type = 135162104, file_req = 0x2, file_req_offset = 0, buffer = 0x0, mem_req = 0x0, io_resp_p = 0x50, flowproto_type = 17, encoding = 135206232, datafile_index_array = 0x0, datafile_count = 0, msgpair_completion_count = 81, flow_completion_count = 0, write_ack_completion_count = 0, contexts = 0x80f13d4, context_count = 135205832, total_cancellations_remaining = 0, retry_count = 135206064, stored_error_code = 3396, total_size = 9, dfile_size_array = 0x0, small_io = 0} --- - test-zero-fill fails with a segfault in the same place as pvfs2-cp: --- #0 0x08065149 in small_io_completion_fn (user_args=0x80e9940, resp_p=0xbfffb86c, index=0) at sys-small-io.sm:317 317 sm_p->u.io.dfile_size_array[index] = resp_p->u.small_io.bstream_size; --- - pvfs2-mkdir (a test contributed by acxiom) fails with a seg fault: --- #0 0x080b134e in PINT_smcb_op (smcb=0x0) at /sandbox/robl/pvfs2-nightly/pvfs2-WALT3/src/common/misc/state-machine-fns.c:348 348 return smcb->op; --- So I think if you can take care of the small-io cases, that would be a good start, as it would knock out 3 of the 5 failures. Once WALT3 passes our nightlies, we can think about merging into HEAD. ==rob -- Dr. Walter B. Ligon III Associate Professor ECE Department Clemson University ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
[Pvfs2-developers] MPI-io tests
Hey guys, It turns out that the mpi-io programs that were stalling on Chiba was not because of any I/O related issues, but due to a simultaneous file create issue that was causing all sorts of weirdness. So, the workaround was to create the file in rank 0 and have all other ranks open the file after a barrier. So please ignore my previous message.. Sam and I will get to the bottom of the simultaneous create/delete bug after our paper deadline later this week, thanks, Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] MPI-io tests
On Tue, Sep 05, 2006 at 11:17:31AM -0500, Murali Vilayannur wrote: > Hey guys, > It turns out that the mpi-io programs that were stalling on Chiba was not > because of any I/O related issues, but due to a simultaneous file create > issue that was causing all sorts of weirdness. So, the workaround was to > create the file in rank 0 and have all other ranks open the file after a > barrier. So please ignore my previous message.. If you opened the file with MPI_COMM_WORLD, that should have already happened in ROMIO. If you need to do independent I/O for some reason, then that wouldn't have helped you.I haven't seen simultaneous indpendent file opens cause problems on jazz, but in those cases I'm not using the VFS either ==rob -- Rob Latham Mathematics and Computer Science DivisionA215 0178 EA2D B059 8CDF Argonne National Labs, IL USAB29D F333 664A 4280 315B ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
Re: [Pvfs2-developers] MPI-io tests
RobL, > If you opened the file with MPI_COMM_WORLD, that should have already > happened in ROMIO. If you need to do independent I/O for some reason, > then that wouldn't have helped you.I haven't seen simultaneous > indpendent file opens cause problems on jazz, but in those cases I'm > not using the VFS either We just the MPI part for starting up a lot of processes.. Sorry for the incorrect phrasing in my emails. We dont use MPI I/O. Used the posix interface directly. Simultaneous create problems with vfs is possibly due to the request scheduler on server not serializing crdirents of the same component name. I doubt you will even see that kind of pattern at the server with the mpi i/o interface. Thanks, Murali ___ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
[Pvfs2-developers] patches: acl test cleanups
These patches clean up the remaining issues from the tacl_xattr.sh test script (this is easy stuff- Murali did all of the hard work). tacl-xattr-homedir.patch: - This makes tacl-xattr.sh slightly more portable. Some Linux distributions have adduser utilities that do not create the home directory for you. This patch explicitly does a mkdir -p and chmod after adduser to make sure that required home directories exist. tacl-xattr-symlink.patch: - This is an important fix to the test script. It was using the -L argument to getfattr to traverse symbolic links when dumping the ACLs from the test directory. This leads to unpredictable results because there was no way to tell whether getfattr would traverse a real directory or its symbolic link first (it depends on the dirent order) and the other would always be left out. ACLs are not supported on symbolic links anyway, so it was just adding noise to the test script. xattr-symlink.patch: - This is the only change to PVFS2 itself. Like most Linux file systems, PVFS2 does not support xattrs on symbolic links (despite what is implied by man pages). This is due to the fact that symbolic links have 777 permissions by default that would allow anyone on the system to store xattrs in any symbolic link. This patch updates PVFS2 semantics slightly, however, to be more in line with how other file systems implement this. In particular listxattr() is now allowed (it just returns that there are zero entries), and setxattr() is implemented to return EPERM rather than EOPNOTSUPP. -Phil Index: pvfs2_src/test/automated/tacl_xattr.sh === --- pvfs2_src/test/automated/tacl_xattr.sh (revision 2425) +++ pvfs2_src/test/automated/tacl_xattr.sh (revision 2426) @@ -91,9 +91,17 @@ CUR_PATH=`pwd` /usr/sbin/adduser -d $CUR_PATH/tacluser1 tacluser1 +mkdir -p $CUR_PATH/tacluser1 +chown tacluser1 $CUR_PATH/tacluser1 /usr/sbin/adduser -d $CUR_PATH/tacluser2 tacluser2 +mkdir -p $CUR_PATH/tacluser2 +chown tacluser2 $CUR_PATH/tacluser2 /usr/sbin/adduser -d $CUR_PATH/tacluser3 tacluser3 +mkdir -p $CUR_PATH/tacluser3 +chown tacluser3 $CUR_PATH/tacluser3 /usr/sbin/adduser -d $CUR_PATH/tacluser4 tacluser4 +mkdir -p $CUR_PATH/tacluser4 +chown tacluser4 $CUR_PATH/tacluser4 if [ ! -e shared ] then Index: pvfs2_src/test/automated/tacl_xattr.sh === --- pvfs2_src/test/automated/tacl_xattr.sh (revision 2426) +++ pvfs2_src/test/automated/tacl_xattr.sh (revision 2427) @@ -702,10 +702,10 @@ # # -getfacl -RL shared > tmp1 +getfacl -RP shared > tmp1 setfacl -m u::--- -m g::--- -m o::--- shared/team1 setfacl --restore tmp1 -getfacl -RL shared > tmp2 +getfacl -RP shared > tmp2 if [ `diff tmp1 tmp2` ] then Index: pvfs2_src/src/kernel/linux-2.6/symlink.c === --- pvfs2_src/src/kernel/linux-2.6/symlink.c (revision 2441) +++ pvfs2_src/src/kernel/linux-2.6/symlink.c (revision 2442) @@ -52,11 +52,15 @@ follow_link : pvfs2_follow_link, setattr : pvfs2_setattr, revalidate : pvfs2_revalidate, +#ifdef HAVE_XATTR +listxattr: pvfs2_listxattr, +#endif #else .readlink = pvfs2_readlink, .follow_link = pvfs2_follow_link, .setattr = pvfs2_setattr, .getattr = pvfs2_getattr, +.listxattr = pvfs2_listxattr, #if defined(HAVE_GENERIC_GETXATTR) && defined(CONFIG_FS_POSIX_ACL) .permission = pvfs2_permission, #endif Index: pvfs2_src/src/kernel/linux-2.6/xattr-default.c === --- pvfs2_src/src/kernel/linux-2.6/xattr-default.c (revision 2442) +++ pvfs2_src/src/kernel/linux-2.6/xattr-default.c (revision 2443) @@ -26,6 +26,11 @@ if (strcmp(name, "") == 0) return -EINVAL; +if ( !S_ISREG(inode->i_mode) && + (!S_ISDIR(inode->i_mode) || inode->i_mode & S_ISVTX)) +{ + return -EPERM; +} gossip_debug(GOSSIP_XATTR_DEBUG, "pvfs2_setxattr_default %s\n", name); internal_flag = convert_to_internal_xattr_flags(flags); return pvfs2_inode_setxattr(inode, PVFS2_XATTR_NAME_DEFAULT_PREFIX, Index: pvfs2_src/src/kernel/linux-2.6/symlink.c === --- pvfs2_src/src/kernel/linux-2.6/symlink.c (revision 2442) +++ pvfs2_src/src/kernel/linux-2.6/symlink.c (revision 2443) @@ -53,6 +53,7 @@ setattr : pvfs2_setattr, revalidate : pvfs2_revalidate, #ifdef HAVE_XATTR +setxattr: pvfs2_setxattr, listxattr: pvfs2_listxattr, #endif #else @@ -62,6 +63,11 @@ .getattr = pvfs2_getattr, .listxattr = pvfs2_listxattr, #if defined(HAVE_GENERIC_GETXATTR) && defined(CONFIG_FS_POSIX_ACL) +.setxattr = generic_setxattr, +#else +.setxattr = pvfs2_setxattr, +#endif +#i
[Pvfs2-developers] patches: bug fixes
pread-pwrite.patch: --- This fixes a bug in a patch that I submitted earlier to provide a simple alternate AIO implementation. It defines _GNU_SOURCE in a limited area for dbpf so that we can get proper definitions of pread() and pwrite() on Linux. I tried using _XOPEN_SOURCE=500, but it will break any .c file that includes dbpf.h due to incompatibilities with Berkeley DB. zero-dfile.patch: --- This fixes a bug in the getattr handling on pvfs2-server if it happens to find an attribute structure with the dfile array zeroed out. In this case, it needs to set the attr flag appropriately to prevent the response encoder from segfaulting while processing the array in the response structure. This condition is very hard to trigger, but the server should be able to gracefully report the error rather than crashing. -Phil Index: pvfs2_src/src/io/trove/trove-dbpf/dbpf-bstream.c === --- pvfs2_src/src/io/trove/trove-dbpf/dbpf-bstream.c (revision 2444) +++ pvfs2_src/src/io/trove/trove-dbpf/dbpf-bstream.c (revision 2445) @@ -1424,8 +1424,8 @@ } /* prototypes for pread and pwrite; _XOPEN_SOURCE causes db.h problems */ -ssize_t pread(int fd, void *buf, size_t count, off_t offset); -ssize_t pwrite(int fd, const void *buf, size_t count, off_t offset); +ssize_t pread64(int fd, void *buf, size_t count, off64_t offset); +ssize_t pwrite64(int fd, const void *buf, size_t count, off64_t offset); static void* alt_lio_thread(void* foo) { struct alt_aio_item* tmp_item = (struct alt_aio_item*)foo; @@ -1433,14 +1433,14 @@ if(tmp_item->cb_p->aio_lio_opcode == LIO_READ) { -ret = pread(tmp_item->cb_p->aio_fildes, +ret = pread64(tmp_item->cb_p->aio_fildes, (void*)tmp_item->cb_p->aio_buf, tmp_item->cb_p->aio_nbytes, tmp_item->cb_p->aio_offset); } else if(tmp_item->cb_p->aio_lio_opcode == LIO_WRITE) { -ret = pwrite(tmp_item->cb_p->aio_fildes, +ret = pwrite64(tmp_item->cb_p->aio_fildes, (const void*)tmp_item->cb_p->aio_buf, tmp_item->cb_p->aio_nbytes, tmp_item->cb_p->aio_offset); Index: pvfs2_src/src/io/trove/trove-dbpf/dbpf-bstream.c === --- pvfs2_src/src/io/trove/trove-dbpf/dbpf-bstream.c (revision 2460) +++ pvfs2_src/src/io/trove/trove-dbpf/dbpf-bstream.c (revision 2461) @@ -1423,9 +1423,6 @@ return(0); } -/* prototypes for pread and pwrite; _XOPEN_SOURCE causes db.h problems */ -ssize_t pread64(int fd, void *buf, size_t count, off64_t offset); -ssize_t pwrite64(int fd, const void *buf, size_t count, off64_t offset); static void* alt_lio_thread(void* foo) { struct alt_aio_item* tmp_item = (struct alt_aio_item*)foo; @@ -1433,14 +1430,14 @@ if(tmp_item->cb_p->aio_lio_opcode == LIO_READ) { -ret = pread64(tmp_item->cb_p->aio_fildes, +ret = pread(tmp_item->cb_p->aio_fildes, (void*)tmp_item->cb_p->aio_buf, tmp_item->cb_p->aio_nbytes, tmp_item->cb_p->aio_offset); } else if(tmp_item->cb_p->aio_lio_opcode == LIO_WRITE) { -ret = pwrite64(tmp_item->cb_p->aio_fildes, +ret = pwrite(tmp_item->cb_p->aio_fildes, (const void*)tmp_item->cb_p->aio_buf, tmp_item->cb_p->aio_nbytes, tmp_item->cb_p->aio_offset); Index: pvfs2_src/src/io/trove/trove-dbpf/module.mk.in === --- pvfs2_src/src/io/trove/trove-dbpf/module.mk.in (revision 2460) +++ pvfs2_src/src/io/trove/trove-dbpf/module.mk.in (revision 2461) @@ -16,5 +16,7 @@ $(DIR)/dbpf-keyval-pcache.c \ $(DIR)/dbpf-sync.c -# grab trove-ledger.h from handle-mgmt. -MODCFLAGS_$(DIR) = -I$(srcdir)/src/io/trove/trove-handle-mgmt +# Grab trove-ledger.h from handle-mgmt. Also make _GNU_SOURCE definition +# required for access to pread/pwrite on Linux. _XOPEN_SOURCE seems to be +# incompatible with Berkeley DB. +MODCFLAGS_$(DIR) = -I$(srcdir)/src/io/trove/trove-handle-mgmt -D_GNU_SOURCE Index: pvfs2_src/src/server/get-attr.sm === --- pvfs2_src/src/server/get-attr.sm (revision 2366) +++ pvfs2_src/src/server/get-attr.sm (revision 2367) @@ -512,6 +512,10 @@ llu(s_op->u.getattr.handle), llu(s_op->u.getattr.handle), (int)s_op->u.getattr.fs_id); +/*If we hit an error the DIST & DFILES are no longer valid*/ +s_op->resp.u.getattr.attr.mask &= ~PVFS_ATTR_META_DIST; +s_op->resp.u.getattr.attr.mask &= ~PVFS_ATTR_META_DFILES; + js_p->error_code = -PVFS_EOVERFLOW; return 1; } @@ -624,6 +628,14 @@ */ js_p->error_code = 0; } +if(js_p->error_code < 0) +{ +if(s_op->val.buffer) +{ +free(s_op->val.buffer); +} +ret
[Pvfs2-developers] patch: binding to specific addresses
bind-specific.patch: This patch adds a new config file option (TCPBindSpecific) that if enabled tells the server to bind only to its specific IP address rather than using INADDR_ANY. This is particularly helpful in failover scenarios where you would like one physical machine to assume two ip addresses (and two servers) to run simultaneously when another server crashes. Without this patch you would need to select a different port on each server to prevent collisions. -Phil Index: pvfs2_src/src/server/pvfs2-server.c === --- pvfs2_src/src/server/pvfs2-server.c (revision 2419) +++ pvfs2_src/src/server/pvfs2-server.c (revision 2420) @@ -976,6 +976,7 @@ PVFS_fs_id orig_fsid; PVFS_ds_flags init_flags = 0; int port_num = 0; +int bmi_flags = BMI_INIT_SERVER; /* Initialize distributions */ ret = PINT_dist_initialize(0); @@ -999,8 +1000,15 @@ "Passing %s as BMI listen address.\n", server_config.host_id); +/* does the configuration dictate that we bind to a specific address? */ +if(server_config.tcp_bind_specific) +{ +bmi_flags |= BMI_TCP_BIND_SPECIFIC; +} + ret = BMI_initialize(server_config.bmi_modules, - server_config.host_id, BMI_INIT_SERVER); + server_config.host_id, + bmi_flags); if (ret < 0) { PVFS_perror_gossip("Error: BMI_initialize", ret); Index: pvfs2_src/src/io/bmi/bmi-types.h === --- pvfs2_src/src/io/bmi/bmi-types.h (revision 2419) +++ pvfs2_src/src/io/bmi/bmi-types.h (revision 2420) @@ -39,7 +39,8 @@ /** BMI method initialization flags */ enum { -BMI_INIT_SERVER = 1 /**< set up to listen for unexpected messages */ +BMI_INIT_SERVER = 1, /**< set up to listen for unexpected messages */ +BMI_TCP_BIND_SPECIFIC = 2 /**< bind to a specific IP address if INIT_SERVER */ }; enum bmi_op_type Index: pvfs2_src/src/io/bmi/bmi_tcp/sockio.c === --- pvfs2_src/src/io/bmi/bmi_tcp/sockio.c (revision 2419) +++ pvfs2_src/src/io/bmi/bmi_tcp/sockio.c (revision 2420) @@ -60,6 +60,27 @@ return (sockd); } +int BMI_sockio_bind_sock_specific(int sockd, + const char *name, + int service) +{ +struct sockaddr saddr; +int ret; + +if ((ret = BMI_sockio_init_sock(&saddr, name, service)) != 0) + return (ret); /* converted to PVFS error code below */ + + bind_sock_restart: +if (bind(sockd, &saddr, sizeof(saddr)) < 0) +{ + if (errno == EINTR) + goto bind_sock_restart; + return (-1); +} +return (sockd); +} + + int BMI_sockio_connect_sock(int sockd, const char *name, int service) Index: pvfs2_src/src/io/bmi/bmi_tcp/bmi-tcp.c === --- pvfs2_src/src/io/bmi/bmi_tcp/bmi-tcp.c (revision 2419) +++ pvfs2_src/src/io/bmi/bmi_tcp/bmi-tcp.c (revision 2420) @@ -1716,6 +1716,7 @@ int oldfl = 0; /* old socket flags */ struct tcp_addr *tcp_addr_data = NULL; int tmp_errno = bmi_tcp_errno_to_pvfs(-EINVAL); +int ret = 0; /* create a socket */ tcp_addr_data = tcp_method_params.listen_addr->method_data; @@ -1737,8 +1738,20 @@ BMI_sockio_set_sockopt(tcp_addr_data->socket, SO_REUSEADDR, 1); /* bind it to the appropriate port */ -if (BMI_sockio_bind_sock(tcp_addr_data->socket, tcp_addr_data->port) < 0) +if(tcp_method_params.method_flags & BMI_TCP_BIND_SPECIFIC) { +ret = BMI_sockio_bind_sock_specific(tcp_addr_data->socket, +tcp_addr_data->hostname, +tcp_addr_data->port); +} +else +{ +ret = BMI_sockio_bind_sock(tcp_addr_data->socket, +tcp_addr_data->port); +} + +if (ret < 0) +{ tmp_errno = errno; gossip_err("Error: BMI_sockio_bind_sock: %s\n", strerror(tmp_errno)); return (bmi_tcp_errno_to_pvfs(-tmp_errno)); Index: pvfs2_src/src/io/bmi/bmi_tcp/sockio.h === --- pvfs2_src/src/io/bmi/bmi_tcp/sockio.h (revision 2419) +++ pvfs2_src/src/io/bmi/bmi_tcp/sockio.h (revision 2420) @@ -32,6 +32,9 @@ int BMI_sockio_new_sock(void); int BMI_sockio_bind_sock(int, int); +int BMI_sockio_bind_sock_specific(int sockd, + const char *name, + int service); int BMI_sockio_connect_sock(int, const char *, int); Index: pvfs2_src/src/common/misc/server-config.c === --- pvfs2_src/src/common/misc/server-config.c (revision 2419) +++ pvfs2_src/src/common/misc/server-config.c (revision 2420) @@ -52,6 +52,7 @@ static DOTCONF_CB(get_unexp_req); static DOTCONF_CB(get_tcp_buffer_send); static DOTCONF_CB(get_tcp_buffer_receive); +static DO