Bob,
The attached patch should apply cleanly to the 1.5.1 release, and fix the problems you are seeing on ppc 64. You will have to rebuild after patching, and then restart the pvfs servers after rebuilding. I've committed the fixes to cvs and they'll get incorporated in the next release of pvfs. If you get a chance to try the patch, let me know.
Thanks, -sam
ledger-fix.patch
Description: Binary data
On Sep 25, 2006, at 10:15 PM, Robert Olson wrote:
One of the servers (on bio-ppc-head-1) was rebooted fairly recently, whilethe others were not. --bob On Mon, 25 Sep 2006, Sam Lang wrote:Hi Bob, I think I found the bug and have a fix for this, but I just want to make sure. Do you remember the last time you restarted the pvfs2 servers? I think this error is caused at server startup on ppc64 after at least 512 files are created. If your servers were re- started recently (with a populated file system), then I know what's wrong. Otherwise, if they were started once when the file system was created and have been running ever since, I've just found a bug elsewhere in our code, and I'll need to keep looking for yours. Let me know. Thanks, -sam On Sep 25, 2006, at 1:51 PM, Robert Olson wrote:This was built new for 1.5.1, and I've been using it on and off for a few weeks. --bob On Sep 25, 2006, at 1:37 PM, Sam Lang wrote:Hi Bob, Is this from a new storage space, or had things been working and then you started getting the permission denied errors? If its a storage space that's been around for a while, did you migrate from 1.3.x or 1.4.0? Based on the logfiles you sent, it doesn't look like a kernel module problem. We haven't seen those specific problems with 1.5.1, but we didn't do much testing of the 1.5.1 release on 64 bit platforms (esp. ppc). Would it be possible to get access to that system so that I can do some debugging? Thanks, -sam On Sep 25, 2006, at 12:34 PM, Robert Olson wrote:This is weird. I appear to getting permission denied on every other file to be copied from a local filesystem to the PVFS2 filesystem. This ring a bell with anyone? Thanks, --bob [EMAIL PROTECTED]:/scratch$ cp -r simtab.0917 /disks/pvfs2/olson cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ sims.10': Permission denied cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ sims.101': Permission denied cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ sims.103': Permission denied cp: cannot create regular file `/disks/pvfs2/olson/simtab.0917/ sims.105': Permission denied [EMAIL PROTECTED]:/scratch$ ls -l /disks/pvfs2/olson/simtab.0917/ total 88044 -rw-r--r-- 1 olson seed-users 37994316 Sep 25 17:26 sims.1 -rw-r--r-- 1 olson seed-users 30049 Sep 25 17:26 sims.100 -rw-r--r-- 1 olson seed-users 7226 Sep 25 17:26 sims.102 -rw-r--r-- 1 olson seed-users 1915643 Sep 25 17:26 sims.104 -rw-r--r-- 1 olson seed-users 50200576 Sep 25 17:26 sims.106 [EMAIL PROTECTED]:/scratch$ for i in simtab.0917/*; do echo $i; / vol/pvfs2/bin/pvfs2-cp -t $i /disks/pvfs2/olson/$i; done simtab.0917/sims.1 Wrote 37994316 bytes in 0.437222 seconds. 82.873696 MB/seconds simtab.0917/sims.10 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.10 Could not open /disks/pvfs2/olson/simtab.0917/sims.10 simtab.0917/sims.100 Wrote 30049 bytes in 0.001754 seconds. 16.339859 MB/seconds simtab.0917/sims.101 Wrote 18867 bytes in 0.035888 seconds. 0.501365 MB/seconds simtab.0917/sims.102 Wrote 7226 bytes in 0.001195 seconds. 5.766959 MB/seconds simtab.0917/sims.103 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.103 Could not open /disks/pvfs2/olson/simtab.0917/sims.103 simtab.0917/sims.104 Wrote 1915643 bytes in 0.022985 seconds. 79.482314 MB/seconds simtab.0917/sims.105 Wrote 113037406 bytes in 3.774026 seconds. 28.563891 MB/seconds simtab.0917/sims.106 Wrote 177660236 bytes in 5.191931 seconds. 32.633331 MB/seconds simtab.0917/sims.107 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.107 Could not open /disks/pvfs2/olson/simtab.0917/sims.107 simtab.0917/sims.108 Wrote 152382253 bytes in 5.079284 seconds. 28.610931 MB/seconds simtab.0917/sims.109 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.109 Could not open /disks/pvfs2/olson/simtab.0917/sims.109 simtab.0917/sims.11 Wrote 55995114 bytes in 1.893243 seconds. 28.206152 MB/seconds simtab.0917/sims.110 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.110 Could not open /disks/pvfs2/olson/simtab.0917/sims.110 simtab.0917/sims.111 Wrote 85578997 bytes in 2.864490 seconds. 28.491805 MB/seconds simtab.0917/sims.112 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.112 Could not open /disks/pvfs2/olson/simtab.0917/sims.112 simtab.0917/sims.113 Wrote 159349826 bytes in 5.469951 seconds. 27.782303 MB/seconds simtab.0917/sims.114 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.114 Could not open /disks/pvfs2/olson/simtab.0917/sims.114 simtab.0917/sims.115 Wrote 385509 bytes in 0.043475 seconds. 8.456602 MB/seconds simtab.0917/sims.116 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.116 Could not open /disks/pvfs2/olson/simtab.0917/sims.116 simtab.0917/sims.117 Wrote 125580543 bytes in 4.246360 seconds. 28.203670 MB/seconds simtab.0917/sims.118 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.118 Could not open /disks/pvfs2/olson/simtab.0917/sims.118 simtab.0917/sims.119 Wrote 74459114 bytes in 2.527315 seconds. 28.096912 MB/seconds simtab.0917/sims.12 open: Permission denied could not open /disks/pvfs2/olson/simtab.0917/sims.12 Could not open /disks/pvfs2/olson/simtab.0917/sims.12 [etc] pvfs-statfs is happy: [EMAIL PROTECTED]:~$ /vol/pvfs2/bin/pvfs2-statfs -m /mnt/pvfs2 aggregate statistics: --------------------------------------- fs_id: 1239905522 total number of servers (meta and I/O): 4 handles available (meta and I/O): 5365714538 handles total (meta and I/O): 5368709115 bytes available: 1218774892544 bytes total: 1495224942592 NOTE: The aggregate total and available statistics are calculated based on an algorithm that assumes data will be distributed evenly; thus the free space is equal to the smallest I/O server capacity multiplied by the number of I/O servers. If this number seems unusually small, then check the individual server statistics below to look for problematic servers. meta server statistics: --------------------------------------- server: tcp://bio-ppc-head-4:3334 RAM bytes total : 4107878400 RAM bytes free : 13307904 uptime (seconds) : 4045972 load averages : 4704 9088 5664 handles available: 2145227856 handles total : 2147483646 bytes available : 310944141312 bytes total : 373806235648 mode: serving both metadata and I/O data I/O server statistics: --------------------------------------- server: tcp://bio-ppc-head-4:3334 RAM bytes total : 4107878400 RAM bytes free : 13307904 uptime (seconds) : 4045972 load averages : 4704 9088 5664 handles available: 2145227856 handles total : 2147483646 bytes available : 310944141312 bytes total : 373806235648 mode: serving both metadata and I/O data server: tcp://bio-ppc-head-1:3334 RAM bytes total : 4107878400 RAM bytes free : 2062213120 uptime (seconds) : 832980 load averages : 30624 12352 3872 handles available: 1073741272 handles total : 1073741823 bytes available : 307329638400 bytes total : 373806235648 mode: serving only I/O data server: tcp://bio-ppc-head-2:3334 RAM bytes total : 4107878400 RAM bytes free : 12697600 uptime (seconds) : 4045977 load averages : 10592 6720 1792 handles available: 1073372712 handles total : 1073741823 bytes available : 304693723136 bytes total : 373806235648 mode: serving only I/O data server: tcp://bio-ppc-head-3:3334 RAM bytes total : 3578347520 RAM bytes free : 12324864 uptime (seconds) : 4046045 load averages : 13056 9376 3136 handles available: 1073372698 handles total : 1073741823 bytes available : 309065900032 bytes total : 373806235648 mode: serving only I/O data _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
