The twod_stripe problem that you describe sounds like the one that Kyle
Schochenmaier found and fixed in this mailing list thread:
http://www.beowulf-underground.org/pipermail/pvfs2-developers/2008-June/004069.html
You can check by running pvfs2-viewdist -f <filename> on the file that
you create with these parameters. If the num_groups shows up as 2 then
you are hitting a bug that prevents the distribution from honoring that
setting.
The attached patch should fix it if that's the case.
-Phil
Tony Kew wrote:
Dear Phil,
On a fist pass it looks like the patch fixes the basic_dist, but breaks
Murali's varstrip_dist xattrs patch. I'll try some more tests just to
make sure I didn't fat finger something ...
I'm also beginning to suspect that there is problem with xattrs
settings for twod_stripe. Testing on a single node (which I must
say makes no sense for a 2D stipe) & setting the strip to use one
group (since there is only one node) the server logfile shows errors
regarding using 2 groups with one server, so the xattr setting might
not be getting honored (or you cannot use this distribution without
at least two servers)
e.g.
$ mkdir twod_stripe
$ setfattr -n "user.pvfs2.dist_name" -v "twod_stripe" twod_stripe
$ setfattr -n "user.pvfs2.dist_params" \
> -v "strip_size:2048;num_groups:1;group_strip_factor:512" twod_stripe
$ cd twod_stripe/
$ dd if=/dev/urandom of=testfile bs=16k count=16
16+0 records in
16+0 records out
262144 bytes (262 kB) copied, 0.162147 seconds, 1.6 MB/s
$ ls -l
total 260
-rw-r--r-- 1 tonykew ccrstaff 262144 Jun 30 15:52 testfile
$
The fie is created (and is the correct size), but the server logfile
shows many errors:
[E 06/30 15:52] physical_to_logical_offset: Invalid
num_groups/server_ct options: gr:2 server:1
[E 06/30 15:52] next_mapped_offset: Invalid num_groups/server_ct
options: gr:2 server:1
[E 06/30 15:52] logical_to_physical_offset: Invalid
num_groups/server_ct options: gr:2 server:1
[E 06/30 15:52] next_mapped_offset: Invalid num_groups/server_ct
options: gr:2 server:1
[E 06/30 15:52] logical_to_physical_offset: Invalid
num_groups/server_ct options: gr:2 server:1
etc...
If I read this correctly, the number of groups "gr:2" is still 2 even
though I set it
to 1 with the fsetattr command
Many Thanks for your help,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
Hi Tony,
Thanks for the bug report. I have not looked at the varstrip issue
yet, but the attached patch (also available in cvs head) should at
least fix the basic_dist problem.
The problem was that basic_dist only works for files with 1 datafile,
but the logic that lets it override the number of datafiles was
happening before the directory hints were processed.
Someone correct me if I'm wrong, but I reorganized the code a little
chose the number of datafiles in this priority (after making sure the
final distribution for the file is set first, via hints or otherwise):
1) count specified in attr's passed into sys_create
- example: -n option to pvfs2-cp
2) directory hints
- example: "user.pvfs2.num_dfiles" extended attribute
3) mount options
- example: num_dfiles option in /etc/pvfs2tab
4) system default
- example: DefaultNumDfiles in server conf, or number of servers
The distribution itself gets a chance to override the value at the end
if (as in the case of basic_dist) it doesn't support an arbitrary
number of datafiles.
I'll look into the varstrip issue soon.
-Phil
Tony Kew wrote:
Dear PVFS users list,
Mounting a pvfs 7.2.1, creating directories and setting the distribution
type with setfattr for each directory, only simple_stipe and twod_stripe
would allow me to write a file in the given directory.
Adding the varstrip_dist xattrs patch from Murali Vilayannur attached to
this message thread:
http://osdir.com/ml/file-systems.pvfs2.users/2007-01/msg00050.html
fixes the variable strip distibution, but the basic distribution still
doesn't work for me.
The patch adds
#include "pvfs2-dist-basic.h
to src/client/sysint/sys-create.sm so I wonder if no one has tried
this...???
Perhaps there is a little more code needed to make basic_dist work with
xargs?
The patch above is against 2.6, but works (and is needed for
varstrip_dist
with xargs) with 7.2.1 & the latest cvs (best I can tell) - perhaps
this patch
could be added to both?
Let me know if there is useful debug info I can provide...
Thanks Much
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
---------------------
PatchSet 6919
Date: 2008/06/23 09:41:52
Author: pcarns
Branch: HEAD
Tag: small-file-branch-point3
Log:
patch from Kyle Schochenmaier to make twod_stripe distribution parameters
work correctly on 64 bit systems
Members:
src/io/description/dist-twod-stripe.c:1.4->1.5
Index: pvfs2-1/src/io/description/dist-twod-stripe.c
diff -u pvfs2-1/src/io/description/dist-twod-stripe.c:1.4 pvfs2-1/src/io/description/dist-twod-stripe.c:1.5
--- pvfs2-1/src/io/description/dist-twod-stripe.c:1.4 Sun Feb 3 09:47:21 2008
+++ pvfs2-1/src/io/description/dist-twod-stripe.c Mon Jun 23 09:41:52 2008
@@ -433,29 +433,29 @@
}
else if(strcmp(param_name, "num_groups")==0)
{
- if(*(uint32_t*)value <= 0)
+ if(*(int64_t*)value <= 0)
{
gossip_err("ERROR: num_groups param <= 0!\n");
}
else
{
gossip_debug(GOSSIP_DIST_DEBUG,
- "%s: num_groups: %d\n",
- __func__, *(uint32_t*)value);
- dparam->num_groups = *(uint32_t*)value;
+ "%s: num_groups: %lld\n",
+ __func__, lld(*(int64_t*)value));
+ dparam->num_groups = *(int64_t*)value;
}
}
else if(strcmp(param_name, "group_strip_factor")==0)
{
- if(*(uint32_t*)value <= 0)
+ if(*(int64_t*)value <= 0)
gossip_err("ERROR: group_strip_factor param <= 0!\n");
else
{
gossip_debug(GOSSIP_DIST_DEBUG,
- "%s: group_strip_factor: %d\n",
- __func__,*(uint32_t*)value);
+ "%s: group_strip_factor: %lld\n",
+ __func__,lld(*(int64_t*)value));
- dparam->group_strip_factor = *(uint32_t*)value;
+ dparam->group_strip_factor = *(int64_t*)value;
}
}
else
@@ -470,7 +470,7 @@
PVFS_twod_stripe_params *dparam =(PVFS_twod_stripe_params*)params;
encode_uint32_t(pptr,&dparam->num_groups);
encode_PVFS_size(pptr, &dparam->strip_size);
- encode_int32_t(pptr,&dparam->group_strip_factor);
+ encode_uint32_t(pptr,&dparam->group_strip_factor);
}
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
