Hi Tony,
Would you mind trying the attached patch? It increases the maximum
xattr value size to 8KB and also fixes a problem further down in the
code regarding how to handle xattrs that large. The latter part of the
patch should solve the problem that you ran into in which you could only
set a large xattr once.
thanks,
-Phil
Tony Kew wrote:
Dear Phil,
I verified that this is not an issue with the underlying filesystem:
c23n12$ mount | grep \ /scratch\
/dev/sda3 on /scratch type ext2 (rw)
c23n12$ sudo umount /scratch
c23n12$ sudo mount -o rw,user_xattr /dev/sda3 /scratch
c23n12$ mount | grep \ /scratch\
/dev/sda3 on /scratch type ext2 (rw,user_xattr)
c23n12$ cd /scratch
c23n12$ mkdir test
c23n12$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
c23n12$ getfattr test/
# file: test
user.pvfs2.dist_name
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
c23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
testc23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
c23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$
I tried changing PVFS_MAX_XATTR_VALUELEN in include/pvfs2-types.h from
256 to 512 (only, rather than 16K) - still has problems , setting the
"user.pvfs2.dist_params"
param with a attribute value >256 characters works the first time you
try it, but no value
( irrespective of how ling it is) can be set on the directory
subsequently...
$ mount | grep /var/tmp/PVFSv2/mnt
tcp://ramones:3334/pvfs2-fs-kew on /var/tmp/PVFSv2/mnt type pvfs2 (rw)
$ cd /var/tmp/PVFSv2/mnt
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr test
# file: test
user.pvfs2.dist_name
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
$ getfattr test
# file: test
user.pvfs2.dist_name
user.pvfs2.dist_params
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" testsetfattr:
test: Invalid argument
$
Thanks,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
Tony Kew wrote:
Dear Phil,
I can't test the setfattr command on a local filesystem as far as I
can tell:
$ cd /var/tmp
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
setfattr: test: Operation not supported
$
...or is there a test I can do on an ext3 filesystem dir?
I changed the value in include/pvfs2-types.h to 16K (should be plenty
big enough I guess...) & recompiled - now setfattr works the first
time,
but fails subsequently..
Note: testing on a single RHEL5 node, if that makes a difference
<cut>
It is not easy for me to test this code on 64 nodes.
I don't know if we can go further with this without me doing so?
Well, it looks like there are still problems just setting the
attributes, so there isn't really any need to try testing it on 64
nodes until that much is fixed.
Regarding how to test on ext3, you will need to follow Emmanuel
Florac's suggestion about the mount option to use for ext3. I just
want to make sure there isn't a system limit on the attribute size
before trying to bump it up in PVFS.
16K may be too large for the request messaging system. If you
experiment with that it might be safer to pick 8K or smaller.
-Phil
---------------------
PatchSet 7167
Date: 2008/10/22 12:02:47
Author: pcarns
Branch: HEAD
Tag: (none)
Log:
bug fix to keyval_write_list(); when it checks for the presence of existing
keys (to handle overwrite flags properly), it must take into account that
even when the key is present the value may be too large for the temporary
buffer.
Members:
src/io/trove/trove-dbpf/dbpf-keyval.c:1.89->1.90
Index: pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c
diff -u pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c:1.89 pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c:1.90
--- pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c:1.89 Mon Sep 8 11:42:46 2008
+++ pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c Wed Oct 22 12:02:47 2008
@@ -1255,7 +1255,10 @@
ret = op_p->coll_p->keyval_db->get(
op_p->coll_p->keyval_db, NULL, &key, &data, 0);
- if (ret != 0)
+ /* check for DB_BUFFER_SMALL in case the key is there but the data
+ * is simply too big for the temporary data buffer used
+ */
+ if (ret != 0 && ret != DB_BUFFER_SMALL)
{
if(ret == DB_NOTFOUND && ((op_p->flags & TROVE_NOOVERWRITE) ||
(!(op_p->flags & TROVE_ONLYOVERWRITE))))
---------------------
PatchSet 7168
Date: 2008/10/22 12:03:16
Author: pcarns
Branch: HEAD
Tag: (none)
Log:
increase xattr value limit from 256 bytes to 8 KB
Members:
include/pvfs2-types.h:1.151->1.152
Index: pvfs2-1/include/pvfs2-types.h
diff -u pvfs2-1/include/pvfs2-types.h:1.151 pvfs2-1/include/pvfs2-types.h:1.152
--- pvfs2-1/include/pvfs2-types.h:1.151 Mon Sep 8 11:42:37 2008
+++ pvfs2-1/include/pvfs2-types.h Wed Oct 22 12:03:16 2008
@@ -439,7 +439,7 @@
*/
#define PVFS_MAX_XATTR_NAMELEN 256 /* Not the same as XATTR_NAME_MAX defined
by <linux/xattr.h> */
-#define PVFS_MAX_XATTR_VALUELEN 256 /* Not the same as XATTR_SIZE_MAX defined
+#define PVFS_MAX_XATTR_VALUELEN 8192 /* Not the same as XATTR_SIZE_MAX defined
by <linux/xattr.h> */
#define PVFS_MAX_XATTR_LISTLEN 8 /* Not the same as XATTR_LIST_MAX
defined by <linux/xattr.h> */
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
