hi-
I'm using OrangeFS 2.8.4 to help test out a large emulab-based cluster
that we are putting together ( http://newmexicoconsortium.org/probe ).
I am trying to bring up a 512 node PVFS cluster using pvfs2-genconfig
to generate the configuration file (non-interactively, from a script).
The problem I am having is that pvfs2-server starts up OK, but then dies
with an assert in getconfig_init() because of PVFS_REQ_LIMIT_CONFIG_FILE_BYTES:
> static PINT_sm_action getconfig_init(
> struct PINT_smcb *smcb, job_status_s *js_p)
> {
> struct PINT_server_op *s_op = PINT_sm_frame(smcb, PINT_FRAME_CURRENT);
> struct server_configuration_s *user_opts = get_server_config_struct();
> assert(user_opts);
>
> /* TODO: handle this properly later */
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
> assert(user_opts->fs_config_buflen <= PVFS_REQ_LIMIT_CONFIG_FILE_BYTES);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I looked at the configuration file generated by pvfs2-genconfig and it
is 68922 bytes. Looking at my startup log file I found:
Warning: Generated config file: /etc/pvfs2-fs.conf
has size: 68741, which is larger than the current PVFS request: 65536
Increase the value of PVFS_REQ_LIMIT_CONFIG_FILE_BYTES in
src/proto/pvfs2-req-proto.h
My questions are:
1. does anyone currently use OrangeFS with 512 or more servers?
If so, is the only solution to edit the source code and recompile
the world, or are there other things I should consider? for example,
are there ways to make smaller config files using something other than
pvfs2-genconfig?
2. pvfs2-req-proto.h says if you change PVFS_REQ_LIMIT_CONFIG_FILE_BYTES
you should also change $req_limit in pvfs2-genconfig as well.
is that all I need to change to run on 512 nodes, or are there other
things I need to change too, before recompiling?
3. if I increase the number of nodes to 1024 or larger, are there other
gotchas that I should be aware of with OrangeFS? What is the biggest
number of nodes you can put into an OrangeFS system?
chuck
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
