Chuck: I think that increasing the limits are the only way, right now, for the system to handle a large config file. I have a question out to Phil Carnes to see if this limit change effects BMI. As soon as I hear from him, I'll let you know. I believe the 65k limit was just so we could send the config file in one message to the client, which is needed when a client starts up. We have ways of sending larger messages but that mechanism is not currently being used for the config file. BTW: PVFS can handle as many servers as you want; it is just the size of the config file that is posing the problem.
We will definitely work on a better way to send a large config file, and as soon as I hear from Phil, I will let you know. Thanks, Becky On Wed, Sep 26, 2012 at 9:49 AM, Chuck Cranor <[email protected]> wrote: > hi- > > I'm using OrangeFS 2.8.4 to help test out a large emulab-based cluster > that we are putting together ( http://newmexicoconsortium.org/probe ). > > I am trying to bring up a 512 node PVFS cluster using pvfs2-genconfig > to generate the configuration file (non-interactively, from a script). > The problem I am having is that pvfs2-server starts up OK, but then dies > with an assert in getconfig_init() because of > PVFS_REQ_LIMIT_CONFIG_FILE_BYTES: > > > static PINT_sm_action getconfig_init( > > struct PINT_smcb *smcb, job_status_s *js_p) > > { > > struct PINT_server_op *s_op = PINT_sm_frame(smcb, PINT_FRAME_CURRENT); > > struct server_configuration_s *user_opts = get_server_config_struct(); > > assert(user_opts); > > > > /* TODO: handle this properly later */ > > vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv > > assert(user_opts->fs_config_buflen <= > PVFS_REQ_LIMIT_CONFIG_FILE_BYTES); > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > I looked at the configuration file generated by pvfs2-genconfig and it > is 68922 bytes. Looking at my startup log file I found: > > Warning: Generated config file: /etc/pvfs2-fs.conf > has size: 68741, which is larger than the current PVFS request: 65536 > Increase the value of PVFS_REQ_LIMIT_CONFIG_FILE_BYTES in > src/proto/pvfs2-req-proto.h > > > > My questions are: > > 1. does anyone currently use OrangeFS with 512 or more servers? > If so, is the only solution to edit the source code and recompile > the world, or are there other things I should consider? for example, > are there ways to make smaller config files using something other > than > pvfs2-genconfig? > > 2. pvfs2-req-proto.h says if you change PVFS_REQ_LIMIT_CONFIG_FILE_BYTES > you should also change $req_limit in pvfs2-genconfig as well. > is that all I need to change to run on 512 nodes, or are there other > things I need to change too, before recompiling? > > 3. if I increase the number of nodes to 1024 or larger, are there other > gotchas that I should be aware of with OrangeFS? What is the > biggest > number of nodes you can put into an OrangeFS system? > > > > chuck > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
