Hello,
I'm not sure what would cause that "Invalid argument" error.
Could you try the following steps:
- kill both servers
- modify your configuration files to set "EventLogging" to "none"
- delete your old log files (or move them to another directory)
- start the servers
You can then send us the complete contents of both log files and we can
go from there. The "all" level is a little hard to interpret because it
generates a lot of information even when servers are idle.
Also, when you went from one server to two, did you delete your old
storage space (/pvfs) and start over, or are you trying to keep that
data and add servers to it?
thanks!
-Phil
Asterios Katsifodimos wrote:
Hello all,
I have been trying to install PVFS 2.8.1 on Ubuntu server, Centos4 and
Scientific Linux 4. I compile it and can run it on a "single host"
configuration
without any problems.
However, when I add more nodes to the configuration(always using the
pvfs2-genconfig defaults ) I have the following problem:
*On the metadata node I get these messages:*
[E 04/02 20:16] batch_create request got: Invalid argument
[E 04/02 20:16] batch_create request got: Invalid argument
[E 04/02 20:16] batch_create request got: Invalid argument
[E 04/02 20:16] batch_create request got: Invalid argument
*In the IO nodes I get:*
[r...@wn140 ~]# tail -50 /tmp/pvfs2-server.log
[D 04/02 23:53] BMI_testcontext completing: 18446744072456767880
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:complete
(status: 1)
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1
[D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00 status_user_tag 1
msgarray_count 1
[D 04/02 23:53] msgpairarray: 1 operations remain
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:complete
(error code: -1073742006), (action: DEFERRED)
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:complete
(status: 0)
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1
[D 04/02 23:53] msgpairarray_complete: sm 0x88f8b00 status_user_tag 0
msgarray_count 1
[D 04/02 23:53] msgpairarray: all operations complete
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:complete
(error code: 190), (action: COMPLETE)
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:completion_fn
(status: 0)
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1
[D 04/02 23:53] (0x88f8b00) msgpairarray state: completion_fn
[E 04/02 23:53] Warning: msgpair failed to tcp://wn141:3334, will retry:
Connection refused
[D 04/02 23:53] *** msgpairarray_completion_fn: msgpair 0 failed, retry 1
[D 04/02 23:53] *** msgpairarray_completion_fn: msgpair retrying after
delay.
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:completion_fn
(error code: 191), (action: COMPLETE)
[D 04/02 23:53] [SM Entering]: (0x88f8b00) msgpairarray_sm:post_retry
(status: 0)
[D 04/02 23:53] [SM frame get]: (0x88f8b00) op-id: 37 index: 0 base-frm: 1
[D 04/02 23:53] msgpairarray_post_retry: sm 0x88f8b00, wait 2000 ms
[D 04/02 23:53] [SM Exiting]: (0x88f8b00) msgpairarray_sm:post_retry
(error code: 0), (action: DEFERRED)
[D 04/02 23:53] [SM Entering]: (0x89476c0) perf_update_sm:do_work
(status: 0)
[P 04/02 23:53] Start times (hr:min:sec): 23:53:11.330 23:53:10.310
23:53:09.287 23:53:08.268 23:53:07.245 23:53:06.225
[P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.026 00:00:01.020
00:00:01.023 00:00:01.019 00:00:01.023 00:00:01.020
[P 04/02 23:53]
-------------------------------------------------------------------------------------------------------------
[P 04/02 23:53] bytes read : 0
0 0 0 0 0
[P 04/02 23:53] bytes written : 0
0 0 0 0 0
[P 04/02 23:53] metadata reads : 0
0 0 0 0 0
[P 04/02 23:53] metadata writes : 0
0 0 0 0 0
[P 04/02 23:53] metadata dspace ops : 0
0 0 0 0 0
[P 04/02 23:53] metadata keyval ops : 1
1 1 1 1 1
[P 04/02 23:53] request scheduler : 0
0 0 0 0 0
[D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work (error
code: 0), (action: DEFERRED)
[D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work (status: 0)
[D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work (error
code: 0), (action: DEFERRED)
[D 04/02 23:53] [SM Entering]: (0x89476c0) perf_update_sm:do_work
(status: 0)
[P 04/02 23:53] Start times (hr:min:sec): 23:53:12.356 23:53:11.330
23:53:10.310 23:53:09.287 23:53:08.268 23:53:07.245
[P 04/02 23:53] Intervals (hr:min:sec) : 00:00:01.020 00:00:01.026
00:00:01.020 00:00:01.023 00:00:01.019 00:00:01.023
[P 04/02 23:53]
-------------------------------------------------------------------------------------------------------------
[P 04/02 23:53] bytes read : 0
0 0 0 0 0
[P 04/02 23:53] bytes written : 0
0 0 0 0 0
[P 04/02 23:53] metadata reads : 0
0 0 0 0 0
[P 04/02 23:53] metadata writes : 0
0 0 0 0 0
[P 04/02 23:53] metadata dspace ops : 0
0 0 0 0 0
[P 04/02 23:53] metadata keyval ops : 1
1 1 1 1 1
[P 04/02 23:53] request scheduler : 0
0 0 0 0 0
[D 04/02 23:53] [SM Exiting]: (0x89476c0) perf_update_sm:do_work (error
code: 0), (action: DEFERRED)
[D 04/02 23:53] [SM Entering]: (0x8948810) job_timer_sm:do_work (status: 0)
[D 04/02 23:53] [SM Exiting]: (0x8948810) job_timer_sm:do_work (error
code: 0), (action: DEFERRED)
The metadata node keeps asking for something that the IO nodes cannot give
the right way. So it complains. This makes the nodes and the metadata node
not to work.
I have installed those services many times. I have tested this using
berkeley
db 4.2 and 4.3 on Redhat systems(centos, scientific linnux) and on
Ubuntu server.
I have also tried the PVFS version 2.6.3 and I get the same problem.
*My config files look like:*
[r...@wn140 ~]# more /etc/pvfs2-fs.conf
<Defaults>
UnexpectedRequests 50
EventLogging all
EnableTracing no
LogStamp datetime
BMIModules bmi_tcp
FlowModules flowproto_multiqueue
PerfUpdateInterval 1000
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
ClientRetryLimit 5
ClientRetryDelayMilliSecs 2000
PrecreateBatchSize 512
PrecreateLowThreshold 256
StorageSpace /pvfs
LogFile /tmp/pvfs2-server.log
</Defaults>
<Aliases>
Alias wn140 tcp://wn140:3334
Alias wn141 tcp://wn141:3334
</Aliases>
<Filesystem>
Name pvfs2-fs
ID 320870944
RootHandle 1048576
FileStuffing yes
<MetaHandleRanges>
Range wn140 3-2305843009213693953
Range wn141 2305843009213693954-4611686018427387904
</MetaHandleRanges>
<DataHandleRanges>
Range wn140 4611686018427387905-6917529027641081855
Range wn141 6917529027641081856-9223372036854775806
</DataHandleRanges>
<StorageHints>
TroveSyncMeta yes
TroveSyncData no
TroveMethod alt-aio
</StorageHints>
</Filesystem>
My setup is made from two nodes that are both IO and Metadata nodes. I
have also tried
a 4 node setup with 2I/O - 2 MD nodes resulting in the same thing.
Any suggestions?
thank you in advance,
--
Asterios Katsifodimos
High Performance Computing systems Lab
Department of Computer Science, University of Cyprus
http://www.asteriosk.gr <http://www.asteriosk.gr/>
------------------------------------------------------------------------
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
