We also see these on some of the clients... Feb 18 15:32:47 r5b-s42 kernel: LustreError: 11-0: an error occurred while communicating with [EMAIL PROTECTED] The mds_close operation failed with -116 Feb 18 15:32:47 r5b-s42 kernel: LustreError: Skipped 3 previous similar messages Feb 18 15:32:47 r5b-s42 kernel: LustreError: 7828:0:(file.c: 97:ll_close_inode_openhandle()) inode 17243099 mdc close failed: rc = -116 Feb 18 15:32:47 r5b-s42 kernel: LustreError: 7828:0:(file.c: 97:ll_close_inode_openhandle()) Skipped 1 previous similar message
I'm assuming some of the threads succeed in opening the file. When one fails, it calls mpi_abort() at which point all those threads that successfully opened the file then try to close it. Apparently they can't close the file at that point either. I'm guessing of course, but it seems plausible. ct On Feb 18, 2008, at 4:57 PM, Oleg Drokin wrote: > Hello! > > On Feb 18, 2008, at 4:55 PM, Charles Taylor wrote: >> Feb 18 15:25:50 hpcmds kernel: LustreError: 7162:0:(mgs_handler.c: >> 515:mgs_handle()) lustre_mgs: operation 101 on unconnected MGS >> Feb 18 15:25:50 hpcmds kernel: LustreError: 7162:0:(mgs_handler.c: >> 515:mgs_handle()) Skipped 263 previous similar messages >> Feb 18 15:29:25 hpcmds kernel: LustreError: 6057:0:(ldlm_lib.c: >> 1442:target_send_reply_msg()) @@@ processing error (-107) >> [EMAIL PROTECTED] x1602651/t0 o101-><?>@<?>:-1 lens 232/0 ref 0 >> fl Interpret:/0/0 rc -107/0 >> Feb 18 15:29:25 hpcmds kernel: LustreError: 6057:0:(ldlm_lib.c: >> 1442:target_send_reply_msg()) Skipped 427 previous similar messages >> Feb 18 15:31:28 hpcmds kernel: LustreError: 7150:0:(mds_open.c: >> 1474:mds_close()) @@@ no handle for file close ino 43116025: >> cookie 0x1938027bf9d67349 [EMAIL PROTECTED] x10000789/t0 o35- >> >beb7df79-6127-c0ca-9d36-2a96817a77a9@:-1 lens 296/1736 ref 0 fl >> Interpret:/0/0 rc 0/0 >> Feb 18 15:31:28 hpcmds kernel: LustreError: 7150:0:(mds_open.c: >> 1474:mds_close()) Skipped 161 previous similar messages >> Feb 18 15:33:17 hpcmds kernel: LustreError: 0:0:(ldlm_lockd.c: >> 210:waiting_locks_callback()) ### lock callback timer expired: >> evicting client 2bdea9d4-43c3-a0b0-2822- >> [EMAIL PROTECTED] nid [EMAIL PROTECTED] ns: >> mds-ufhpc-MDT0000_UUID lock: ffff810053d3f100/0x688cfbc7df2ef487 >> lrc: 1/0,0 mode: CR/CR res: 21878337/3424633214 bits 0x3 rrc: 582 >> type: IBT flags: 4000030 remote: 0x95c1d2685c2c76d9 expref: 21 pid >> 6090 >> Feb 18 15:33:17 hpcmds kernel: LustreError: 0:0:(ldlm_lockd.c: >> 210:waiting_locks_callback()) Skipped 3 previous similar messages >> Feb 18 15:33:17 hpcmds kernel: LustreError: 6265:0:(ldlm_lockd.c: >> 962:ldlm_handle_enqueue()) ### lock on destroyed export >> ffff8101096ec000 ns: mds-ufhpc-MDT0000_UUID lock: >> ffff810225fe12c0/0x688cfbc7df2ef505 lrc: 2/0,0 mode: CR/CR res: >> 21878337/3424633214 bits 0x3 rrc: 579 type: IBT flags: 4000030 >> remote: 0x95c1d2685c2c76e0 expref: 6 pid 6265 >> Feb 18 15:33:17 hpcmds kernel: LustreError: 6265:0:(ldlm_lockd.c: >> 962:ldlm_handle_enqueue()) Skipped 3 previous similar messages >> Feb 18 15:33:17 hpcmds kernel: Lustre: 6061:0:(mds_reint.c: >> 127:mds_finish_transno()) commit transaction for disconnected >> client 2bdea9d4-43c3-a0b0-2822-c49ecfe6e044: rc 0 > > This looks like in the middle of eviction storm, and by this point > MDS and MGS anlready evicted tons of clients for unknown reasons > (should be in the log before those messages). > > Bye, > Oleg _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss