Re: [Lustre-discuss] clients gets EINTR from time to time

2011-03-04 Thread Francois Chassaing
Dear list,
still investigating on this issue, I am now struggling with debugging..
The issue arose once more yesterday, so I started to look at it deeper and 
decided that the trace debug should be written to disk using debug_daemon.
Alas, debugging with only the trace debug active spits more than 100 MB/s 
worth of log ! (yes these are busy clients)...
I've tried several strategies like using debug_kernel from a cron job, or while 
watching my products error log, but even there dk would dump 70MB worth of data 
representing less that one second of debug log...
So chances for me to trace the signal seems looow.
Is there any debug flag less verbose but that may include the signal I'm 
looking for ?

Given John's answers could I maybe use /proc/sys/lustre/dump_on_timeout to dump 
the log only when timeout happens, but this will work only if my problem is 
matching what John can reproduce.

Please also note that I've looked around for abnormal threads_started numbers, 
it is everywhere at the same value than threads_min, except for one mdt entry 
which is at thread_min+1... 


Regards

weboramalineFrançois Chassaing Directeur Technique - CTO 

- Mail Original -
De: John Hammond jhamm...@tacc.utexas.edu
À: Andreas Dilger adil...@whamcloud.com
Cc: lustre-discuss@lists.lustre.org
Envoyé: Vendredi 25 Février 2011 21h16:36 GMT +01:00 Amsterdam / Berlin / Berne 
/ Rome / Stockholm / Vienne
Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

On 02/25/2011 11:39 AM, Andreas Dilger wrote:
 On 2011-02-25, at 6:28, Brian J. Murrell br...@whamcloud.com wrote:
 On 11-02-25 06:18 AM, Francois  wrote:

 I continue to parse debug logs and keep them posted.

 I don't understand why you don't just fix your application to handle a
 perfectly valid and expected condition (that it's currently not
 handling) instead of wasting time trying to find the cause of the
 expected condition.  Even if you find it, it's likely not a bug and not
 something that can/will be fixed.  It's your application that needs to
 be fixed.
 
 In all fairness Brian, it isn't always possible to fix an application like 
 you suggest. It might be commercial (binary only), it might be complex code 
 using 3rd party libraries to do the IO that would lose support if modifed, 
 etc. 
 
 I think the first action to debug this is to run on the client with lctl 
 set_param debug=+trace or =~0 which will enable function entry/exit 
 tracing in Lustre. Then when the problem us hit run lctl dk /tmp/debug to 
 dump the Lustre debug log, and search for -4 (which is -EINTR) to see where 
 this error is first appearing. 
 
 At that point we can make a determination where the source of the error is, 
 and if it is Lustre's fault. I know at one time there was a related problem 
 in the l_wait_event() macro that was improperly masking signals, but I 
 thought it was fixed by 1.8.5. 

Setting aside the moral question of which calls should be interruptible,
I think that the handling of the LUSTRE_FATAL_SIGS (defined in
lustre_lib.h to be SIGKILL, SIGINT, SIGTERM, SIGQUIT, SIGALRM) is
slightly broken.  Under certain situations, Lustre will return -EINTR
although no signals were delivered.  That's probably not the end of the
world for most applications, but OTOH I don't think anybody assumes that
-EINTR will be delivered spuriously.

Consider the following sequence:

1) Process P has a Lustre file F open.

2) P has SIGALRM pending (but blocked).

3) P starts to writing to F and ends up sleeping in (something like):

  sys_write()
   ...
ll_extent_lock()
 ...
  osc_enqueue()
   ...
ptlrpc_queue_wait().

4) The OST does not respond to the request before the deadline, so
l_wait_event() replaces the signal mask of P with the LUSTRE_FATAL_SIGS,
notices that SIGALRM is now deliverable, restores the signal mask of P,
and ptlrpc_queue_wait() returns -EINTR.

5) P is exiting from sys_write(), SIGALRM is blocked (but still pending)
so it doesn't get delivered.

6) P spuriously returns -EINTR from sys_write().

I can reproduce this on 1.8.5/RHEL 5.5.  If the goal is to emulate NFS's
interruptibility during congestion then returning -ERESTARTSYS would be
more appropriate.  Also, it might be worthwhile to make this extra
interruptibility a mount flag, as NFS does.

Best,

John

-- 
John L. Hammond, Ph.D.
TACC, The University of Texas at Austin
jhamm...@tacc.utexas.edu
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-25 Thread Francois Chassaing
Thanks, but anyway, logs on the MDS/MGS does not show evicted client of any 
kind.
Also, the log output by lctl debug_kernel on clients does not show much, I can 
only see in there the last administrative actions I've taken (such as setting 
striping policy on a directory, creating a new server pool, ...) and four 
unrelated (because not happening at my problem hours) Dropping PUT from

I continue to parse debug logs and keep them posted.

Thanks

weboramalineFrançois Chassaing Directeur Technique - CTO 

- Mail Original -
De: Kevin Van Maren kevin.van.ma...@oracle.com
À: DEGREMONT Aurelien aurelien.degrem...@cea.fr
Cc: Francois Chassaing f...@weborama.com, lustre-discuss@lists.lustre.org
Envoyé: Jeudi 24 Février 2011 18h43:25 GMT +01:00 Amsterdam / Berlin / Berne / 
Rome / Stockholm / Vienne
Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

No, in case of an eviction or IO errors, EIO is returned to the 
application, not EINTR.

Kevin


DEGREMONT Aurelien wrote:
 Hello

  From my understanding, Lustre can return EINTR for some I/O error cases.
 I think that when a client gets evicted in the middle of one of its RPC, 
 it can returns EINTR to the caller.
 Is this can explain your issue?

 Can your verify your clients where not evicted at the same time?

 Aurélien

 Francois Chassaing a écrit :
   
 OK, thanks it makes it more clear.
 I indeed messed up my mind (and words) between signals and error return 
 codes.
 I did understood that the write()/pwrite() system was returning the EINTR 
 error code because it received a signal, but I supposed that the signal was 
 sent because of an error condition somewhere in the FS. 
 This is where I now think I'm wrong. 
  
 As for your questions :
 - I have to mention that I always had had this issue, and this is why I've 
 upgraded from 1.8.4 to 1.8.5, hoping this would solve it.
 - I will try to have that SA_RESTART flag set in the app... if I can find 
 where the signal handler is set.
 - How can I see that lustre is returning EINTR for any other reason ? As I 
 said no logs shows nothing neither on MDS or OSSs, but I didn't go through 
 examining lctl debug_kernel yet... which I'm going to do right away...

 my last question is : how can I tell which signal I am receiving ? because 
 my app doesn't say, it just dumps outs the write/pwrite error code. 
 And if there is no signal handler, then it should follow the standard 
 actions (as of man 7 signal). On the other hand, my app does not stop or 
 dump core, and is not ignored, so it has to be handled in the code. Correct 
 me if I'm wrong...

 At that point, you realize that I didn't write the app, nor am I a good 
 Linux guru ;-)

 Tnaks a lot.

 weborama lineFrançois Chassaing Directeur Technique - CTO 

 - Mail Original -
 De: Ken Hornstein k...@cmf.nrl.navy.mil
 À: Francois Chassaing f...@weborama.com
 Cc: lustre-discuss@lists.lustre.org
 Envoyé: Jeudi 24 Février 2011 15h54:24 GMT +01:00 Amsterdam / Berlin / Berne 
 / Rome / Stockholm / Vienne
 Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

   
 
 OK, the app is used to deal with standard disks, that is why it is not
 handling the EINTR signal propoerly.
 
   
 I think you're misunderstanding what a signal is in the Unix sense.

 EINTR isn't a signal; it's a return code from the write() system call
 that says, Hey, you got a signal in the middle of this write() call
 and it didn't complete.  It doesn't mean that there was an error
 writing the file; if that was happening, you'd get a (presumably
 different) error code.  Signals can be sent by the operating system,
 but those signals are things like SIGSEGV, which basically means, you're
 program screwed up.  Programs can also send signals to each other,
 with kill(2) and the like.

 Now, NORMALLY systems calls like write() are interrupted by signals
 when you're writing to slow devices, like network sockets.  According
 to the signal(7) man page, disks are not normally considered slow
 devices, so I can understand the application not being used to handling
 this.  And you know, now that I think about it I'm not even sure that
 network filesystems SHOULD allow I/O system calls to be interrupted by
 signals ... I'd have to think more about it.

 I suspect what happened is that something changed between 1.8.5 and the
 previous version of Lustre that you were using that allowed some operations
 to be interruptable by signals.  Some things to try:

 - Check to see if you are, in fact, receiving a signal in your application
   and Lustre isn't returning EINTR for some other reason.
 - If you are receiving a signal, when you set the signal handler for it
   you could use the SA_RESTART flag to restart the interrupted I/O; I think
   that would make everything work like it did before.

 --Ken
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman

Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-25 Thread Brian J. Murrell
On 11-02-25 06:18 AM, Francois Chassaing wrote:
 Thanks, but anyway, logs on the MDS/MGS does not show evicted client of any 
 kind.
 Also, the log output by lctl debug_kernel on clients does not show much, I 
 can only see in there the last administrative actions I've taken (such as 
 setting striping policy on a directory, creating a new server pool, ...) and 
 four unrelated (because not happening at my problem hours) Dropping PUT from
 
 I continue to parse debug logs and keep them posted.

I don't understand why you don't just fix your application to handle a
perfectly valid and expected condition (that it's currently not
handling) instead of wasting time trying to find the cause of the
expected condition.  Even if you find it, it's likely not a bug and not
something that can/will be fixed.  It's your application that needs to
be fixed.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.



signature.asc
Description: OpenPGP digital signature
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-25 Thread Francois Chassaing
Maybe, once you'll explain to me why it is a perfectly expected condition. 
Because this I still don't get.
How can it be expected that a file cannot be written because of an interrupted 
system call, when all conditions are apparently met to write successfully to 
this file : a single client writing to a single file, no locking or concurrent 
access from other clients, no client eviction shown, no hardware failure, no 
nothing.
Please remember that this application traditionnaly deals with standard disks 
which DO NOT get EINTR except on error conditions...

Regards

weboramalineFrançois Chassaing Directeur Technique - CTO 

- Mail Original -
De: Brian J. Murrell br...@whamcloud.com
À: lustre-discuss@lists.lustre.org
Envoyé: Vendredi 25 Février 2011 14h28:02 GMT +01:00 Amsterdam / Berlin / Berne 
/ Rome / Stockholm / Vienne
Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

On 11-02-25 06:18 AM, Francois Chassaing wrote:
 Thanks, but anyway, logs on the MDS/MGS does not show evicted client of any 
 kind.
 Also, the log output by lctl debug_kernel on clients does not show much, I 
 can only see in there the last administrative actions I've taken (such as 
 setting striping policy on a directory, creating a new server pool, ...) and 
 four unrelated (because not happening at my problem hours) Dropping PUT from
 
 I continue to parse debug logs and keep them posted.

I don't understand why you don't just fix your application to handle a
perfectly valid and expected condition (that it's currently not
handling) instead of wasting time trying to find the cause of the
expected condition.  Even if you find it, it's likely not a bug and not
something that can/will be fixed.  It's your application that needs to
be fixed.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-25 Thread Ken Hornstein
I don't understand why you don't just fix your application to handle a
perfectly valid and expected condition (that it's currently not
handling) instead of wasting time trying to find the cause of the
expected condition.  Even if you find it, it's likely not a bug and not
something that can/will be fixed.  It's your application that needs to
be fixed.

To be fair ... normally disk I/O operations are not interruptable by
signals, so it's not an unreasonable behavior on the part of an
application.  I did check POSIX, and it doesn't say that behavior is
restricted only to network sockets, so yeah, it's TECHNICALLY allowable
behavior according to the standard (although the Linux manpage for
signal(7) says that it will not happen).  But honestly, I've seen
plenty of cases where applications handle this for network I/O; it's
normal, everyone knows it will happen there.  But for _disk_ I/O?
Never seen it done.  I'm not saying that there are no applications that
handle this case, but it's certainly very uncommon.  I freely admit
that network filesystems sort of mix the concepts of network socket
and disk I/O together, and what is the right behavior is unclear.
But calling this perfectly valid and expected is not quite accurate.
It would be interesting to see what other network filesystems do under
the same circumstances.

--Ken
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-25 Thread Cory Spitz
Hi.

I think it would help if you knew what the signal was.  Do you have that 
yet?

I have a report from a user that is is getting EINTR when a SIGALRM goes 
off on his write().  It isn't unexpected to get SIGALRM because he 
called the alarm, but he also has SA_RESTART set.  I can't remember 
whose responsibility it is to restart the call, syscall or whereever, 
but it seems that someone is dropping the ball because if EINTR is 
returned then SA_RESTART didn't seem to do the trick, right?

Thanks,
-Cory

On 2/25/2011 8:00 AM, Ken Hornstein wrote:
 I don't understand why you don't just fix your application to handle a
 perfectly valid and expected condition (that it's currently not
 handling) instead of wasting time trying to find the cause of the
 expected condition.  Even if you find it, it's likely not a bug and not
 something that can/will be fixed.  It's your application that needs to
 be fixed.

 To be fair ... normally disk I/O operations are not interruptable by
 signals, so it's not an unreasonable behavior on the part of an
 application.  I did check POSIX, and it doesn't say that behavior is
 restricted only to network sockets, so yeah, it's TECHNICALLY allowable
 behavior according to the standard (although the Linux manpage for
 signal(7) says that it will not happen).  But honestly, I've seen
 plenty of cases where applications handle this for network I/O; it's
 normal, everyone knows it will happen there.  But for _disk_ I/O?
 Never seen it done.  I'm not saying that there are no applications that
 handle this case, but it's certainly very uncommon.  I freely admit
 that network filesystems sort of mix the concepts of network socket
 and disk I/O together, and what is the right behavior is unclear.
 But calling this perfectly valid and expected is not quite accurate.
 It would be interesting to see what other network filesystems do under
 the same circumstances.

 --Ken
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-25 Thread Ken Hornstein
I have a report from a user that is is getting EINTR when a SIGALRM goes 
off on his write().  It isn't unexpected to get SIGALRM because he 
called the alarm, but he also has SA_RESTART set.  I can't remember 
whose responsibility it is to restart the call, syscall or whereever, 
but it seems that someone is dropping the ball because if EINTR is 
returned then SA_RESTART didn't seem to do the trick, right?

I would agree with you on that one; if you're setting SA_RESTART then
you shouldn't ever get EINTR.  It looks like what should be happening
is that if you get interrupted the system call should return
ERESTARTSYS and then after the signal handler is done the system call
should be re-run for you by the signal handling code.

I see that at least for some cases, Lustre will use ERESTARTSYS; just a
guess, but maybe somewhere Lustre is returning EINTR itself instead of
returning ERESTARTSYS?

--Ken
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-25 Thread John Hammond
On 02/25/2011 11:39 AM, Andreas Dilger wrote:
 On 2011-02-25, at 6:28, Brian J. Murrell br...@whamcloud.com wrote:
 On 11-02-25 06:18 AM, Francois  wrote:

 I continue to parse debug logs and keep them posted.

 I don't understand why you don't just fix your application to handle a
 perfectly valid and expected condition (that it's currently not
 handling) instead of wasting time trying to find the cause of the
 expected condition.  Even if you find it, it's likely not a bug and not
 something that can/will be fixed.  It's your application that needs to
 be fixed.
 
 In all fairness Brian, it isn't always possible to fix an application like 
 you suggest. It might be commercial (binary only), it might be complex code 
 using 3rd party libraries to do the IO that would lose support if modifed, 
 etc. 
 
 I think the first action to debug this is to run on the client with lctl 
 set_param debug=+trace or =~0 which will enable function entry/exit 
 tracing in Lustre. Then when the problem us hit run lctl dk /tmp/debug to 
 dump the Lustre debug log, and search for -4 (which is -EINTR) to see where 
 this error is first appearing. 
 
 At that point we can make a determination where the source of the error is, 
 and if it is Lustre's fault. I know at one time there was a related problem 
 in the l_wait_event() macro that was improperly masking signals, but I 
 thought it was fixed by 1.8.5. 

Setting aside the moral question of which calls should be interruptible,
I think that the handling of the LUSTRE_FATAL_SIGS (defined in
lustre_lib.h to be SIGKILL, SIGINT, SIGTERM, SIGQUIT, SIGALRM) is
slightly broken.  Under certain situations, Lustre will return -EINTR
although no signals were delivered.  That's probably not the end of the
world for most applications, but OTOH I don't think anybody assumes that
-EINTR will be delivered spuriously.

Consider the following sequence:

1) Process P has a Lustre file F open.

2) P has SIGALRM pending (but blocked).

3) P starts to writing to F and ends up sleeping in (something like):

  sys_write()
   ...
ll_extent_lock()
 ...
  osc_enqueue()
   ...
ptlrpc_queue_wait().

4) The OST does not respond to the request before the deadline, so
l_wait_event() replaces the signal mask of P with the LUSTRE_FATAL_SIGS,
notices that SIGALRM is now deliverable, restores the signal mask of P,
and ptlrpc_queue_wait() returns -EINTR.

5) P is exiting from sys_write(), SIGALRM is blocked (but still pending)
so it doesn't get delivered.

6) P spuriously returns -EINTR from sys_write().

I can reproduce this on 1.8.5/RHEL 5.5.  If the goal is to emulate NFS's
interruptibility during congestion then returning -ERESTARTSYS would be
more appropriate.  Also, it might be worthwhile to make this extra
interruptibility a mount flag, as NFS does.

Best,

John

-- 
John L. Hammond, Ph.D.
TACC, The University of Texas at Austin
jhamm...@tacc.utexas.edu
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


[Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Francois Chassaing
Dear list members, 

We are using Lustre 1.8.5 (upgraded from 1.8.4) running on 1 MGS, 3 OSS over 
DDR IB, and 2 patched clients mounted with the flock option. 
We are experiencing issues with an application that gets a EINTR when trying to 
write to a file. 
Those errors happens randomly on both clients, which makes it difficult to 
clearly spot the problem. So my app treats the error as if the file was full 
(which is the case when dealing with a normal disk) when it is not. 
I've tryed to change the IB switch, so it is most probably not coming from here 
(while it is a cheap switch). I've also tried to change the client mount 
options, changed the stripping policy from -1 to 1, but it did not change 
anything neither. 
And no log of any kind is helpful on MDS or OSSs. 
I would really appreciate pointers or suggestions to debug this issue. 

Thanks 

François CHASSAING 
Directeur Technique - CTO 
WEBORAMA 

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Brian J. Murrell
On 11-02-24 05:50 AM, Francois Chassaing wrote:
 Dear list members, 

Hi,

 We are experiencing issues with an application that gets a EINTR when trying 
 to write to a file. 

If I understand that errno properly, that is to be expected.

 Those errors happens randomly on both clients,

Well, not randomly.  It happens when a signal arrives.

 So my app treats the error as if the file was full

This is wrong.  Your app is broken and needs to be fixed.

 I've tryed to change the IB switch, so it is most probably not coming from 
 here (while it is a cheap switch). I've also tried to change the client 
 mount options, changed the stripping policy from -1 to 1, but it did not 
 change anything neither. 

None of this is going to resolve your problem.  Yours is a problem of
application programming defect, not a system fault.

 I would really appreciate pointers or suggestions to debug this issue. 

Maybe some understanding of how signals can affect system calls.  A
quick google found this for me:

http://www.gnu.org/s/libc/manual/html_node/Interrupted-Primitives.html#Interrupted-Primitives

Probably there is more detailed text out there to help you and your
application programmer to handle this application programming fault
better.  But alas, it is an application programming problem and not a
Lustre filesystem or equipment problem.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.



signature.asc
Description: OpenPGP digital signature
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Francois Chassaing
Well, as I understand your point and I do also understand that this signal is 
not a malfunction, 
my question was regarding to the intrinsic why (and when) does this signal is 
sent to the client.

Thnaks

line
weboramalineFrançois Chassaing Directeur Technique - CTO 
weborama.com - f...@weborama.com 
T : +33 (0)1 53 19 21 51 F : +33 (0)1 53 19 21 41 
Weborama - 15 rue Clavel 75019 Paris 

- Mail Original -
De: Brian J. Murrell br...@whamcloud.com
À: lustre-discuss@lists.lustre.org
Envoyé: Jeudi 24 Février 2011 13h17:33 GMT +01:00 Amsterdam / Berlin / Berne / 
Rome / Stockholm / Vienne
Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

On 11-02-24 05:50 AM, Francois Chassaing wrote:
 Dear list members, 

Hi,

 We are experiencing issues with an application that gets a EINTR when trying 
 to write to a file. 

If I understand that errno properly, that is to be expected.

 Those errors happens randomly on both clients,

Well, not randomly.  It happens when a signal arrives.

 So my app treats the error as if the file was full

This is wrong.  Your app is broken and needs to be fixed.

 I've tryed to change the IB switch, so it is most probably not coming from 
 here (while it is a cheap switch). I've also tried to change the client 
 mount options, changed the stripping policy from -1 to 1, but it did not 
 change anything neither. 

None of this is going to resolve your problem.  Yours is a problem of
application programming defect, not a system fault.

 I would really appreciate pointers or suggestions to debug this issue. 

Maybe some understanding of how signals can affect system calls.  A
quick google found this for me:

http://www.gnu.org/s/libc/manual/html_node/Interrupted-Primitives.html#Interrupted-Primitives

Probably there is more detailed text out there to help you and your
application programmer to handle this application programming fault
better.  But alas, it is an application programming problem and not a
Lustre filesystem or equipment problem.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Brian J. Murrell
On 11-02-24 08:16 AM, Francois Chassaing wrote:
 Well, as I understand your point and I do also understand that this signal is 
 not a malfunction, 

No, but not handling it properly is.  Interpreting an EINTR as the disk
must be full (i.e. a fatal error) is wrong.

 my question was regarding to the intrinsic why (and when) does this signal 
 is sent to the client.

That's completely up to your application.  It's the way your application
has been written that is determining the hows and whys of signals.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.



signature.asc
Description: OpenPGP digital signature
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Francois Chassaing
OK, the app is used to deal with standard disks, that is why it is not handling 
the EINTR signal propoerly. 
But I assumed that Lustre is 'just' a filesystem, so applications do not need 
to handle access to it any other way that the usual way.
Anyhow, the signal is from the OS not from the App... So it means that the OS 
signals the app that it has encoutered an error while trying to write to a 
file, and it is the source of that that I want to track down. 
Because this app error only arise every few days, it means that it is not a 
normal condition : something sowewhere in the FS causes it.
Interpreting it as a fatal error is certainly a mistake, but I still don't know 
why I'm getting this EINTR signal from the OS...

Regards


weboramalineFrançois Chassaing Directeur Technique - CTO 

- Mail Original -
De: Brian J. Murrell br...@whamcloud.com
À: lustre-discuss@lists.lustre.org
Envoyé: Jeudi 24 Février 2011 14h29:27 GMT +01:00 Amsterdam / Berlin / Berne / 
Rome / Stockholm / Vienne
Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

On 11-02-24 08:16 AM, Francois Chassaing wrote:
 Well, as I understand your point and I do also understand that this signal is 
 not a malfunction, 

No, but not handling it properly is.  Interpreting an EINTR as the disk
must be full (i.e. a fatal error) is wrong.

 my question was regarding to the intrinsic why (and when) does this signal 
 is sent to the client.

That's completely up to your application.  It's the way your application
has been written that is determining the hows and whys of signals.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Ken Hornstein
OK, the app is used to deal with standard disks, that is why it is not
handling the EINTR signal propoerly.

I think you're misunderstanding what a signal is in the Unix sense.

EINTR isn't a signal; it's a return code from the write() system call
that says, Hey, you got a signal in the middle of this write() call
and it didn't complete.  It doesn't mean that there was an error
writing the file; if that was happening, you'd get a (presumably
different) error code.  Signals can be sent by the operating system,
but those signals are things like SIGSEGV, which basically means, you're
program screwed up.  Programs can also send signals to each other,
with kill(2) and the like.

Now, NORMALLY systems calls like write() are interrupted by signals
when you're writing to slow devices, like network sockets.  According
to the signal(7) man page, disks are not normally considered slow
devices, so I can understand the application not being used to handling
this.  And you know, now that I think about it I'm not even sure that
network filesystems SHOULD allow I/O system calls to be interrupted by
signals ... I'd have to think more about it.

I suspect what happened is that something changed between 1.8.5 and the
previous version of Lustre that you were using that allowed some operations
to be interruptable by signals.  Some things to try:

- Check to see if you are, in fact, receiving a signal in your application
  and Lustre isn't returning EINTR for some other reason.
- If you are receiving a signal, when you set the signal handler for it
  you could use the SA_RESTART flag to restart the interrupted I/O; I think
  that would make everything work like it did before.

--Ken
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Francois Chassaing
OK, thanks it makes it more clear.
I indeed messed up my mind (and words) between signals and error return codes.
I did understood that the write()/pwrite() system was returning the EINTR error 
code because it received a signal, but I supposed that the signal was sent 
because of an error condition somewhere in the FS. 
This is where I now think I'm wrong. 
 
As for your questions :
- I have to mention that I always had had this issue, and this is why I've 
upgraded from 1.8.4 to 1.8.5, hoping this would solve it.
- I will try to have that SA_RESTART flag set in the app... if I can find where 
the signal handler is set.
- How can I see that lustre is returning EINTR for any other reason ? As I said 
no logs shows nothing neither on MDS or OSSs, but I didn't go through examining 
lctl debug_kernel yet... which I'm going to do right away...

my last question is : how can I tell which signal I am receiving ? because my 
app doesn't say, it just dumps outs the write/pwrite error code. 
And if there is no signal handler, then it should follow the standard actions 
(as of man 7 signal). On the other hand, my app does not stop or dump core, and 
is not ignored, so it has to be handled in the code. Correct me if I'm wrong...

At that point, you realize that I didn't write the app, nor am I a good Linux 
guru ;-)

Tnaks a lot.

weboramalineFrançois Chassaing Directeur Technique - CTO 

- Mail Original -
De: Ken Hornstein k...@cmf.nrl.navy.mil
À: Francois Chassaing f...@weborama.com
Cc: lustre-discuss@lists.lustre.org
Envoyé: Jeudi 24 Février 2011 15h54:24 GMT +01:00 Amsterdam / Berlin / Berne / 
Rome / Stockholm / Vienne
Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

OK, the app is used to deal with standard disks, that is why it is not
handling the EINTR signal propoerly.

I think you're misunderstanding what a signal is in the Unix sense.

EINTR isn't a signal; it's a return code from the write() system call
that says, Hey, you got a signal in the middle of this write() call
and it didn't complete.  It doesn't mean that there was an error
writing the file; if that was happening, you'd get a (presumably
different) error code.  Signals can be sent by the operating system,
but those signals are things like SIGSEGV, which basically means, you're
program screwed up.  Programs can also send signals to each other,
with kill(2) and the like.

Now, NORMALLY systems calls like write() are interrupted by signals
when you're writing to slow devices, like network sockets.  According
to the signal(7) man page, disks are not normally considered slow
devices, so I can understand the application not being used to handling
this.  And you know, now that I think about it I'm not even sure that
network filesystems SHOULD allow I/O system calls to be interrupted by
signals ... I'd have to think more about it.

I suspect what happened is that something changed between 1.8.5 and the
previous version of Lustre that you were using that allowed some operations
to be interruptable by signals.  Some things to try:

- Check to see if you are, in fact, receiving a signal in your application
  and Lustre isn't returning EINTR for some other reason.
- If you are receiving a signal, when you set the signal handler for it
  you could use the SA_RESTART flag to restart the interrupted I/O; I think
  that would make everything work like it did before.

--Ken
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Ken Hornstein
As for your questions :
- I have to mention that I always had had this issue, and this is why
I've upgraded from 1.8.4 to 1.8.5, hoping this would solve it.

Ah, okay, I misunderstood that; my apologies.

- I will try to have that SA_RESTART flag set in the app... if I can
find where the signal handler is set.

Searching for sigaction or signal should help there.

- How can I see that lustre is returning EINTR for any other reason ?
As I said no logs shows nothing neither on MDS or OSSs, but I didn't go
through examining lctl debug_kernel yet... which I'm going to do
right away...

Weeelll ... that was just a guess on my part.  I did a quick grep
though the Lustre sources and saw a few places where EINTR was
returned, but most of those seemed to deal with the case where I/O was
interrupted (those places happened fairly far down in the stack; it
wasn't clear to me that those errors would ever bubble back up to a
return code to a system call).  If _that_ is the issue, then tracking
that down will be a challenge.

my last question is : how can I tell which signal I am receiving ?
because my app doesn't say, it just dumps outs the write/pwrite error
code.

I think your easiest way is to use strace; something like strace -e signal
should do the right thing (that will only trace signals, not all system calls).

And if there is no signal handler, then it should follow the standard
actions (as of man 7 signal). On the other hand, my app does not stop
or dump core, and is not ignored, so it has to be handled in the code.
Correct me if I'm wrong...

That is my understanding as well; if you don't have a signal handler
installed, the default action should be taking place, and if the
default action is to ignore the signal that you shouldn't be getting
EINTR.  But hey, I've been wrong before :-)

--Ken
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread DEGREMONT Aurelien
Hello

 From my understanding, Lustre can return EINTR for some I/O error cases.
I think that when a client gets evicted in the middle of one of its RPC, 
it can returns EINTR to the caller.
Is this can explain your issue?

Can your verify your clients where not evicted at the same time?

Aurélien

Francois Chassaing a écrit :
 OK, thanks it makes it more clear.
 I indeed messed up my mind (and words) between signals and error return codes.
 I did understood that the write()/pwrite() system was returning the EINTR 
 error code because it received a signal, but I supposed that the signal was 
 sent because of an error condition somewhere in the FS. 
 This is where I now think I'm wrong. 
  
 As for your questions :
 - I have to mention that I always had had this issue, and this is why I've 
 upgraded from 1.8.4 to 1.8.5, hoping this would solve it.
 - I will try to have that SA_RESTART flag set in the app... if I can find 
 where the signal handler is set.
 - How can I see that lustre is returning EINTR for any other reason ? As I 
 said no logs shows nothing neither on MDS or OSSs, but I didn't go through 
 examining lctl debug_kernel yet... which I'm going to do right away...

 my last question is : how can I tell which signal I am receiving ? because my 
 app doesn't say, it just dumps outs the write/pwrite error code. 
 And if there is no signal handler, then it should follow the standard 
 actions (as of man 7 signal). On the other hand, my app does not stop or dump 
 core, and is not ignored, so it has to be handled in the code. Correct me if 
 I'm wrong...

 At that point, you realize that I didn't write the app, nor am I a good Linux 
 guru ;-)

 Tnaks a lot.

 weborama  lineFrançois Chassaing Directeur Technique - CTO 

 - Mail Original -
 De: Ken Hornstein k...@cmf.nrl.navy.mil
 À: Francois Chassaing f...@weborama.com
 Cc: lustre-discuss@lists.lustre.org
 Envoyé: Jeudi 24 Février 2011 15h54:24 GMT +01:00 Amsterdam / Berlin / Berne 
 / Rome / Stockholm / Vienne
 Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

   
 OK, the app is used to deal with standard disks, that is why it is not
 handling the EINTR signal propoerly.
 

 I think you're misunderstanding what a signal is in the Unix sense.

 EINTR isn't a signal; it's a return code from the write() system call
 that says, Hey, you got a signal in the middle of this write() call
 and it didn't complete.  It doesn't mean that there was an error
 writing the file; if that was happening, you'd get a (presumably
 different) error code.  Signals can be sent by the operating system,
 but those signals are things like SIGSEGV, which basically means, you're
 program screwed up.  Programs can also send signals to each other,
 with kill(2) and the like.

 Now, NORMALLY systems calls like write() are interrupted by signals
 when you're writing to slow devices, like network sockets.  According
 to the signal(7) man page, disks are not normally considered slow
 devices, so I can understand the application not being used to handling
 this.  And you know, now that I think about it I'm not even sure that
 network filesystems SHOULD allow I/O system calls to be interrupted by
 signals ... I'd have to think more about it.

 I suspect what happened is that something changed between 1.8.5 and the
 previous version of Lustre that you were using that allowed some operations
 to be interruptable by signals.  Some things to try:

 - Check to see if you are, in fact, receiving a signal in your application
   and Lustre isn't returning EINTR for some other reason.
 - If you are receiving a signal, when you set the signal handler for it
   you could use the SA_RESTART flag to restart the interrupted I/O; I think
   that would make everything work like it did before.

 --Ken
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Brian J. Murrell
On 11-02-24 11:57 AM, DEGREMONT Aurelien wrote:
 Hello

Hi,

  From my understanding, Lustre can return EINTR for some I/O error cases.

I think that should/would be an EIO.

 I think that when a client gets evicted in the middle of one of its RPC, 
 it can returns EINTR to the caller.

An evicted client should get an EIO on it's I/O calls, IIRC.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.



signature.asc
Description: OpenPGP digital signature
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss


Re: [Lustre-discuss] clients gets EINTR from time to time

2011-02-24 Thread Kevin Van Maren
No, in case of an eviction or IO errors, EIO is returned to the 
application, not EINTR.

Kevin


DEGREMONT Aurelien wrote:
 Hello

  From my understanding, Lustre can return EINTR for some I/O error cases.
 I think that when a client gets evicted in the middle of one of its RPC, 
 it can returns EINTR to the caller.
 Is this can explain your issue?

 Can your verify your clients where not evicted at the same time?

 Aurélien

 Francois Chassaing a écrit :
   
 OK, thanks it makes it more clear.
 I indeed messed up my mind (and words) between signals and error return 
 codes.
 I did understood that the write()/pwrite() system was returning the EINTR 
 error code because it received a signal, but I supposed that the signal was 
 sent because of an error condition somewhere in the FS. 
 This is where I now think I'm wrong. 
  
 As for your questions :
 - I have to mention that I always had had this issue, and this is why I've 
 upgraded from 1.8.4 to 1.8.5, hoping this would solve it.
 - I will try to have that SA_RESTART flag set in the app... if I can find 
 where the signal handler is set.
 - How can I see that lustre is returning EINTR for any other reason ? As I 
 said no logs shows nothing neither on MDS or OSSs, but I didn't go through 
 examining lctl debug_kernel yet... which I'm going to do right away...

 my last question is : how can I tell which signal I am receiving ? because 
 my app doesn't say, it just dumps outs the write/pwrite error code. 
 And if there is no signal handler, then it should follow the standard 
 actions (as of man 7 signal). On the other hand, my app does not stop or 
 dump core, and is not ignored, so it has to be handled in the code. Correct 
 me if I'm wrong...

 At that point, you realize that I didn't write the app, nor am I a good 
 Linux guru ;-)

 Tnaks a lot.

 weborama lineFrançois Chassaing Directeur Technique - CTO 

 - Mail Original -
 De: Ken Hornstein k...@cmf.nrl.navy.mil
 À: Francois Chassaing f...@weborama.com
 Cc: lustre-discuss@lists.lustre.org
 Envoyé: Jeudi 24 Février 2011 15h54:24 GMT +01:00 Amsterdam / Berlin / Berne 
 / Rome / Stockholm / Vienne
 Objet: Re: [Lustre-discuss] clients gets EINTR from time to time

   
 
 OK, the app is used to deal with standard disks, that is why it is not
 handling the EINTR signal propoerly.
 
   
 I think you're misunderstanding what a signal is in the Unix sense.

 EINTR isn't a signal; it's a return code from the write() system call
 that says, Hey, you got a signal in the middle of this write() call
 and it didn't complete.  It doesn't mean that there was an error
 writing the file; if that was happening, you'd get a (presumably
 different) error code.  Signals can be sent by the operating system,
 but those signals are things like SIGSEGV, which basically means, you're
 program screwed up.  Programs can also send signals to each other,
 with kill(2) and the like.

 Now, NORMALLY systems calls like write() are interrupted by signals
 when you're writing to slow devices, like network sockets.  According
 to the signal(7) man page, disks are not normally considered slow
 devices, so I can understand the application not being used to handling
 this.  And you know, now that I think about it I'm not even sure that
 network filesystems SHOULD allow I/O system calls to be interrupted by
 signals ... I'd have to think more about it.

 I suspect what happened is that something changed between 1.8.5 and the
 previous version of Lustre that you were using that allowed some operations
 to be interruptable by signals.  Some things to try:

 - Check to see if you are, in fact, receiving a signal in your application
   and Lustre isn't returning EINTR for some other reason.
 - If you are receiving a signal, when you set the signal handler for it
   you could use the SA_RESTART flag to restart the interrupted I/O; I think
   that would make everything work like it did before.

 --Ken
 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
   
 

 ___
 Lustre-discuss mailing list
 Lustre-discuss@lists.lustre.org
 http://lists.lustre.org/mailman/listinfo/lustre-discuss
   

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss