date:20200309

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

2020-03-09 Thread Chris Horn

(Re-sending my response to the list)

Yes, I believe that there are cases when problems on a remote node can be 
interpreted as local failures.


From: "nathan.dau...@noaa.gov" 
Date: Sunday, March 8, 2020 at 3:56 AM
To: Chris Horn , "lustre-discuss@lists.lustre.org" 

Cc: "nathan.dau...@noaa.gov" 
Subject: Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq
Resent-From: 
Resent-Date: Sunday, March 8, 2020 at 4:56 AM

Chris, all,

We are also seeing similar messages primarily on our servers, but from 
lnet_handle_local_failure() instead. I don't find any issues with the local 
o2ib interfere, yet, but there _may_ be a correlation with a client hang. Could 
this also be caused on a server by remote network problems or a client dropping 
out, in spite of the "local" name?

Thanks,
Nathan


On Mar 6, 2020 1:10 PM, Chris Horn  wrote:

> lneterror: 10164:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked())
> lpni  added to recovery queue.  Health = 900

The message means that the health value of a remote peer interface has been 
decremented, and as a result, the interface has been put into recovery mode. 
This mechanism is part of the LNet health feature.

Health values are decremented when a PUT or GET fails. Usually there are other 
messages in the log that can tell you more about the specific failure. 
Depending on your network type you should probably see messages from socklnd or 
o2iblnd. Network congestion could certainly lead to message timeouts, which 
would in turn result in interfaces being placed into recovery mode.

Chris Horn

On 3/6/20, 8:59 AM, "lustre-discuss on behalf of Michael Di Domenico" 
 
wrote:

along the aforementioned error i also see these at the same time

lustreerror: 9675:0:(obd_config.c:1428:class_modify_config())
<...>-clilov-<...>; failed to send uevent qos_threshold_rr=100

On Fri, Mar 6, 2020 at 9:39 AM Michael Di Domenico
 wrote:
>
> On Fri, Mar 6, 2020 at 9:36 AM Degremont, Aurelien  
wrote:
> >
> > Did you see any actual error on your system?
> >
> > Because there is a patch that is just decreasing the verbosity level of 
such messages, which looks like could be ignored.
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.whamcloud.com_browse_LU-2D13071&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=jp8DpDcylEQYlbd9-s3efysfDy2KdLvBrptsplqR1ks&e=
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__review.whamcloud.com_-23_c_37718_&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=8EUQ5wHRCuFFbd4PKxQCnTB_L9IgffvkzFw4_v6MEHg&e=
>
> thanks.  it's not entirely clear just yet.  i'm trying to track down a
> "slow jobs" issue.  i see these messages everywhere, so it might be a
> non issue or a sign of something more pressing.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=d36yZXUxMDJOjluQt2LUPivEkfLhScuCLIQT6Fl-Qhs&e=





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] unable to precreate -52/-116

2020-03-09 Thread Kumar, Amit

Hi Marco,

Thank you for the response on this issue.  

We have an HA setup, I tried to fail over MDT to the secondary pair and then 
fail it back. This did not help. 
I also tried restart of the MDS servers, that did not help.
I have rebooted OSS servers as well, that did not help
I also tried completely stopping MDS and unmounting MDS for a little while and 
that did not help either. 

This error ritually comes back right after MDT is mounted. Additionally I am 
not able to manually create any files on that particular OST. Any other 
thoughts.  

Thank you,
Amit

-Original Message-
From: Marco Grossi  
Sent: Monday, March 9, 2020 11:23 AM
To: Kumar, Amit 
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] unable to precreate -52/-116

Hi Amit,

We had a similar issue after a set_param of "max_create_count=0"

In our case re-mounting the MDT (not the OST) fixed the issue.

Hope it helps.

Regards,
Marco


On 3/3/20 8:25 PM, Kumar, Amit wrote:
> Dear Lustre,
> 
>  
> 
> Recently we had a degraded(Not failed) RAID and had to wait longer to 
> get compatible disk, as we had received incompatible one and it took 
> over a week to get the correct one back in place.
> 
>  
> 
> During this wait I ended up disabling the OST first and then noticed 
> continuous IO to the OST and thought of disabling object creation on 
> it as well. Everything looked normal after that and once the disk was 
> replaced I reenabled object creation and enabled OST. Since then I 
> started seeing these messages on OST
> 
> .(ofd_dev.c:1784:ofd_create_hdl()) scratch0-OST0029: unable to
> precreate: rc = -52
> 
> And following messages on MDS
> 
> .(osp_precreate.c:1282:osp_precreate_thread())
> scratch0-OST0029-osc-MDT: cannot precreate objects: rc = -116
> 
> .(osp_precreate.c:657:osp_precreate_send())
> scratch0-OST0029-osc-MDT: precreate fid 
> [0x10029:0x101b39a:0x0] < local used fid 
> [0x10029:0x101b39a:0x0]: rc = -116
> 
>  
> 
> These messages don't seem to stop. I am wondering what impact could 
> these errors have in long run? I have noticed I am not able to create 
> files on this particular OST using lfs setstripe, when I do so it gets 
> me an object on another OST by default. Just want to make sure this is 
> not causing any data loss for files the currently on them and new requests?
> 
> We plan to upgrade to 2.12 in the summer downtime and assuming that 
> has a fix based on LU-9442 & LU-11186.  Currently running servers on 
> lustre
> 10.4.1 over ZFS-0.7.9-1
> 
>  
> 
> Any help is greatly appreciated.
> 
>  
> 
> Thank you,
> Amit
> 
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 

--
Marco Grossi
ICHEC Systems Team


IF CLASSIFICATION START

IF CLASSIFICATION END
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

2020-03-09 Thread Chris Horn

Network failures cause an interface's health value to decrement. Recovery mode 
is the mechanism that raises the health value back up. Interfaces are ping'd on 
a regular interval by the "lnet_monitor_thread". Successful pings increase the 
health value of the interface (remote or local). 

When LNet is selecting the local and remote interfaces to use for a PUT or GET, 
it considers the health value of each interface. Healthier interfaces are 
preferred.

Chris Horn

On 3/9/20, 4:22 AM, "Degremont, Aurelien"  wrote:

What's the impact of being in recovery mode with LNET health?

Le 06/03/2020 21:12, « lustre-discuss au nom de Chris Horn » 
 a écrit :

> lneterror: 
10164:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked())
> lpni  added to recovery queue.  Health = 900

The message means that the health value of a remote peer interface has 
been decremented, and as a result, the interface has been put into recovery 
mode. This mechanism is part of the LNet health feature.

Health values are decremented when a PUT or GET fails. Usually there 
are other messages in the log that can tell you more about the specific 
failure. Depending on your network type you should probably see messages from 
socklnd or o2iblnd. Network congestion could certainly lead to message 
timeouts, which would in turn result in interfaces being placed into recovery 
mode.

Chris Horn

On 3/6/20, 8:59 AM, "lustre-discuss on behalf of Michael Di Domenico" 

wrote:

along the aforementioned error i also see these at the same time

lustreerror: 9675:0:(obd_config.c:1428:class_modify_config())
<...>-clilov-<...>; failed to send uevent qos_threshold_rr=100

On Fri, Mar 6, 2020 at 9:39 AM Michael Di Domenico
 wrote:
>
> On Fri, Mar 6, 2020 at 9:36 AM Degremont, Aurelien 
 wrote:
> >
> > Did you see any actual error on your system?
> >
> > Because there is a patch that is just decreasing the verbosity 
level of such messages, which looks like could be ignored.
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.whamcloud.com_browse_LU-2D13071&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=jp8DpDcylEQYlbd9-s3efysfDy2KdLvBrptsplqR1ks&e=
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__review.whamcloud.com_-23_c_37718_&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=8EUQ5wHRCuFFbd4PKxQCnTB_L9IgffvkzFw4_v6MEHg&e=
>
> thanks.  it's not entirely clear just yet.  i'm trying to track 
down a
> "slow jobs" issue.  i see these messages everywhere, so it might 
be a
> non issue or a sign of something more pressing.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=d36yZXUxMDJOjluQt2LUPivEkfLhScuCLIQT6Fl-Qhs&e=

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwIGaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=MWzLz3rQZoSqu_bMB83a0EdO1KMglAndLsxrBlOT9fA&s=Y-NtxxGn4LIKwsK_QtBwjw13E0CYycKLLS9PNuiGvms&e=

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] unable to precreate -52/-116

2020-03-09 Thread Marco Grossi

Hi Amit,

We had a similar issue after a set_param of "max_create_count=0"

In our case re-mounting the MDT (not the OST) fixed the issue.

Hope it helps.

Regards,
Marco


On 3/3/20 8:25 PM, Kumar, Amit wrote:
> Dear Lustre,
> 
>  
> 
> Recently we had a degraded(Not failed) RAID and had to wait longer to
> get compatible disk, as we had received incompatible one and it took
> over a week to get the correct one back in place.
> 
>  
> 
> During this wait I ended up disabling the OST first and then noticed
> continuous IO to the OST and thought of disabling object creation on it
> as well. Everything looked normal after that and once the disk was
> replaced I reenabled object creation and enabled OST. Since then I
> started seeing these messages on OST
> 
> …(ofd_dev.c:1784:ofd_create_hdl()) scratch0-OST0029: unable to
> precreate: rc = -52
> 
> And following messages on MDS
> 
> …(osp_precreate.c:1282:osp_precreate_thread())
> scratch0-OST0029-osc-MDT: cannot precreate objects: rc = -116
> 
> …(osp_precreate.c:657:osp_precreate_send())
> scratch0-OST0029-osc-MDT: precreate fid [0x10029:0x101b39a:0x0]
> < local used fid [0x10029:0x101b39a:0x0]: rc = -116
> 
>  
> 
> These messages don’t seem to stop. I am wondering what impact could
> these errors have in long run? I have noticed I am not able to create
> files on this particular OST using lfs setstripe, when I do so it gets
> me an object on another OST by default. Just want to make sure this is
> not causing any data loss for files the currently on them and new requests?
> 
> We plan to upgrade to 2.12 in the summer downtime and assuming that has
> a fix based on LU-9442 & LU-11186.  Currently running servers on lustre
> 10.4.1 over ZFS-0.7.9-1
> 
>  
> 
> Any help is greatly appreciated.
> 
>  
> 
> Thank you,
> Amit
> 
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 

-- 
Marco Grossi
ICHEC Systems Team
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] old Lustre 2.8.0 panic'ing continously

2020-03-09 Thread Andreas Dilger



On Mar 5, 2020, at 09:11, Mohr Jr, Richard Frank 
mailto:rm...@utk.edu>> wrote:



On Mar 5, 2020, at 2:48 AM, Torsten Harenberg 
mailto:harenb...@physik.uni-wuppertal.de>> 
wrote:

[QUOTA WARNING] Usage inconsistent for ID 2901:actual (757747712, 217)
!= expected (664182784, 215)

I assume you are running ldiskfs as the backend?  If so, have you tried 
regenerating the quota info for the OST?  I believe the command is “tune2fs -O 
^quota ” to clear quotas and then “tune2fs -O quota 
” to reenable/regenerate them.  I don’t know if that would 
work, but it might be worth a shot.

Just to clarify, the "tune2fs -O ^quota; tune2fs -O quota" trick is not really 
the best way to do this, even though this is widely circulated.

It would be better to run a full e2fsck, since that not only rebuilds the quota 
tables, but also ensures that the values going into the quota tables are 
correct.  Since the time taken by "tune2fs -O quota" is almost the same as 
running e2fsck, it is better to do it the right way.

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud






___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

2020-03-09 Thread Degremont, Aurelien

What's the impact of being in recovery mode with LNET health?


Le 06/03/2020 21:12, « lustre-discuss au nom de Chris Horn » 
 a écrit :

> lneterror: 10164:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked())
> lpni  added to recovery queue.  Health = 900

The message means that the health value of a remote peer interface has been 
decremented, and as a result, the interface has been put into recovery mode. 
This mechanism is part of the LNet health feature.

Health values are decremented when a PUT or GET fails. Usually there are 
other messages in the log that can tell you more about the specific failure. 
Depending on your network type you should probably see messages from socklnd or 
o2iblnd. Network congestion could certainly lead to message timeouts, which 
would in turn result in interfaces being placed into recovery mode.

Chris Horn

On 3/6/20, 8:59 AM, "lustre-discuss on behalf of Michael Di Domenico" 
 
wrote:

along the aforementioned error i also see these at the same time

lustreerror: 9675:0:(obd_config.c:1428:class_modify_config())
<...>-clilov-<...>; failed to send uevent qos_threshold_rr=100

On Fri, Mar 6, 2020 at 9:39 AM Michael Di Domenico
 wrote:
>
> On Fri, Mar 6, 2020 at 9:36 AM Degremont, Aurelien 
 wrote:
> >
> > Did you see any actual error on your system?
> >
> > Because there is a patch that is just decreasing the verbosity 
level of such messages, which looks like could be ignored.
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.whamcloud.com_browse_LU-2D13071&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=jp8DpDcylEQYlbd9-s3efysfDy2KdLvBrptsplqR1ks&e=
> > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__review.whamcloud.com_-23_c_37718_&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=8EUQ5wHRCuFFbd4PKxQCnTB_L9IgffvkzFw4_v6MEHg&e=
>
> thanks.  it's not entirely clear just yet.  i'm trying to track down a
> "slow jobs" issue.  i see these messages everywhere, so it might be a
> non issue or a sign of something more pressing.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=d36yZXUxMDJOjluQt2LUPivEkfLhScuCLIQT6Fl-Qhs&e=





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

Re: [lustre-discuss] unable to precreate -52/-116

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

Re: [lustre-discuss] unable to precreate -52/-116

Re: [lustre-discuss] old Lustre 2.8.0 panic'ing continously

Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq

6 matches

Site Navigation

Mail list logo

Footer information