[Lustre-discuss] unmounting with open requests

[Hebenstreit, Michael]

Is there a way to force the Lustre kernel modules to cancel/terminate all 
outstanding requests and allow the file system to be unmounted and the kernel 
modules to be removed from the running kernel?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Software and Services Group/DCE
4100 Sara Road  Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] unmounting with open requests

[Hebenstreit, Michael]

"umount -f" will not work if you have open kernel calls. The problem is if 
something goes wrong, requests still handled even if the programs original 
initiating the kernel call have been killed. Under such instances I think the 
kernel modules need to first do some clean up work, before 
unmounting/lustre_rmmod will work

Michael

From: Lee, Brett
Sent: Wednesday, January 15, 2014 12:56 PM
To: Hebenstreit, Michael; lustre-discuss@lists.lustre.org
Subject: RE: unmounting with open requests

Michael,

Not that I know of.  But you might try a different order, unmounting any Lustre 
mounts with a "umount -f " followed by removing the Lustre modules with 
"lustre_rmmod" - will this work well enough for you?

--
Brett Lee
Sr. Systems Engineer
Intel High Performance Data Division

From: 
lustre-discuss-boun...@lists.lustre.org<mailto:lustre-discuss-boun...@lists.lustre.org>
 [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Hebenstreit, 
Michael
Sent: Wednesday, January 15, 2014 11:41 AM
To: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: [Lustre-discuss] unmounting with open requests

Is there a way to force the Lustre kernel modules to cancel/terminate all 
outstanding requests and allow the file system to be unmounted and the kernel 
modules to be removed from the running kernel?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Software and Services Group/DCE
4100 Sara Road  Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com<mailto:michael.hebenstr...@intel.com>

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] unmounting with open requests

[Hebenstreit, Michael]

Clients have not been evicted. 

I can't name any "typical" reason - the codes and error situations my customers 
create on this cluster are legion (after all it's a cluster used for 
development and benchmarking). But the result is even with all processes gone I 
have a 5% chance that I cannot unmount the FS or remove the modules because the 
kernel has still open requests to handle.

So my question if there is any chance to clean up such a state (short of 
rebooting)

Thanks
Michael

-Original Message-
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rm...@utk.edu] 
Sent: Thursday, January 16, 2014 7:22 AM
To: Hebenstreit, Michael
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] unmounting with open requests


On Jan 15, 2014, at 3:08 PM, "Hebenstreit, Michael" 
 wrote:

> "umount -f" will not work if you have open kernel calls. The problem 
> is if something goes wrong, requests still handled even if the 
> programs original initiating the kernel call have been killed. Under 
> such instances I think the kernel modules need to first do some clean 
> up work, before unmounting/lustre_rmmod will work

If you manually evicted the client from the lustre servers after running 
"umount -f", would that force the lustre client to drop any outstanding 
requests?  (I am not sure exactly what happens on the client side when it is 
evicted.)

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu


___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] module dependecies

[Hebenstreit, Michael]

Please do not ask why, but I need to be able to replace the IB stack (as in - 
all InfiniBand modules) at runtime with a Lustre FS mounted. Is there a 
possibility to tell lnet to completely switch to tcp, unload ko2iblnd.ko and 
next unload the IB stack, load the new IB stack, load a matching ko2iblnd.ko 
and then switch lnet back to preferring o2ib?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Software and Services Group/DCE
4100 Sara Road  Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[lustre-discuss] file distribution over OSTs

[Hebenstreit, Michael]

Can I find out how a file (size 130GB, stripesize 1MB) is distributed over OSTs?

thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Software and Services Group/DCE
4100 Sara Road  Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] file distribution over OSTs

[Hebenstreit, Michael]

That is not really very verbose – an example

[mhebenst@ehs141 ~]$ lfs getstripe /lfs/lfs10s/mhebenst/IOR.0001
/lfs/lfs10s/mhebenst/IOR.0001
lmm_stripe_count:   1
lmm_stripe_size:1048576
lmm_pattern:1
lmm_layout_gen: 0
lmm_stripe_offset:  8
obdidx   objid   objid   group
 8 4989642   0x4c22ca0


Does that mean the file is only present on OST8 of 11(there are 12 OSTs)? What 
if the stripesize 2 or higher? Will the distribution be exactly round robin 
over the listed OSTs?

Thanks
Michael


From: Colin Faber [mailto:cfa...@gmail.com] 
Sent: Tuesday, April 07, 2015 3:45 PM
To: Hebenstreit, Michael
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] file distribution over OSTs

"lfs getstripe" is the command you're looking for.

On Tue, Apr 7, 2015 at 3:43 PM, Hebenstreit, Michael 
 wrote:
Can I find out how a file (size 130GB, stripesize 1MB) is distributed over OSTs?

thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Software and Services Group/DCE
4100 Sara Road              Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] file distribution over OSTs

[Hebenstreit, Michael]

Thanks – and sorry – my comment on “verbose” was for the command, not about 
your info  ☺

Case closed, thanks all for answers
That was quick ☺

From: Colin Faber [mailto:cfa...@gmail.com]
Sent: Tuesday, April 07, 2015 3:54 PM
To: Hebenstreit, Michael
Subject: Re: [lustre-discuss] file distribution over OSTs

Hi,


On Tue, Apr 7, 2015 at 3:50 PM, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:
That is not really very verbose – an example

Heh, sorry I'll try and be in less of a hurry the next time around =)


[mhebenst@ehs141 ~]$ lfs getstripe /lfs/lfs10s/mhebenst/IOR.0001
/lfs/lfs10s/mhebenst/IOR.0001
lmm_stripe_count:   1
lmm_stripe_size:1048576
lmm_pattern:1
lmm_layout_gen: 0
lmm_stripe_offset:  8
obdidx   objid   objid   group
 8 4989642   0x4c22ca0


Does that mean the file is only present on OST8 of 11

Yes, as the stripe count is 1

(there are 12 OSTs)? What if the stripesize 2 or higher? Will the distribution 
be exactly round robin over the listed OSTs?

It would be striped across 2 OST's in as balanced a way as possible. Not 
exactly round robin, more balanced (or attempted balanced as much as possible) 
depending on OST usage.

-cf



Thanks
Michael


From: Colin Faber [mailto:cfa...@gmail.com<mailto:cfa...@gmail.com>]
Sent: Tuesday, April 07, 2015 3:45 PM
To: Hebenstreit, Michael
Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: Re: [lustre-discuss] file distribution over OSTs

"lfs getstripe" is the command you're looking for.

On Tue, Apr 7, 2015 at 3:43 PM, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:
Can I find out how a file (size 130GB, stripesize 1MB) is distributed over OSTs?

thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Software and Services Group/DCE
4100 Sara Road  Tel.:   +1 
505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com<mailto:michael.hebenstr...@intel.com>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Do I need Lustre?

[Hebenstreit, Michael]

You can do a simple test. Run a small sample of you application directly out of 
/dev/shm (the ram-disk). Then run it from the NFS file server. If you measure 
significant speedups your application is I/O sensitive and a Lustre configured 
with OPA or other InfiniBand solution will help.

From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf 
Of Thackeray, Neil L
Sent: Friday, April 27, 2018 11:08 AM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] Do I need Lustre?

I'm new to the cluster realm, so I'm hoping for some good advice. We are 
starting up a new cluster, and I've noticed that lustre seems to be used widely 
in datacenters. The thing is I'm not sure the scale of our cluster will need it.

We are planning a small cluster, starting with 6 -8 nodes with 2 GPUs per node. 
They will be used for Deep Learning, MRI data processing, and Matlab among 
other things. With the size of the cluster we figure that 10Gb networking will 
be sufficient. We aren't going to allow persistent storage on the cluster. 
Users will just upload and download data. I'm mostly concerned about I/O 
speeds. I don't know if NFS would be fast enough to handle the data.

We are hoping that the cluster will grow over time. We are already talking 
about buying more nodes next fiscal year.

Thanks.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] rhel 7.5

[Hebenstreit, Michael]

I have 2.11 already running on a 7.5 clone (client only)

-Original Message-
From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf 
Of Michael Di Domenico
Sent: Monday, April 30, 2018 11:49 AM
Cc: lustre-discuss 
Subject: Re: [lustre-discuss] rhel 7.5

On Mon, Apr 30, 2018 at 10:09 AM, Jeff Johnson  
wrote:
> RHEL 7.5 support comes in Lustre 2.10.4. Only path I can think of off 
> the top of my head is to git clone and build a 2.10.4 prerelease and 
> live on the bleeding edge. I’m not sure if all of the 7.5 work is 
> finished in the current prerelease or not.

argh...  not sure i want to be that bleeding edge...  sadly, i can't find a 
release schedule for 2.10.4.  i wonder if 2.11 will work



> —Jeff
>
> On Mon, Apr 30, 2018 at 06:21 Michael Di Domenico 
> 
> wrote:
>>
>> On Mon, Apr 30, 2018 at 9:19 AM, Michael Di Domenico 
>>  wrote:
>> > when i tried to compile 2.10.2 patchless client into rpms under 
>> > rhel
>> > 7.5 using kernel 3.10.0-862.el7.x86_64
>> >
>> > the compilation went fine as far as i can tell and the rpm creation 
>> > seemed to work
>> >
>> > but when i went install the rpms i got
>> >
>> > Error: Package: kmod-lustre-client-2.10.2-1.el7.x86_64
>> > (/kmod-lustre-client-2.10.2-1.el7.x86_64
>> > requires: kernel < 3.10.0-694
>>
>> premature send...
>>
>> requires: kernel < 3.10.0-694
>> Installed: kernel-3.10.0-862.el7.x86_64 (@updates/7.5)
>>
>> did i do something wrong in the recompile of the rpms for the target 
>> kernel or is there a workaround for this?
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
> --
> --
> Jeff Johnson
> Co-Founder
> Aeon Computing
>
> jeff.john...@aeoncomputing.com
> www.aeoncomputing.com
> t: 858-412-3810 x1001   f: 858-412-3845
> m: 619-204-9061
>
> 4170 Morena Boulevard, Suite D - San Diego, CA 92117
>
> High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] problem with one orphan

[Hebenstreit, Michael]

I've got an orphan problem  - this process on the combined mds/mgs is running 
at 100%

   PID USER  PR  NI    VIRT    RES    SHR S  %CPU %MEM TIME+ COMMAND
  5902 root  20   0   0  0  0 R 100.0  0.0 974:22.49 
orph_lfs11-MDD0

And I literally got millions of dmesg entries

[56994.471189] LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) 
lfs11-MDD: could not delete orphan [0x243a7:0xf03:0x0]: rc = -2
[56994.487994] LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) 
Skipped 37467674 previous similar messages
[57594.446202] LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) 
lfs11-MDD: could not delete orphan [0x243a7:0xf03:0x0]: rc = -2
[57594.462993] LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) 
Skipped 37411029 previous similar messages
[58194.421226] LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) 
lfs11-MDD: could not delete orphan [0x243a7:0xf03:0x0]: rc = -2
[58194.438033] LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) 
Skipped 36973695 previous similar messages

any suggestion how I can get rid of this ONE orphan?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Core and Visual Compute Group (DCE)
4100 Sara Road  Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] problem with one orphan

[Hebenstreit, Michael]

A reboot just brings the same problem back


-Original Message-
From: Riccardo Veraldi [mailto:riccardo.vera...@gmail.com] 
Sent: Tuesday, June 05, 2018 9:16 AM
To: Hebenstreit, Michael 
Subject: Re: [lustre-discuss] problem with one orphan

what I do in these cases is reboot the MDS.

cheers

Rick


On 6/5/18 7:51 AM, Hebenstreit, Michael wrote:
> I've got an orphan problem  - this process on the combined mds/mgs is 
> running at 100%
>
>    PID USER  PR  NI    VIRT    RES    SHR S  %CPU %MEM TIME+ 
> COMMAND
>   5902 root  20   0   0  0  0 R 100.0  0.0 974:22.49 
> orph_lfs11-MDD0
>
> And I literally got millions of dmesg entries
>
> [56994.471189] LustreError: 
> 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) lfs11-MDD: could 
> not delete orphan [0x243a7:0xf03:0x0]: rc = -2 [56994.487994] 
> LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) Skipped 
> 37467674 previous similar messages [57594.446202] LustreError: 
> 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) lfs11-MDD: could 
> not delete orphan [0x243a7:0xf03:0x0]: rc = -2 [57594.462993] 
> LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) Skipped 
> 37411029 previous similar messages [58194.421226] LustreError: 
> 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) lfs11-MDD: could 
> not delete orphan [0x243a7:0xf03:0x0]: rc = -2 [58194.438033] 
> LustreError: 5902:0:(mdd_orphans.c:324:mdd_orphan_destroy()) Skipped 
> 36973695 previous similar messages
>
> any suggestion how I can get rid of this ONE orphan?
>
> Thanks
> Michael
>
> --
> -- Michael Hebenstreit Senior Cluster Architect Intel 
> Corporation, MS: RR1-105/H14  Core and Visual Compute Group (DCE)
> 4100 Sara Road  Tel.:   +1 505-794-3144 Rio 
> Rancho, NM 87124 UNITED STATES   E-mail: 
> michael.hebenstr...@intel.com
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] server_bulk_callback errors until server reboots

[Hebenstreit, Michael]

Hello

I have now 2 Lustre systems that suddenly show this error - on a single OST the 
kernel log is filling with messages 

[58858.365663] LustreError: 123642:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880524f7e000
[58865.328317] LustreError: 123640:0:(events.c:447:server_bulk_callback()) 
event type 5, status -61, desc 880cab4ec800
[58865.340792] LustreError: 123641:0:(events.c:447:server_bulk_callback()) 
event type 5, status -61, desc 880524f7c600
[58865.353167] LustreError: 123640:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880cab4ec800
[58865.365503] LustreError: 123641:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880524f7c600

until the server reboots. Clients are on 2.11/RH7.5, servers are on 
2.7.19.10/RH7.4 . Has anyone experienced this before?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Core and Visual Compute Group (DCE)
4100 Sara Road  Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] server_bulk_callback errors until server reboots

[Hebenstreit, Michael]

No, clients do not show any issues. 

-Original Message-
From: White, Cliff 
Sent: Thursday, June 07, 2018 9:26 AM
To: Hebenstreit, Michael ; lustre-discuss 

Subject: Re: [lustre-discuss] server_bulk_callback errors until server reboots


On 6/7/18, 7:00 AM, "lustre-discuss on behalf of Hebenstreit, Michael" 
 wrote:

Hello

I have now 2 Lustre systems that suddenly show this error - on a single OST 
the kernel log is filling with messages 

[58858.365663] LustreError: 123642:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880524f7e000
[58865.328317] LustreError: 123640:0:(events.c:447:server_bulk_callback()) 
event type 5, status -61, desc 880cab4ec800
[58865.340792] LustreError: 123641:0:(events.c:447:server_bulk_callback()) 
event type 5, status -61, desc 880524f7c600
[58865.353167] LustreError: 123640:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880cab4ec800
[58865.365503] LustreError: 123641:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880524f7c600

until the server reboots. Clients are on 2.11/RH7.5, servers are on 
2.7.19.10/RH7.4 . Has anyone experienced this before?

There should be some corresponding error messages on your clients, have you 
checked there? 
cliffw

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Core and Visual Compute Group (DCE)
4100 Sara Road  Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] server_bulk_callback errors until server reboots

[Hebenstreit, Michael]

Thanks – I do not have different type IB within one fabric. But with this info 
I found a few nodes that showed that error, but they are not matching the 
errors I see on the server.

Btw - I got the problem resolved on one FS after upgrading to Lustre 2.11

From: Raj [mailto:rajgau...@gmail.com]
Sent: Thursday, June 07, 2018 10:36 AM
To: Hebenstreit, Michael 
Cc: White, Cliff ; lustre-discuss 

Subject: Re: [lustre-discuss] server_bulk_callback errors until server reboots

I seen the error when we had mix of FDR (using mlx4) and EDR(using mlx5) 
devices in lustre network. server_bulk_callback should have the corresponding 
client_bulk_callback in client.

http://wiki.lustre.org/Infiniband_Configuration_Howto
On Thu, Jun 7, 2018 at 11:24 AM Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:
No, clients do not show any issues.

-Original Message-
From: White, Cliff
Sent: Thursday, June 07, 2018 9:26 AM
To: Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>>; 
lustre-discuss 
mailto:lustre-discuss@lists.lustre.org>>
Subject: Re: [lustre-discuss] server_bulk_callback errors until server reboots


On 6/7/18, 7:00 AM, "lustre-discuss on behalf of Hebenstreit, Michael" 
mailto:lustre-discuss-boun...@lists.lustre.org>
 on behalf of 
michael.hebenstr...@intel.com<mailto:michael.hebenstr...@intel.com>> wrote:

Hello

I have now 2 Lustre systems that suddenly show this error - on a single OST 
the kernel log is filling with messages

[58858.365663] LustreError: 123642:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880524f7e000
[58865.328317] LustreError: 123640:0:(events.c:447:server_bulk_callback()) 
event type 5, status -61, desc 880cab4ec800
[58865.340792] LustreError: 123641:0:(events.c:447:server_bulk_callback()) 
event type 5, status -61, desc 880524f7c600
[58865.353167] LustreError: 123640:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880cab4ec800
[58865.365503] LustreError: 123641:0:(events.c:447:server_bulk_callback()) 
event type 3, status -61, desc 880524f7c600

until the server reboots. Clients are on 2.11/RH7.5, servers are on 
2.7.19.10/RH7.4<http://2.7.19.10/RH7.4> . Has anyone experienced this before?

There should be some corresponding error messages on your clients, have you 
checked there?
cliffw

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Core and Visual Compute Group (DCE)
4100 Sara 
Road<https://maps.google.com/?q=4100+Sara+Road&entry=gmail&source=g>
  Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com<mailto:michael.hebenstr...@intel.com>



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] request for help - ZFS based Lustre, MDT disk not mounting

[Hebenstreit, Michael]

Mount commands hangs, no error messages in kernel log, tried already rebooting 
(twice)  - any ideas?

Thanks
Michael

root   4137  0.0  0.0 123520  1048 pts/0S+   07:56   0:00 mount -t 
lustre mgsmdt/mdt /lfs/lfs11/mdt
root   4138  0.0  0.0  81728  3272 pts/0S+   07:56   0:00 
/sbin/mount.lustre mgsmdt/mdt /lfs/lfs11/mdt -o rw

[root@elfs11m1 ~]# dmesg | grep -i MDT
[  107.238438] LustreError: 137-5: lfs11-MDT_UUID: not available for 
connect from 36.101.16.38@tcp (no target). If you are running an HA pair check 
that the target is mounted on the other server.
[  107.656990] Lustre: lfs11-MDT: Not available for connect from 
36.101.16.39@tcp (not set up)

ZFS does not report any issues, mgs (on the same zpool) mounted without issues

[root@elfs11m1 ~]# zfs list
NAME USED  AVAIL  REFER  MOUNTPOINT
mgsmdt   149G  18.9T  40.8K  /mgsmdt
mgsmdt/mdt   147G  18.9T   147G  /mgsmdt/mdt
mgsmdt/mgs  4.48M  18.9T  4.48M  /mgsmdt/mgs
[root@elfs11m1 ~]# zpool status
  pool: mgsmdt
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 2h14m with 0 errors on Thu May 31 13:36:58 2018
config:

NAMESTATE READ WRITE CKSUM
mgsmdt  ONLINE   0 0 0
  raidz2-0  ONLINE   0 0 0
sdb ONLINE   0 0 0
sdc ONLINE   0 0 0
sdd ONLINE   0 0 0
sde ONLINE   0 0 0
sdf ONLINE   0 0 0
sdg ONLINE   0 0 0
sdh ONLINE   0 0 0
sdi ONLINE   0 0 0
sdj ONLINE   0 0 0
sdk ONLINE   0 0 0
sdl ONLINE   0 0 0
sdm ONLINE   0 0 0
sdn ONLINE   0 0 0
sdo ONLINE   0 0 0
sdp ONLINE   0 0 0
sdq ONLINE   0 0 0
sdr ONLINE   0 0 0
sds ONLINE   0 0 0
sdt ONLINE   0 0 0
sdu ONLINE   0 0 0
sdv ONLINE   0 0 0
sdw ONLINE   0 0 0
sdx ONLINE   0 0 0
sdy ONLINE   0 0 0

errors: No known data errors




Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Core and Visual Compute Group (DCE)
4100 Sara Road  Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] request for help - ZFS based Lustre, MDT disk not mounting

[Hebenstreit, Michael]

Yes, both mgs and mdt modules are present

Module  Size  Used by
osp   341968  27
mdd   408582  17
lod   508629  17
mdt   818340  18
lfsck 735096  20 lod,mdd,mdt
mgs   351348  1
mgc94061  2 mgs
osd_zfs   363587  31
lquota363475  77 mdt,osd_zfs

From: Feng Zhang [mailto:prod.f...@gmail.com]
Sent: Tuesday, June 19, 2018 8:13 AM
To: Hebenstreit, Michael 
Cc: lustre-discuss 
Subject: Re: [lustre-discuss] request for help - ZFS based Lustre, MDT disk not 
mounting

Is lustre module loaded already in kernel and started?

On Tue, Jun 19, 2018 at 10:07 AM, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:
Mount commands hangs, no error messages in kernel log, tried already rebooting 
(twice)  - any ideas?

Thanks
Michael

root   4137  0.0  0.0 123520  1048 pts/0S+   07:56   0:00 mount -t 
lustre mgsmdt/mdt /lfs/lfs11/mdt
root   4138  0.0  0.0  81728  3272 pts/0S+   07:56   0:00 
/sbin/mount.lustre mgsmdt/mdt /lfs/lfs11/mdt -o rw

[root@elfs11m1 ~]# dmesg | grep -i MDT
[  107.238438] LustreError: 137-5: lfs11-MDT_UUID: not available for 
connect from 36.101.16.38@tcp<mailto:36.101.16.38@tcp> (no target). If you are 
running an HA pair check that the target is mounted on the other server.
[  107.656990] Lustre: lfs11-MDT: Not available for connect from 
36.101.16.39@tcp<mailto:36.101.16.39@tcp> (not set up)

ZFS does not report any issues, mgs (on the same zpool) mounted without issues

[root@elfs11m1 ~]# zfs list
NAME USED  AVAIL  REFER  MOUNTPOINT
mgsmdt   149G  18.9T  40.8K  /mgsmdt
mgsmdt/mdt   147G  18.9T   147G  /mgsmdt/mdt
mgsmdt/mgs  4.48M  18.9T  4.48M  /mgsmdt/mgs
[root@elfs11m1 ~]# zpool status
  pool: mgsmdt
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 2h14m with 0 errors on Thu May 31 13:36:58 2018
config:

NAMESTATE READ WRITE CKSUM
mgsmdt  ONLINE   0 0 0
  raidz2-0  ONLINE   0 0 0
sdb ONLINE   0 0 0
sdc ONLINE   0 0 0
sdd ONLINE   0 0 0
sde ONLINE   0 0 0
sdf ONLINE   0 0 0
sdg ONLINE   0 0 0
sdh ONLINE   0 0 0
sdi ONLINE   0 0 0
sdj ONLINE   0 0 0
sdk ONLINE   0 0 0
sdl ONLINE   0 0 0
sdm ONLINE   0 0 0
sdn ONLINE   0 0 0
sdo ONLINE   0 0 0
sdp ONLINE   0 0 0
sdq ONLINE   0 0 0
sdr ONLINE   0 0 0
sds ONLINE   0 0 0
sdt ONLINE   0 0 0
sdu ONLINE   0 0 0
sdv ONLINE   0 0 0
sdw ONLINE   0 0 0
sdx ONLINE   0 0 0
sdy ONLINE   0 0 0

errors: No known data errors




Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  Core and Visual Compute Group (DCE)
4100 Sara Road  Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com<mailto:michael.hebenstr...@intel.com>


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



--
Best,

Feng
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] request for help - ZFS based Lustre, MDT disk not mounting

[Hebenstreit, Michael]

Mount succeeded finally after 30+min
Mount as zfs worked

-Original Message-
From: Peter Bortas [mailto:bor...@gmail.com] 
Sent: Tuesday, June 19, 2018 4:08 PM
To: Hebenstreit, Michael 
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] request for help - ZFS based Lustre, MDT disk not 
mounting

On Tue, Jun 19, 2018 at 4:14 PM Hebenstreit, Michael 
 wrote:
> Mount commands hangs, no error messages in kernel log, tried already 
> rebooting (twice)  - any ideas?
>
> Thanks
> Michael
>
> root   4137  0.0  0.0 123520  1048 pts/0S+   07:56   0:00 mount -t 
> lustre mgsmdt/mdt /lfs/lfs11/mdt
> root   4138  0.0  0.0  81728  3272 pts/0S+   07:56   0:00 
> /sbin/mount.lustre mgsmdt/mdt /lfs/lfs11/mdt -o rw
>

What happens if you try to mount it as -t zfs -o ro?

Regards,
--
Peter Bortas, NSC
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] request for help - ZFS based Lustre, MDT disk not mounting

[Hebenstreit, Michael]

Sort of

The FS is still in a whacky state. Existing clients are ok, but new mounts fail 
- unless I increase the timeout on the client:

echo 3000 > /sys/fs/lustre/timeout



-Original Message-
From: Peter Bortas [mailto:bor...@gmail.com] 
Sent: Tuesday, June 19, 2018 4:24 PM
To: Hebenstreit, Michael 
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] request for help - ZFS based Lustre, MDT disk not 
mounting

On Wed, Jun 20, 2018 at 12:13 AM Hebenstreit, Michael 
 wrote:
> Mount succeeded finally after 30+min

I've had similar situations after unclean shutdowns, but never as bad as 30min. 
With how verbose Lustre is about absolutely everything else maybe it would be 
nice to have some sort of progress report every few minutes when it does 
whatever sturcture traversals it does when mounting.

But problem solved then? Tentative congratulations.

Regards,
--
Peter Bortas, NSC
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[Lustre-discuss] trying to port 1.8.5 to RH6 I'm facing Kernel panics due to an error in the handling of page->private

[Hebenstreit, Michael]

If I understand everything correctly page->private should contain either 0 or a 
pointer to kernel space. For reasons I can not currently comprehend sometimes 
the value is set to 2. llap_cast_private() then tries to access 
page->private->llap_magic, and naturally this leads to NULL pointer dereference 
 
this problem seems to creep up only during parallel reads/writes
 
any ideas?
anyone doing a RH6 port of the 1.8 branch (and no, 2.1 won't help me, my 
backends are 1.7 and can not be changed at this time)?
 
thanks
Michael
 

Michael Hebenstreit Senior Cluster Architect
Intel Corporation   Software and Services Group/HTE
2800 N Center Dr, DP3-307   Tel.:   +1 253 371 3144
WA 98327, DuPont   
UNITED STATES   E-mail: michael.hebenstr...@intel.com
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] Help! Newbie trying to set up Lustre network

[Hebenstreit, Michael]

you should do a "lsmod" and check what modules are loaded :)


-Original Message-
From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Brian J. Murrell
Sent: Tuesday, February 22, 2011 11:17 AM
To: lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] Help! Newbie trying to set up Lustre network

On 11-02-22 02:10 PM, Xiang, Yang wrote:
> Dmesg and syslog are clean and has no entries about lustre client.

That seems suspect.  I don't recall a situation where I have failed a
client mount and there hasn't been something in the client log.  Lustre
is quite verbose, especially when something goes wrong.

b.

-- 
Brian J. Murrell
Senior Software Engineer
Whamcloud, Inc.

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] rhel6.1?

[Hebenstreit, Michael]

I have 1.8.5.56 working for almost a 9 months now patchless with RH61 and OFED 
1.5.3 - so I guess you are safe using 1.8.6 or later

Michael

-Original Message-
From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Michael Di 
Domenico
Sent: Tuesday, November 22, 2011 2:12 PM
To: lustre-discuss
Subject: [Lustre-discuss] rhel6.1?

I'm guessing the 1.8.7-wc1 release from whamcloud should support rhel6 update 
1.  but can anyone lend a yes/no on patchless client support for 1.8.7 on 
rhel6.1 x86_64 with infiniband?  i'm anticipating a cluster upgrade from 
rhel5.4 to rhel6.1 shortly, just checking all the kernel/software support 
lists...
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] is there a way to run Lustre over UDP instead TCP?

[Hebenstreit, Michael]

See title...

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation   Software and Services Group/HTE
2800 N Center Dr, DP3-307   Tel.:   +1 253 371 3144
WA 98327, DuPont
UNITED STATES   E-mail: michael.hebenstr...@intel.com
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5

[Hebenstreit, Michael]

Configure fails at testing for openib - anyone an idea?
Thanks
Michaell

configure:10034: checking whether to enable OpenIB gen2 support
configure:10138: cp conftest.c build && make -d modules  CC=gcc -f 
/home/mhebenst/lustre-2.3.0/build/Makefile LUSTRE_LINUX_CONFIG=/adm
in/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/.config LINUXINCLUDE= 
-I/usr/local/ofed/3.5/src/compat-rdma-3.5/include -I/admin/extra/l
inux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include 
-I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include/generated 
-I
/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include 
-I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include -I/admin/extra/l
inux-2.6.32-358.2.1.el6.x86_64.crt1/include2 -include include/linux/autoconf.h 
-o tmp_include_depends -o scripts -o include/config/MAR
KER -C /admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1 
EXTRA_CFLAGS=-Werror-implicit-function-declaration -g -I/home/mhebenst/lustre
-2.3.0/libcfs/include -I/home/mhebenst/lustre-2.3.0/lnet/include 
-I/home/mhebenst/lustre-2.3.0/lustre/include -I/usr/local/ofed/3.5/sr
c/compat-rdma-3.5/include  M=/home/mhebenst/lustre-2.3.0/build
make[1]: Warning: File `/home/mhebenst/lustre-2.3.0/build/conftest.c' has 
modification time 7.2e+03 s in the future
In file included from 
/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/rdma_cm.h:39,
 from /home/mhebenst/lustre-2.3.0/build/conftest.c:58:
/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h: In function 
âiboe_get_rateâ:
/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:223: error: 
implicit declaration of function ârtnl_lockâ
/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:224: error: 
implicit declaration of function â__ethtool_get_settingsâ
/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:225: error: 
implicit declaration of function ârtnl_unlockâ
make[1]: *** [/home/mhebenst/lustre-2.3.0/build/conftest.o] Error 1
make: *** [_module_/home/mhebenst/lustre-2.3.0/build] Error 2
configure:10141: $? = 2
configure: failed program was:
| /* confdefs.h.  */
| #define PACKAGE_NAME "Lustre"
| #define PACKAGE_TARNAME "lustre"
| #define PACKAGE_VERSION "LUSTRE_VERSION"
| #define PACKAGE_STRING "Lustre LUSTRE_VERSION"
| #define PACKAGE_BUGREPORT "http://bugs.whamcloud.com/";
| #define PACKAGE "lustre"
| #define VERSION "2.3.0"
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define SIZEOF_UNSIGNED_LONG_LONG 8
| #define CDEBUG_ENABLED 1
| #define CDEBUG_ENTRY_EXIT 1
| #define LIBCFS_DEBUG 1
| #define HAVE_SYS_QUOTA_H 1
| #define HAVE_MODULE_LOADING_SUPPORT 1
| #define HAVE_KERN__U64_LONG_LONG 1
| #define HAVE_IS_COMPAT_TASK 1
| #define HAVE_REGISTER_SHRINKER 1
| #define HAVE_SYSCTL_UNNUMBERED 1
| #define HAVE_SCATTERLIST_SETPAGE 1
| #define HAVE_INIT_NET 1
| #define HAVE_DUMP_TRACE 1
| #define HAVE_TRACE_ADDRESS_RELIABLE 1
| #define HAVE_STACKTRACE_WARNING 1
| #define HAVE_CRED_WRAPPERS 1
| #define HAVE_CPUMASK_SIZE 1
| #define HAVE_STRUCT_CRED 1
| #define HAVE_CPU_TOPOLOGY 1
| #define HAVE_TOPOLOGY_CORE_CPUMASK 1
| #define HAVE_TOPOLOGY_THREAD_CPUMASK 1
| #define HAVE_CPUMASK_OF_NODE 1
| #define HAVE_STRUCT_SHASH_ALG 1
| #define HAVE_UNSHARE_FS_STRUCT 1
| #define HAVE_SOCK_MAP_FD_2ARG 1
| #define HAVE_SET_MEMS_ALLOWED 1
| #define STACKTRACE_OPS_HAVE_WALK_STACK 1
| #define HAVE_SHRINKER_WANT_SHRINK_PTR 1
| #define HAVE_LINUX_OOM_H 1
| #define HAVE_OOMADJ_IN_SIG 1
| #define HAVE_SYSCTL_CTLNAME 1
| #define HAVE_DEV_GET_BY_NAME_2ARG 1
| /* end confdefs.h.  */
|
| #include 
|
|   #include 
|   #include 
|   #include 
|   #include 
|   #include 
|   #include 
|   #include 
|
| int
| main (void)
| {
|
|   struct rdma_cm_id  *cm_idi __attribute__ ((unused));
|   struct rdma_conn_param  conn_param __attribute__ 
((unused));
|   struct ib_device_attr   device_attr __attribute__ 
((unused));
|   struct ib_qp_attr   qp_attr __attribute__ 
((unused));
|   struct ib_pool_fmr  pool_fmr __attribute__ 
((unused));
|   enum   ib_cm_rej_reason rej_reason __attribute__ 
((unused));
|
|   rdma_destroy_id(NULL);
|
|   ;
|   return 0;
| }
configure:10158: result: no
configure:10165: error: can't compile with OpenIB gen2 headers under 
/usr/local/ofed/3.5/src/compat-rdma-3.5

___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Re: [Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5

[Hebenstreit, Michael]

That's not my problem - OFED is working, Lustre is not willing to compile :P

Michael

-Original Message-
From: Diep, Minh 
Sent: Friday, April 19, 2013 5:33 PM
To: Hebenstreit, Michael; Lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5

Ofed 3.5 does not support Rhel6.4 kernel yet. I believe 3.5.1 will

Thanks
-Minh

On 4/19/13 5:05 PM, "Hebenstreit, Michael" 
wrote:

>Configure fails at testing for openib - anyone an idea?
>Thanks
>Michaell
>
>configure:10034: checking whether to enable OpenIB gen2 support
>configure:10138: cp conftest.c build && make -d modules  CC=gcc -f 
>/home/mhebenst/lustre-2.3.0/build/Makefile LUSTRE_LINUX_CONFIG=/adm 
>in/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/.config LINUXINCLUDE= 
>-I/usr/local/ofed/3.5/src/compat-rdma-3.5/include -I/admin/extra/l 
>inux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include
>-I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include/ge
>ner
>ated -I
>/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include
>-I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include
>-I/admin/extra/l
>inux-2.6.32-358.2.1.el6.x86_64.crt1/include2 -include 
>include/linux/autoconf.h -o tmp_include_depends -o scripts -o 
>include/config/MAR KER -C 
>/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1
>EXTRA_CFLAGS=-Werror-implicit-function-declaration -g 
>-I/home/mhebenst/lustre -2.3.0/libcfs/include 
>-I/home/mhebenst/lustre-2.3.0/lnet/include
>-I/home/mhebenst/lustre-2.3.0/lustre/include -I/usr/local/ofed/3.5/sr 
>c/compat-rdma-3.5/include  M=/home/mhebenst/lustre-2.3.0/build
>make[1]: Warning: File `/home/mhebenst/lustre-2.3.0/build/conftest.c' 
>has modification time 7.2e+03 s in the future In file included from 
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/rdma_cm.h:39,
> from /home/mhebenst/lustre-2.3.0/build/conftest.c:58:
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h: In 
>function âiboe_get_rateâ:
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:223:
>error: implicit declaration of function ârtnl_lockâ
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:224:
>error: implicit declaration of function â__ethtool_get_settingsâ
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:225:
>error: implicit declaration of function ârtnl_unlockâ
>make[1]: *** [/home/mhebenst/lustre-2.3.0/build/conftest.o] Error 1
>make: *** [_module_/home/mhebenst/lustre-2.3.0/build] Error 2
>configure:10141: $? = 2
>configure: failed program was:
>| /* confdefs.h.  */
>| #define PACKAGE_NAME "Lustre"
>| #define PACKAGE_TARNAME "lustre"
>| #define PACKAGE_VERSION "LUSTRE_VERSION"
>| #define PACKAGE_STRING "Lustre LUSTRE_VERSION"
>| #define PACKAGE_BUGREPORT "http://bugs.whamcloud.com/";
>| #define PACKAGE "lustre"
>| #define VERSION "2.3.0"
>| #define STDC_HEADERS 1
>| #define HAVE_SYS_TYPES_H 1
>| #define HAVE_SYS_STAT_H 1
>| #define HAVE_STDLIB_H 1
>| #define HAVE_STRING_H 1
>| #define HAVE_MEMORY_H 1
>| #define HAVE_STRINGS_H 1
>| #define HAVE_INTTYPES_H 1
>| #define HAVE_STDINT_H 1
>| #define HAVE_UNISTD_H 1
>| #define SIZEOF_UNSIGNED_LONG_LONG 8
>| #define CDEBUG_ENABLED 1
>| #define CDEBUG_ENTRY_EXIT 1
>| #define LIBCFS_DEBUG 1
>| #define HAVE_SYS_QUOTA_H 1
>| #define HAVE_MODULE_LOADING_SUPPORT 1 #define 
>| HAVE_KERN__U64_LONG_LONG 1 #define HAVE_IS_COMPAT_TASK 1 #define 
>| HAVE_REGISTER_SHRINKER 1 #define HAVE_SYSCTL_UNNUMBERED 1 #define 
>| HAVE_SCATTERLIST_SETPAGE 1 #define HAVE_INIT_NET 1 #define 
>| HAVE_DUMP_TRACE 1 #define HAVE_TRACE_ADDRESS_RELIABLE 1 #define 
>| HAVE_STACKTRACE_WARNING 1 #define HAVE_CRED_WRAPPERS 1 #define 
>| HAVE_CPUMASK_SIZE 1 #define HAVE_STRUCT_CRED 1 #define 
>| HAVE_CPU_TOPOLOGY 1 #define HAVE_TOPOLOGY_CORE_CPUMASK 1 #define 
>| HAVE_TOPOLOGY_THREAD_CPUMASK 1 #define HAVE_CPUMASK_OF_NODE 1 #define 
>| HAVE_STRUCT_SHASH_ALG 1 #define HAVE_UNSHARE_FS_STRUCT 1 #define 
>| HAVE_SOCK_MAP_FD_2ARG 1 #define HAVE_SET_MEMS_ALLOWED 1 #define 
>| STACKTRACE_OPS_HAVE_WALK_STACK 1 #define 
>| HAVE_SHRINKER_WANT_SHRINK_PTR 1 #define HAVE_LINUX_OOM_H 1 #define 
>| HAVE_OOMADJ_IN_SIG 1 #define HAVE_SYSCTL_CTLNAME 1 #define 
>| HAVE_DEV_GET_BY_NAME_2ARG 1
>| /* end confdefs.h.  */
>|
>| #include 
>|
>|   #include 
>|   #include 
>|   #include 
>|   #include 
>|   #include 
>|   #include 
>|   #include 
>|
>| int
>| main (void)
>| {
>|
>|   struct rdma_cm_i

Re: [Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5

[Hebenstreit, Michael]

Could solve it - the problem was only in configure; to compile conftest.c some 
changes in the OFED header were necessary (note - I think the changes make 
sense there, as the header file uses functions otherwise undefined):

ofed/3.5/src/compat-rdma/include/rdma/ib_addr.h

#include 
+   #include 
+
+   #if (LINUX_VERSION_CODE < KERNEL_VERSION(3,2,0))
+
+   extern int __ethtool_get_settings(struct net_device *dev,
+  struct ethtool_cmd *cmd);
+   #endif

struct rdma_addr_client {
atomic_t refcount;
struct completion comp;
};

There is also another error, but without any impact. Configure tries to source 
src/ofa_kernel/config.mk - in earlier versions this was a simple config file of 
the form "PARAM=VALUE"; now the file is in Makefile format; sourcing that from 
configure as bourne shell script is .. not advisable

Michael

-Original Message-
From: lustre-discuss-boun...@lists.lustre.org 
[mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Hebenstreit, 
Michael
Sent: Friday, April 19, 2013 5:39 PM
To: Diep, Minh; Lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5

That's not my problem - OFED is working, Lustre is not willing to compile :P

Michael

-Original Message-
From: Diep, Minh 
Sent: Friday, April 19, 2013 5:33 PM
To: Hebenstreit, Michael; Lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5

Ofed 3.5 does not support Rhel6.4 kernel yet. I believe 3.5.1 will

Thanks
-Minh

On 4/19/13 5:05 PM, "Hebenstreit, Michael" 
wrote:

>Configure fails at testing for openib - anyone an idea?
>Thanks
>Michaell
>
>configure:10034: checking whether to enable OpenIB gen2 support
>configure:10138: cp conftest.c build && make -d modules  CC=gcc -f 
>/home/mhebenst/lustre-2.3.0/build/Makefile LUSTRE_LINUX_CONFIG=/adm 
>in/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/.config LINUXINCLUDE= 
>-I/usr/local/ofed/3.5/src/compat-rdma-3.5/include -I/admin/extra/l 
>inux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include
>-I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include/ge
>ner
>ated -I
>/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include
>-I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include
>-I/admin/extra/l
>inux-2.6.32-358.2.1.el6.x86_64.crt1/include2 -include 
>include/linux/autoconf.h -o tmp_include_depends -o scripts -o 
>include/config/MAR KER -C 
>/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1
>EXTRA_CFLAGS=-Werror-implicit-function-declaration -g 
>-I/home/mhebenst/lustre -2.3.0/libcfs/include 
>-I/home/mhebenst/lustre-2.3.0/lnet/include
>-I/home/mhebenst/lustre-2.3.0/lustre/include -I/usr/local/ofed/3.5/sr 
>c/compat-rdma-3.5/include  M=/home/mhebenst/lustre-2.3.0/build
>make[1]: Warning: File `/home/mhebenst/lustre-2.3.0/build/conftest.c' 
>has modification time 7.2e+03 s in the future In file included from 
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/rdma_cm.h:39,
> from /home/mhebenst/lustre-2.3.0/build/conftest.c:58:
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h: In 
>function âiboe_get_rateâ:
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:223:
>error: implicit declaration of function ârtnl_lockâ
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:224:
>error: implicit declaration of function â__ethtool_get_settingsâ
>/usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/ib_addr.h:225:
>error: implicit declaration of function ârtnl_unlockâ
>make[1]: *** [/home/mhebenst/lustre-2.3.0/build/conftest.o] Error 1
>make: *** [_module_/home/mhebenst/lustre-2.3.0/build] Error 2
>configure:10141: $? = 2
>configure: failed program was:
>| /* confdefs.h.  */
>| #define PACKAGE_NAME "Lustre"
>| #define PACKAGE_TARNAME "lustre"
>| #define PACKAGE_VERSION "LUSTRE_VERSION"
>| #define PACKAGE_STRING "Lustre LUSTRE_VERSION"
>| #define PACKAGE_BUGREPORT "http://bugs.whamcloud.com/";
>| #define PACKAGE "lustre"
>| #define VERSION "2.3.0"
>| #define STDC_HEADERS 1
>| #define HAVE_SYS_TYPES_H 1
>| #define HAVE_SYS_STAT_H 1
>| #define HAVE_STDLIB_H 1
>| #define HAVE_STRING_H 1
>| #define HAVE_MEMORY_H 1
>| #define HAVE_STRINGS_H 1
>| #define HAVE_INTTYPES_H 1
>| #define HAVE_STDINT_H 1
>| #define HAVE_UNISTD_H 1
>| #define SIZEOF_UNSIGNED_LONG_LONG 8
>| #define CDEBUG_ENABLED 1
>| #define CDEBUG_ENTRY_EXIT 1
>| #define LIBCFS_DEBUG 1
>| #define HAVE_SYS_QUOTA_H 1
>| #define HAVE_MODULE_LOADING_SUPPORT 1 #define 
>| HAVE_KERN__U64_LONG_LONG 1 #define HAVE_IS_COMPAT_TASK 1 #define 
>| HAVE_REGISTER_SHRINKER 1 #define H

Re: [Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5

[Hebenstreit, Michael]

I had to do quite a few patches to OFE 3.5 to get it compiled. Essentially 
nothing but taking care of functions backported by RH to the kernel

Michael

-Original Message-
From: Shuichi Ihara [mailto:sih...@ddn.com] 
Sent: Saturday, April 20, 2013 3:23 AM
To: Hebenstreit, Michael
Cc: Lustre-discuss@lists.lustre.org
Subject: Re: [Lustre-discuss] configure error using Lustre 2.3 and OFED 3.5


Hi,

The lustre patches to build with OFED-3.5 are landed in master branch, but not 
included in lustre-2.3.
try this patch http://review.whamcloud.com/3011 (also need 
http://review.whamcloud.com/6048)

btw, did you apply any patches to OFED-3.5 to build with RHEL6.4's kernel?
As far as I tested, compat-driver didn't work on the latest RHEL6.4's kernel. 
I've filed on compat-driver's bugzilla and it was fixed in the latest branch. 
https://bugzilla.kernel.org/show_bug.cgi?id=55971

Also, compat-rdma failed on the latest RHEL6.3 kernel as well as RHEL6.4. It 
was filed at ofed bugzilla as well.
see http://bugs.openfabrics.org/show_bug.cgi?id=2421 and 
https://jira.hpdd.intel.com/browse/LU-2975

Anyway, we are waiting for these fixes for the Lustre.

Thanks
Ihara
 
On Apr 20, 2013, at 2:15 PM, "Hebenstreit, Michael" 
 wrote:

> Could solve it - the problem was only in configure; to compile conftest.c 
> some changes in the OFED header were necessary (note - I think the changes 
> make sense there, as the header file uses functions otherwise undefined):
> 
> ofed/3.5/src/compat-rdma/include/rdma/ib_addr.h
> 
>#include 
> +   #include 
> +
> +   #if (LINUX_VERSION_CODE < KERNEL_VERSION(3,2,0))
> +
> +   extern int __ethtool_get_settings(struct net_device *dev,
> +  struct ethtool_cmd *cmd);
> +   #endif
> 
>struct rdma_addr_client {
>atomic_t refcount;
>struct completion comp;
>};
> 
> There is also another error, but without any impact. Configure tries 
> to source src/ofa_kernel/config.mk - in earlier versions this was a 
> simple config file of the form "PARAM=VALUE"; now the file is in 
> Makefile format; sourcing that from configure as bourne shell script 
> is .. not advisable
> 
> Michael
> 
> -Original Message-
> From: lustre-discuss-boun...@lists.lustre.org 
> [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of 
> Hebenstreit, Michael
> Sent: Friday, April 19, 2013 5:39 PM
> To: Diep, Minh; Lustre-discuss@lists.lustre.org
> Subject: Re: [Lustre-discuss] configure error using Lustre 2.3 and 
> OFED 3.5
> 
> That's not my problem - OFED is working, Lustre is not willing to 
> compile :P
> 
> Michael
> 
> -Original Message-
> From: Diep, Minh
> Sent: Friday, April 19, 2013 5:33 PM
> To: Hebenstreit, Michael; Lustre-discuss@lists.lustre.org
> Subject: Re: [Lustre-discuss] configure error using Lustre 2.3 and 
> OFED 3.5
> 
> Ofed 3.5 does not support Rhel6.4 kernel yet. I believe 3.5.1 will
> 
> Thanks
> -Minh
> 
> On 4/19/13 5:05 PM, "Hebenstreit, Michael" 
> 
> wrote:
> 
>> Configure fails at testing for openib - anyone an idea?
>> Thanks
>> Michaell
>> 
>> configure:10034: checking whether to enable OpenIB gen2 support
>> configure:10138: cp conftest.c build && make -d modules  CC=gcc -f 
>> /home/mhebenst/lustre-2.3.0/build/Makefile LUSTRE_LINUX_CONFIG=/adm 
>> in/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/.config LINUXINCLUDE= 
>> -I/usr/local/ofed/3.5/src/compat-rdma-3.5/include -I/admin/extra/l 
>> inux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include
>> -I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include/
>> ge
>> ner
>> ated -I
>> /admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include
>> -I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include
>> -I/admin/extra/l
>> inux-2.6.32-358.2.1.el6.x86_64.crt1/include2 -include 
>> include/linux/autoconf.h -o tmp_include_depends -o scripts -o 
>> include/config/MAR KER -C
>> /admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1
>> EXTRA_CFLAGS=-Werror-implicit-function-declaration -g 
>> -I/home/mhebenst/lustre -2.3.0/libcfs/include 
>> -I/home/mhebenst/lustre-2.3.0/lnet/include
>> -I/home/mhebenst/lustre-2.3.0/lustre/include -I/usr/local/ofed/3.5/sr 
>> c/compat-rdma-3.5/include  M=/home/mhebenst/lustre-2.3.0/build
>> make[1]: Warning: File `/home/mhebenst/lustre-2.3.0/build/conftest.c' 
>> has modification time 7.2e+03 s in the future In file included from 
>> /usr/local/ofed/3.5/src/compat-rdma-3.5/include/rdma/rdma_cm.h:39,
>>from /home/mhebenst/lustre-2.3.0/build/conftest.c:58:
>> /usr/

Re: [lustre-discuss] lustre client on centos 7.7

[Hebenstreit, Michael]

In the new kernel include/linux/pci-dma.h is missing – I copied it from an 
older version

From: lustre-discuss  On Behalf Of 
w...@umich.edu
Sent: Wednesday, September 25, 2019 15:13
To: discussion 
Subject: Re: [lustre-discuss] lustre client on centos 7.7

Hi All
We are also encounting problems when compiling the lustre client on Redhat 7.7, 
kernel
3.10.0-1062.1.1.el7.x86_64 #1 SMP
cat /etc/redhat-release
Scientific Linux release 7.7 (Nitrogen)

#rpmbuild  --rebuild --without servers --with lnet-dlc lustre-2.12.2-1.src.rpm
Just a few lines from the error messages:
In file included from 
/root/rpmbuild/BUILD/lustre-2.12.2/libcfs/include/libcfs/libcfs.h:43:0,
 from 
/root/rpmbuild/BUILD/lustre-2.12.2/lnet/klnds/o2iblnd/o2iblnd.h:82,
 from 
/root/rpmbuild/BUILD/lustre-2.12.2/lnet/klnds/o2iblnd/o2iblnd.c:38:
/root/rpmbuild/BUILD/lustre-2.12.2/lnet/klnds/o2iblnd/o2iblnd.c:2352:16: error: 
'struct kib_tx' has no member named 'tx_frags'
  sizeof(*tx->tx_frags));

I wonder if anyone has any idea about this.

Cheers!



-Wenjing
w...@umich.edu

From: Arman Khalatyan
Date: 2019-09-25 07:45
To: Degremont, Aurelien
CC: Lustre discussion
Subject: Re: [lustre-discuss] lustre client on centos 7.7
Hi,Aurélien
thanks for pointing me in the right direction.
On my system I needed to add a few libs in order to compile the client properly:
yum install openmpi-devel.x86_64 readline-devel
module load mpi/openmpi-x86_64
rpmbuild  --rebuild --without servers --with lnet-dlc   
lustre-2.12.58_80_g9209e91-1.src.rpm

now everything looks good...
a.

On Sun, Sep 22, 2019 at 5:18 PM Degremont, Aurelien 
mailto:degre...@amazon.com>> wrote:
The Lustre 2.10 branch don't support CentOS 7.7
Lustre 2.12.3 and Lustre 2.13 will.

However Lustre 2.10.8 can built on CentOS 7.7 if you can afford to miss 
Infiniband support.
This is the part of the code that creates this problem.  ./configure 
--with-ofed=no will disable it.

Aurélien

De : lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 au nom de Arman Khalatyan mailto:arm2...@gmail.com>>
Date : dimanche 22 septembre 2019 à 16:26
À : Lustre discussion 
mailto:lustre-discuss@lists.lustre.org>>
Objet : [lustre-discuss] lustre client on centos 7.7

Hello,
are there any lustre client with the CentOS 7.7?
i was trying to compile the listre 2.10 with the recent centos 7.7 it was 
failing with the error:
rpmbuild  --rebuild --without servers --with lnet-dlc lustre-2.10.8-1.src.rpm

/root/rpmbuild/BUILD/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.h:69:27: fatal 
error: linux/pci-dma.h: No such file or directory
 #include 
   ^
compilation terminated.

looks like something changed in the Kernel drivers:(

thank you beforehand,
Arman
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

Could anyone point out to me what the downside of having an inode size of 1k on 
the MDT would be (compared to the 4k default)?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

This is for an archiving system (1PB) with no performance or other special 
requirements but a lot of small files (I know, not a Lustre speciality)

Thanks
Michael

From: Colin Faber 
Sent: Wednesday, October 02, 2019 11:19
To: Hebenstreit, Michael 
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

It's too small to accommodate new features

On Wed, Oct 2, 2019 at 11:08 AM Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:
Could anyone point out to me what the downside of having an inode size of 1k on 
the MDT would be (compared to the 4k default)?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com<mailto:michael.hebenstr...@intel.com>


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS

and I'd like to use --mkfsoptions='-i 1024' to have more inodes in the MDT. We 
already run out of inodes on that FS (probably due to an ZFS bug in early IEEL 
version) - so I'd like to increase #inodes if possible


-Original Message-
From: Mohr Jr, Richard Frank  
Sent: Wednesday, October 02, 2019 13:39
To: Hebenstreit, Michael 
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT



> On Oct 2, 2019, at 1:08 PM, Hebenstreit, Michael 
>  wrote:
> 
> Could anyone point out to me what the downside of having an inode size of 1k 
> on the MDT would be (compared to the 4k default)?

Are you talking about the inode size, or the “-i” option to mkfs.lustre (which 
actually controls the ratio of inodes to free space)?

—Rick

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

So you are saying on a zfs based Lustre there is no way to increase the number 
of available inodes? I have 8TB MDT with roughly 17G inodes

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt

Formating under Lustre 2.10.8

mkfs.lustre --mdt --backfstype=zfs --fsname=lfsarc01 --index=0 
--mgsnid="36.101.92.22@tcp" --reformat mdt/mdt

this translates to only 948M inodes on the Lustre FS.

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt
mdt/mdt   948016092263   9480158291% /lfs/lfsarc01/mdt

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt
mdt/mdt  8.2T   24M  8.2T   1% /lfs/lfsarc01/mdt

and there is no reasonable option to provide more file entries except for 
adding another MDT?

Thanks
Michael

From: Andreas Dilger 
Sent: Wednesday, October 02, 2019 18:49
To: Hebenstreit, Michael 
Cc: Mohr Jr, Richard Frank ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

There are several confusing/misleading comments on this thread that need to be 
clarified...

On Oct 2, 2019, at 13:45, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS

Note that I've updated this page to reflect current defaults.  The Lustre 
Operations Manual has a much better description of these parameters.


and I'd like to use --mkfsoptions='-i 1024' to have more inodes in the MDT. We 
already run out of inodes on that FS (probably due to an ZFS bug in early IEEL 
version) - so I'd like to increase #inodes if possible.

The "-i 1024" option (bytes-per-inode ratio) is only needed for ldiskfs since 
it statically allocates the inodes at mkfs time, it is not relevant for ZFS 
since ZFS dynamically allocates inodes and blocks as needed.

On Oct 2, 2019, at 14:00, Colin Faber 
mailto:cfa...@gmail.com>> wrote:
With 1K inodes you won't have space to accommodate new features, IIRC the 
current minimal limit on modern lustre is 2K now. If you're running out of MDT 
space you might consider DNE and multiple MDT's to accommodate that larger name 
space.

To clarify, since Lustre 2.10 any new ldiskfs MDT will allocate 1024 bytes for 
the inode itself (-I 1024).  That allows enough space *within* the inode to 
efficiently store xattrs for more complex layouts (PFL, FLR, DoM).  If xattrs 
do not fit inside the inode itself then they will be stored in an external 4KB 
inode block.

The MDT is formatted with a bytes-per-inode *ratio* of 2.5KB, which means 
(approximately) one inode will be created for every 2.5kB of the total MDT 
size.  That 2.5KB of space includes the 1KB for the inode itself, plus space 
for a directory entry (or multiple if hard-linked), extra xattrs, the journal 
(up to 4GB for large MDTs), Lustre recovery logs, ChangeLogs, etc.  Each 
directory inode will have at least one 4KB block allocated.

So, it is _possible_ to reduce the inode *ratio* below 2.5KB if you know what 
you are doing (e.g. 2KB/inode or 1.5KB/inode, this can be an arbitrary number 
of bytes, it doesn't have to be an even multiple of anything) but it definitely 
isn't possible to have 1KB inode size and 1KB per inode ratio, as there 
wouldn't be *any* space left for directories, log files, journal, etc.

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

Please note the differences between inodes on the ZFS and inodes on the mdt 
lustre. In the previous incarnation the file system run out of ionodes as 
reported on the Lustre, even though the mdt was only half filled and zfs 
backend still reported free inodes.

From: Shaun Tancheff 
Sent: Thursday, October 03, 2019 05:41
To: Degremont, Aurelien ; Hebenstreit, Michael 
; Andreas Dilger 
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

Hi,

A little pedantic but for ‘inodes’ don’t exist in a zfs pool per-se. So the 
code which attempts to report the number of inodes used/available guesses based 
on the average per-object utilization rate. If you have a many large files your 
reported number of inodes goes down faster than if you have a many small files.

I think the primary take away is that for a zfs backend you will run out of 
_inodes_ at the same time as you run out of _space_, there simply is no 
distinction.

From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 on behalf of "Degremont, Aurelien" 
mailto:degre...@amazon.com>>
Date: Thursday, October 3, 2019 at 6:11 PM
To: "Hebenstreit, Michael" 
mailto:michael.hebenstr...@intel.com>>, Andreas 
Dilger mailto:adil...@whamcloud.com>>
Cc: "lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>" 
mailto:lustre-discuss@lists.lustre.org>>
Subject: Re: [lustre-discuss] changing inode size on MDT

As Andreas said "it is not relevant for ZFS since ZFS dynamically allocates 
inodes and blocks as needed"

"as needed" is the important part. In your example, your MDT is almost empty, 
so 17G inodes for an empty MDT seems pretty sufficient.
As you will create new files and use these inodes, you will see the total 
number of inodes change.

But I don't what is the maximum number of inodes you can estimate for a 8.3 TiB 
MDT…

De : lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 au nom de "Hebenstreit, Michael" 
mailto:michael.hebenstr...@intel.com>>
Date : jeudi 3 octobre 2019 à 13:04
À : Andreas Dilger mailto:adil...@whamcloud.com>>
Cc : "lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>" 
mailto:lustre-discuss@lists.lustre.org>>
Objet : Re: [lustre-discuss] changing inode size on MDT

So you are saying on a zfs based Lustre there is no way to increase the number 
of available inodes? I have 8TB MDT with roughly 17G inodes

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt

Formating under Lustre 2.10.8

mkfs.lustre --mdt --backfstype=zfs --fsname=lfsarc01 --index=0 
--mgsnid="36.101.92.22@tcp<mailto:36.101.92.22@tcp>" --reformat mdt/mdt

this translates to only 948M inodes on the Lustre FS.

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt
mdt/mdt   948016092263   9480158291% /lfs/lfsarc01/mdt

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt
mdt/mdt  8.2T   24M  8.2T   1% /lfs/lfsarc01/mdt

and there is no reasonable option to provide more file entries except for 
adding another MDT?

Thanks
Michael

From: Andreas Dilger mailto:adil...@whamcloud.com>>
Sent: Wednesday, October 02, 2019 18:49
To: Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>>
Cc: Mohr Jr, Richard Frank mailto:rm...@utk.edu>>; 
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: Re: [lustre-discuss] changing inode size on MDT

There are several confusing/misleading comments on this thread that need to be 
clarified...

On Oct 2, 2019, at 13:45, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS

Note that I've updated this page to reflect current defaults.  The Lustre 
Operations Manual has a much better description of these parameters.


and I'd like to use --mkfsoptions='-i 1024' to have more inodes in the MDT. We 
already run out of inodes on that FS (probably due to an ZFS bug in early IEEL 
version) - so I'd like to increase #inodes if possible.

The "-i 1024" option (bytes-per-inode ratio) is only needed for ldiskfs since 
it statically allocates the inodes at mkfs time, it is not relevant for ZFS 
since ZFS dynamically allocates inodes and blocks as needed.

On Oct 2, 2019, at 14:00, Colin Faber 
mailto:cfa...@gmail.com>> wrote:
With 1K inodes you won't have space to accommodate new features, 

[lustre-discuss] error compiling Lustre on new Mellanox OFED 4.7

[Hebenstreit, Michael]

Running new Mellanox OFED 4.7 on CentOS 7.7 kernel I tried to compile Lustre 
and got the following error

make[3]: Entering directory 
`/global/panfs01/admin/src/3.10.0-1062.1.2.el7.crt1.x86_64'
  CC [M]  /tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.o
In file included from /tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c:38:0:
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.h: In function 
'kiblnd_sg_dma_address':
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.h:1162:9: error: implicit 
declaration of function 'ib_sg_dma_address' 
[-Werror=implicit-function-declaration]
 return ib_sg_dma_address(dev, sg);
 ^
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.h: In function 
'kiblnd_sg_dma_len':
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.h:1168:9: error: implicit 
declaration of function 'ib_sg_dma_len' [-Werror=implicit-function-declaration]
 return ib_sg_dma_len(dev, sg);
 ^
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c: In function 
'kiblnd_create_fmr_pool':
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c:1676:30: error: 'struct 
ib_device' has no member named 'alloc_fmr'
  if (fpo->fpo_hdev->ibh_ibdev->alloc_fmr &&
  ^
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c:1677:30: error: 'struct 
ib_device' has no member named 'dealloc_fmr'
  fpo->fpo_hdev->ibh_ibdev->dealloc_fmr &&
  ^
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c:1678:30: error: 'struct 
ib_device' has no member named 'map_phys_fmr'
  fpo->fpo_hdev->ibh_ibdev->map_phys_fmr &&
  ^
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c:1679:30: error: 'struct 
ib_device' has no member named 'unmap_fmr'
  fpo->fpo_hdev->ibh_ibdev->unmap_fmr) {
  ^
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c: In function 
'kiblnd_fmr_pool_unmap':
/tmp/lustre-2.10.8/lnet/klnds/o2iblnd/o2iblnd.c:1825:7: error: void value not 
ignored as it ought to be
rc = ib_fmr_pool_unmap(fmr->fmr_pfmr);
   ^
cc1: all warnings being treated as errors

looks like Mallenaox changed some names, and also the ib_device declaration. 
Using these patches I got it to work, but it would be nice if someone could 
check if I did something completely stupid 😊

--- lnet/klnds/o2iblnd/o2iblnd.c.orig   2019-10-03 11:11:43.0 -0600
+++ lnet/klnds/o2iblnd/o2iblnd.c2019-10-03 11:32:33.0 -0600
@@ -1673,10 +1673,10 @@

/* Check for FMR or FastReg support */
fpo->fpo_is_fmr = 0;
-   if (fpo->fpo_hdev->ibh_ibdev->alloc_fmr &&
-   fpo->fpo_hdev->ibh_ibdev->dealloc_fmr &&
-   fpo->fpo_hdev->ibh_ibdev->map_phys_fmr &&
-   fpo->fpo_hdev->ibh_ibdev->unmap_fmr) {
+   if (fpo->fpo_hdev->ibh_ibdev->ops.alloc_fmr &&
+   fpo->fpo_hdev->ibh_ibdev->ops.dealloc_fmr &&
+   fpo->fpo_hdev->ibh_ibdev->ops.map_phys_fmr &&
+   fpo->fpo_hdev->ibh_ibdev->ops.unmap_fmr) {
LCONSOLE_INFO("Using FMR for registration\n");
fpo->fpo_is_fmr = 1;
} else if (dev_attr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS) {
@@ -1822,8 +1822,8 @@
fps = fpo->fpo_owner;
if (fpo->fpo_is_fmr) {
if (fmr->fmr_pfmr) {
-   rc = ib_fmr_pool_unmap(fmr->fmr_pfmr);
-   LASSERT(!rc);
+   ib_fmr_pool_unmap(fmr->fmr_pfmr);
+// LASSERT(!rc);
fmr->fmr_pfmr = NULL;
}

--- lnet/klnds/o2iblnd/o2iblnd.h.orig   2019-10-03 11:13:30.0 -0600
+++ lnet/klnds/o2iblnd/o2iblnd.h2019-10-03 11:20:32.0 -0600
@@ -1159,13 +1159,13 @@
static inline __u64 kiblnd_sg_dma_address(struct ib_device *dev,
   struct scatterlist *sg)
{
-return ib_sg_dma_address(dev, sg);
+return sg_dma_address(sg);
}

static inline unsigned int kiblnd_sg_dma_len(struct ib_device *dev,
  struct scatterlist *sg)
{
-return ib_sg_dma_len(dev, sg);
+return sg_dma_len(sg);
}

/* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

So bottom line - don't change the default values, it won't get better?

From: Andreas Dilger 
Sent: Thursday, October 03, 2019 19:38
To: Hebenstreit, Michael 
Cc: Mohr Jr, Richard Frank ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

On Oct 3, 2019, at 05:03, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

So you are saying on a zfs based Lustre there is no way to increase the number 
of available inodes? I have 8TB MDT with roughly 17G inodes

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt

For ZFS the only way to increase inodes on the *MDT* is to increase the size of 
the MDT, though more on that below.  Note that the "number of inodes" reported 
by ZFS is an estimate based on the currently-allocated blocks and inodes (i.e. 
bytes_per_inode_ratio = bytes_used / inodes_used, total inode estimate = 
bytes_free / inode_ratio + inodes_used), which becomes more accurate as the MDT 
becomes more full.  With 17B inodes on a 8TB MDT that is an bytes-per-inode 
ratio of 497, which is unrealistically low for Lustre since the MDT will always 
stores multiple xattrs on each inode.  Note that the filesystem only has 6 
inodes allocated, so the ZFS total inodes estimate is unrealistically high and 
will get better as more inodes are allocated in the filesystem.

Formating under Lustre 2.10.8

mkfs.lustre --mdt --backfstype=zfs --fsname=lfsarc01 --index=0 
--mgsnid="36.101.92.22@tcp<mailto:36.101.92.22@tcp>" --reformat mdt/mdt

this translates to only 948M inodes on the Lustre FS.

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt
mdt/mdt   948016092263   9480158291% /lfs/lfsarc01/mdt

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt
mdt/mdt  8.2T   24M  8.2T   1% /lfs/lfsarc01/mdt

and there is no reasonable option to provide more file entries except for 
adding another MDT?

The Lustre statfs code will weight in some initial estimates for the 
bytes-per-inode ratio when computing the total inode estimate for the 
filesystem.  When the filesystem is nearly empty, as is the case here, then 
those initial estimates will dominate, but once you've allocated a few thousand 
inodes in the filesystem the actual values will dominate and you will have a 
much more accurate number for the total inode count.  This will probably be 
more in the range of 2B-4B inodes in the end, unless you also use Data-on-MDT 
(Lustre 2.11 and later) to store small files directly on the MDT.

You've also excluded the OST lines from the above output?  For the Lustre 
filesystem you (typically) also need at least one OST inode (object) for each 
file in the filesystem, possibly more than one, so "df" of the Lustre 
filesystem may also be limited by the number of inodes reported by the OSTs 
(which may themselves depend on the average bytes-per-inode for files stored on 
the OST).  If you use Data-on-MDT and only have a small files, then no OST 
object is needed for small files, but you consume correspondingly more space on 
the MDT.

Cheers, Andreas


From: Andreas Dilger mailto:adil...@whamcloud.com>>
Sent: Wednesday, October 02, 2019 18:49
To: Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>>
Cc: Mohr Jr, Richard Frank mailto:rm...@utk.edu>>; 
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: Re: [lustre-discuss] changing inode size on MDT

There are several confusing/misleading comments on this thread that need to be 
clarified...

On Oct 2, 2019, at 13:45, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

http://wiki.lustre.org/Lustre_Tuning#Number_of_Inodes_for_MDS

Note that I've updated this page to reflect current defaults.  The Lustre 
Operations Manual has a much better description of these parameters.


and I'd like to use --mkfsoptions='-i 1024' to have more inodes in the MDT. We 
already run out of inodes on that FS (probably due to an ZFS bug in early IEEL 
version) - so I'd like to increase #inodes if possible.

The "-i 1024" option (bytes-per-inode ratio) is only needed for ldiskfs since 
it statically allocates the inodes at mkfs time, it is not relevant for ZFS 
since ZFS dynamically allocates inodes and blocks as needed.

On Oct 2, 2019, at 14:00, Colin Faber 
mailto:cfa...@gmail.com>> wrote:
With 1K inodes you won't have space to accommodate new features, IIRC the 
current minimal limit on modern lustre is 2K now. If you're running out

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

So we went ahead and used the FS – using rsync to duplicate the existing FS. 
The inodes available on the NEW mdt (which is almost twice the size of the 
second mdt) are dropping rapidly and are now LESS than on the smaller mdt (even 
though the sync is only 90% complete). Both FS are running almost identical 
Lustre 2.10. I cannot say anymore which ZFS version was used to format the good 
FS.

Any ideas why those 2 MDTs behave so differently?

old GOOD FS:
# df -i
mgt/mgt   81718714   205   817185091% /lfs/lfsarc02/mgt
mdt/mdt  458995000 130510339  328484661   29% /lfs/lfsarc02/mdt
# df -h
mgt/mgt  427G  7.0M  427G   1% /lfs/lfsarc02/mgt
mdt/mdt  4.6T  1.4T  3.3T  29% /lfs/lfsarc02/mdt
# rpm -q -a | grep zfs
libzfs2-0.7.9-1.el7.x86_64
lustre-osd-zfs-mount-2.10.4-1.el7.x86_64
lustre-zfs-dkms-2.10.4-1.el7.noarch
zfs-0.7.9-1.el7.x86_64
zfs-dkms-0.7.9-1.el7.noarch

new BAD FS
# df -ih
mgt/mgt83M   169   83M1% /lfs/lfsarc01/mgt
mdt/mdt   297M  122M  175M   42% /lfs/lfsarc01/mdt
# df -h
mgt/mgt  427G  5.8M  427G   1% /lfs/lfsarc01/mgt
mdt/mdt  8.2T  3.4T  4.9T  41% /lfs/lfsarc01/mdt
# rpm -q -a | grep zfs
libzfs2-0.7.9-1.el7.x86_64
lustre-osd-zfs-mount-2.10.8-1.el7.x86_64
lustre-zfs-dkms-2.10.8-1.el7.noarch
zfs-0.7.9-1.el7.x86_64
zfs-dkms-0.7.9-1.el7.noarch

From: Andreas Dilger 
Sent: Thursday, October 03, 2019 20:38
To: Hebenstreit, Michael 
Cc: Mohr Jr, Richard Frank ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

On Oct 3, 2019, at 20:09, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

So bottom line – don’t change the default values, it won’t get better?

Like I wrote previously, there *are* no default/tunable values to change for 
ZFS.  The tunables are only for ldiskfs, which statically allocates everything, 
but is will cause problems if you guessed incorrectly at the instant you format 
the filesystem.

The number reported by raw ZFS and by Lustre-on-ZFS is just an estimate, and 
you will (essentially) run out of inodes once you run out of space on the MDT 
or all OSTs.  And I didn't say "it won't get better", actually I said the 
estimate _will_ get better once you actually start using the filesystem.

If the (my estimate) 2-3B inodes on the MDT is insufficient, you can always add 
another (presumably mirrored) VDEV to the MDT, or add a new MDT to the 
filesystem to increase the number of inodes available.

Cheers, Andreas



From: Andreas Dilger mailto:adil...@whamcloud.com>>
Sent: Thursday, October 03, 2019 19:38
To: Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>>
Cc: Mohr Jr, Richard Frank mailto:rm...@utk.edu>>; 
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: Re: [lustre-discuss] changing inode size on MDT

On Oct 3, 2019, at 05:03, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

So you are saying on a zfs based Lustre there is no way to increase the number 
of available inodes? I have 8TB MDT with roughly 17G inodes

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt

For ZFS the only way to increase inodes on the *MDT* is to increase the size of 
the MDT, though more on that below.  Note that the "number of inodes" reported 
by ZFS is an estimate based on the currently-allocated blocks and inodes (i.e. 
bytes_per_inode_ratio = bytes_used / inodes_used, total inode estimate = 
bytes_free / inode_ratio + inodes_used), which becomes more accurate as the MDT 
becomes more full.  With 17B inodes on a 8TB MDT that is an bytes-per-inode 
ratio of 497, which is unrealistically low for Lustre since the MDT will always 
stores multiple xattrs on each inode.  Note that the filesystem only has 6 
inodes allocated, so the ZFS total inodes estimate is unrealistically high and 
will get better as more inodes are allocated in the filesystem.

Formating under Lustre 2.10.8

mkfs.lustre --mdt --backfstype=zfs --fsname=lfsarc01 --index=0 
--mgsnid="36.101.92.22@tcp<mailto:36.101.92.22@tcp>" --reformat mdt/mdt

this translates to only 948M inodes on the Lustre FS.

[root@elfsa1m1 ~]# df -i
Filesystem   Inodes  IUsed   IFree IUse% Mounted on
mdt 17678817874  6 176788178681% /mdt
mdt/mdt   948016092263   9480158291% /lfs/lfsarc01/mdt

[root@elfsa1m1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
mdt  8.3T  256K  8.3T   1% /mdt
mdt/mdt  8.2T   24M  8.2T   1% /lfs/lfsarc01/mdt

and there is no reasonable option to provide more file entries except for 
adding another MDT?

The Lustre statfs code will weight in so

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

Recordsize/ahift: in both cases default values were used (but on different 
versions of Lustre). How can I check different values for recordsize/ashift for 
actual values to compare?

zpool mirroring is quite different though – bad drive is a simple raidz2:

  raidz2-0  ONLINE   0 0 0
sdd ONLINE   0 0 0
….

errors: No known data errors

the good drive uses 10 mirrors:

NAME STATE READ WRITE CKSUM
mdt  ONLINE   0 0 0
  mirror-0   ONLINE   0 0 0
sdd  ONLINE   0 0 0
sde  ONLINE   0 0 0
  mirror-1   ONLINE   0 0 0
sdf  ONLINE   0 0 0
sdg  ONLINE   0 0 0
  mirror-2   ONLINE   0 0 0
sdh  ONLINE   0 0 0
sdi  ONLINE   0 0 0
  mirror-3   ONLINE   0 0 0
sdj  ONLINE   0 0 0
sdk  ONLINE   0 0 0
  mirror-4   ONLINE   0 0 0
sdl  ONLINE   0 0 0
sdm  ONLINE   0 0 0
  mirror-5   ONLINE   0 0 0
sdn  ONLINE   0 0 0
sdo  ONLINE   0 0 0
  mirror-6   ONLINE   0 0 0
sdp  ONLINE   0 0 0
sdq  ONLINE   0 0 0
  mirror-7   ONLINE   0 0 0
sdr  ONLINE   0 0 0
sds  ONLINE   0 0 0
  mirror-8   ONLINE   0 0 0
sdt  ONLINE   0 0 0
sdu  ONLINE   0 0 0
  mirror-9   ONLINE   0 0 0
sdv  ONLINE   0 0 0
sdw  ONLINE   0 0 0
  mirror-10  ONLINE   0 0 0
sdx  ONLINE   0 0 0
sdy  ONLINE   0 0 0

thanks
Michael

From: Andreas Dilger 
Sent: Monday, November 11, 2019 14:42
To: Hebenstreit, Michael 
Cc: Mohr Jr, Richard Frank ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

There isn't really enough information to make any kind of real analysis.


My guess would be that you are using a larger ZFS recordsize or ashift on the 
new filesystem, or the RAID config is different?

Cheers, Andreas

On Nov 7, 2019, at 08:45, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:
So we went ahead and used the FS – using rsync to duplicate the existing FS. 
The inodes available on the NEW mdt (which is almost twice the size of the 
second mdt) are dropping rapidly and are now LESS than on the smaller mdt (even 
though the sync is only 90% complete). Both FS are running almost identical 
Lustre 2.10. I cannot say anymore which ZFS version was used to format the good 
FS.

Any ideas why those 2 MDTs behave so differently?

old GOOD FS:
# df -i
mgt/mgt   81718714   205   817185091% /lfs/lfsarc02/mgt
mdt/mdt  458995000 130510339  328484661   29% /lfs/lfsarc02/mdt
# df -h
mgt/mgt  427G  7.0M  427G   1% /lfs/lfsarc02/mgt
mdt/mdt  4.6T  1.4T  3.3T  29% /lfs/lfsarc02/mdt
# rpm -q -a | grep zfs
libzfs2-0.7.9-1.el7.x86_64
lustre-osd-zfs-mount-2.10.4-1.el7.x86_64
lustre-zfs-dkms-2.10.4-1.el7.noarch
zfs-0.7.9-1.el7.x86_64
zfs-dkms-0.7.9-1.el7.noarch

new BAD FS
# df -ih
mgt/mgt83M   169   83M1% /lfs/lfsarc01/mgt
mdt/mdt   297M  122M  175M   42% /lfs/lfsarc01/mdt
# df -h
mgt/mgt  427G  5.8M  427G   1% /lfs/lfsarc01/mgt
mdt/mdt  8.2T  3.4T  4.9T  41% /lfs/lfsarc01/mdt
# rpm -q -a | grep zfs
libzfs2-0.7.9-1.el7.x86_64
lustre-osd-zfs-mount-2.10.8-1.el7.x86_64
lustre-zfs-dkms-2.10.8-1.el7.noarch
zfs-0.7.9-1.el7.x86_64
zfs-dkms-0.7.9-1.el7.noarch

From: Andreas Dilger mailto:adil...@whamcloud.com>>
Sent: Thursday, October 03, 2019 20:38
To: Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>>
Cc: Mohr Jr, Richard Frank mailto:rm...@utk.edu>>; 
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: Re: [lustre-discuss] changing inode size on MDT

On Oct 3, 2019, at 20:09, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:

So bottom line – don’t change the default values, it won’t get better?

Like I wrote previously, there *are* no default/tunable values to change for 
ZFS.  The tunables are only for ldiskfs, which statically allocates everything, 
but is will cause problems if you guessed incorrectly at the instant you format 
the filesystem.

The number reported by raw ZFS and by Lustre-on-ZFS is just an estimate, and 
you will (essentially) run out of inodes once you run out of space on the MDT 
or all OSTs.  And I didn't say "

Re: [lustre-discuss] changing inode size on MDT

[Hebenstreit, Michael]

Thanks Andreas!

Comparing “zpool get all” on both systems I found ashift is 0 on both systems – 
but a number of features are different on the “bad” mdt. Except for “extensible 
dataset” they are all enabled on the “bad” one. Could one of them be the 
problem? Is it possible to change those features and reclaim the space?

“good system”
- mdt  feature@multi_vdev_crash_dump  disabled   local
- mdt  feature@extensible_dataset enabledlocal
- mdt  feature@filesystem_limits  disabled   local
- mdt  feature@large_blocks   disabled   local
- mdt  feature@large_dnodedisabled   local
- mdt  feature@sha512 disabled   local
- mdt  feature@skein  disabled   local
- mdt  feature@edonr  disabled   local
- mdt  feature@userobj_accounting disabled   local

“Bad” system
+ mdt  feature@multi_vdev_crash_dump  enabledlocal
+ mdt  feature@extensible_dataset active local
+ mdt  feature@filesystem_limits  enabledlocal
+ mdt  feature@large_blocks   enabledlocal
+ mdt  feature@large_dnodeenabledlocal
+ mdt  feature@sha512 enabledlocal
+ mdt  feature@skein  enabledlocal
+ mdt  feature@edonr  enabledlocal
+ mdt  feature@userobj_accounting active local

From: Andreas Dilger 
Sent: Monday, November 11, 2019 15:55
To: Hebenstreit, Michael 
Cc: Mohr Jr, Richard Frank ; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] changing inode size on MDT

You can check the ashift of the zpool via "zpool get all | grep ashift".  If 
this is different, it will make a huge difference in space usage. There are a 
number of ZFS articles that discuss this, it isn't specific to Lustre.

Also, RAID-Z2 is going to have much more space overhead for the MDT than 
mirroring, because the MDT is almost entirely small blocks.  Normally the MDT 
is using mirrored VDEVs.

The reason is that RAID-Z2 has two parity sectors per data stripe vs. a single 
extra mirror per data block, so if all data blocks are 4KB that would double 
the parity overhead vs. mirroring. Secondly, depending on the geometry, RAID-Z2 
needs padding sectors to align the variable RAID-Z stripes, which mirrors do 
not.

For large files/blocks RAID-Z2 is better, but that isn't the workload on the 
MDT unless you are storing DoM files there (eg. 64KB or larger).
Cheers, Andreas

On Nov 11, 2019, at 13:48, Hebenstreit, Michael 
mailto:michael.hebenstr...@intel.com>> wrote:
Recordsize/ahift: in both cases default values were used (but on different 
versions of Lustre). How can I check different values for recordsize/ashift for 
actual values to compare?

zpool mirroring is quite different though – bad drive is a simple raidz2:

  raidz2-0  ONLINE   0 0 0
sdd ONLINE   0 0 0
….

errors: No known data errors

the good drive uses 10 mirrors:

NAME STATE READ WRITE CKSUM
mdt  ONLINE   0 0 0
  mirror-0   ONLINE   0 0 0
sdd  ONLINE   0 0 0
sde  ONLINE   0 0 0
  mirror-1   ONLINE   0 0 0
sdf  ONLINE   0 0 0
sdg  ONLINE   0 0 0
  mirror-2   ONLINE   0 0 0
sdh  ONLINE   0 0 0
sdi  ONLINE   0 0 0
  mirror-3   ONLINE   0 0 0
sdj  ONLINE   0 0 0
sdk  ONLINE   0 0 0
  mirror-4   ONLINE   0 0 0
sdl  ONLINE   0 0 0
sdm  ONLINE   0 0 0
  mirror-5   ONLINE   0 0 0
sdn  ONLINE   0 0 0
sdo  ONLINE   0 0 0
  mirror-6   ONLINE   0 0 0
sdp  ONLINE   0 0 0
sdq  ONLINE   0 0 0
  mirror-7   ONLINE   0 0 0
sdr  ONLINE   0 0 0
sds  ONLINE   0 0 0
  mirror-8   ONLINE   0 0 0
sdt  ONLINE   0 0 0
sdu  ONLINE   0 0 0
  mirror-9   ONLINE   0 0 0
sdv  ONLINE   0 0 0
sdw  ONLINE   0 0 0
  mirror-10  ONLINE   0  

[lustre-discuss] Lustre client on RedHat 7.8

[Hebenstreit, Michael]

Trying to compile for new RH kernel I got some errors, I think because of 
https://patchwork.kernel.org/patch/11061873/. My solution was this patch

$ cat  
/admin/work/buildenv_rh7/OFED/mlnx-5.0-2.1.8.0-1127.8.2-2.10.8/extra/gss_svc_upcall.patch
--- lustre/ptlrpc/gss/gss_svc_upcall.c.orig 2020-05-14 15:58:38.0 
-0600
+++ lustre/ptlrpc/gss/gss_svc_upcall.c  2020-05-14 16:12:55.0 -0600
@@ -1175,14 +1175,14 @@
 * drop the request direclty, thus lead to unnecessary recovery time.
 * here we wait at miximum 1.5 seconds. */
for (i = 0; i < 6; i++) {
-   if (atomic_read(&rsi_cache.readers) > 0)
+   if (atomic_read(&rsi_cache.writers) > 0)
break;
set_current_state(TASK_UNINTERRUPTIBLE);
LASSERT(msecs_to_jiffies(MSEC_PER_SEC) >= 4);
schedule_timeout(msecs_to_jiffies(MSEC_PER_SEC / 4));
}

-   if (atomic_read(&rsi_cache.readers) == 0)
+   if (atomic_read(&rsi_cache.writers) == 0)
CWARN("Init channel is not opened by lsvcgssd, following "
  "request might be dropped until lsvcgssd is active\n");


Does that look correct?

Thanks
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] problem after upgrading 2.10.4 to 2.12.4

[Hebenstreit, Michael]

We experienced on our Archive Lustre (ZFS based, 4 OST servers with 6 OSTs 
pools each) the very same issues as described here:
https://jira.whamcloud.com/browse/LU-13392
Certain directories cannot be accessed, and the OSTs shows thousands of errors 
"Can't find FID Sequence". Unfortunately I cannot even start the recommended 
file system checking on the OST devices  - example:
[root@elfsa2o1 ~]# lctl lfsck_start -o -M lfsarc02-OST0002
Fail to start LFSCK: Operation not permitted
[root@elfsa2o1 ~]# lctl lfsck_start -M lfsarc02-OST0002
Fail to start LFSCK: Operation not supported
On a similar system that was first installed as 2.10.4, then upgraded to 
2.10.8, and now is also running on 2.12.4, at least the second command starts:
# lctl lfsck_start -M lfsarc01-OST0002
The commands are issued on the system with the actual ZFS pools running.

Questions:
Is there any way to force the file system checks?
Has anyone found a workaround for the FID sequence errors?
Can I downgrade from 2.12.4 to 2.10.8 without destroying the FS?
Has the error described in https://jira.whamcloud.com/browse/LU-13392 been 
fixed in 
2.12.5?

Thanks
Michael



Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] problem after upgrading 2.10.4 to 2.12.4

[Hebenstreit, Michael]

Another tidbit: the 2 OST nodes showing problems have an lsfsck running and I 
cannot stop it

[root@elfsa2o1 ~]# grep status /proc/fs/lustre/osd-zfs/lfsarc02-OST*/oi_scrub
/proc/fs/lustre/osd-zfs/lfsarc02-OST/oi_scrub:status: completed
/proc/fs/lustre/osd-zfs/lfsarc02-OST0002/oi_scrub:status: scanning
/proc/fs/lustre/osd-zfs/lfsarc02-OST0004/oi_scrub:status: scanning
/proc/fs/lustre/osd-zfs/lfsarc02-OST0006/oi_scrub:status: scanning
/proc/fs/lustre/osd-zfs/lfsarc02-OST0008/oi_scrub:status: scanning
/proc/fs/lustre/osd-zfs/lfsarc02-OST000a/oi_scrub:status: scanning

An lfsck on the MDT hangs, as orphaned inodes cannot be deleted

[ 9568.345851] LustreError: 
6592:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) 
lfsarc02-OST0006-osc-MDT: cannot cleanup orphans: rc = -22
[ 9568.364339] LustreError: 
6592:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 6590 
previous similar messages

Is there any way to stop the scans on the OSTs?

From: Hebenstreit, Michael
Sent: Tuesday, June 23, 2020 11:19
To: lustre-discuss@lists.lustre.org
Subject: problem after upgrading 2.10.4 to 2.12.4

We experienced on our Archive Lustre (ZFS based, 4 OST servers with 6 OSTs 
pools each) the very same issues as described here:

https://jira.whamcloud.com/browse/LU-13392

Certain directories cannot be accessed, and the OSTs shows thousands of errors 
"Can't find FID Sequence". Unfortunately I cannot even start the recommended 
file system checking on the OST devices  - example:

[root@elfsa2o1 ~]# lctl lfsck_start -o -M lfsarc02-OST0002
Fail to start LFSCK: Operation not permitted
[root@elfsa2o1 ~]# lctl lfsck_start -M lfsarc02-OST0002
Fail to start LFSCK: Operation not supported

On a similar system that was first installed as 2.10.4, then upgraded to 
2.10.8, and now is also running on 2.12.4, at least the second command starts:
# lctl lfsck_start -M lfsarc01-OST0002

The commands are issued on the system with the actual ZFS pools running.

Questions:
Is there any way to force the file system checks?
Has anyone found a workaround for the FID sequence errors?
Can I downgrade from 2.12.4 to 2.10.8 without destroying the FS?
Has the error described in https://jira.whamcloud.com/browse/LU-13392 been 
fixed in 
2.12.5<https://jira.whamcloud.com/browse/LU-13392%20been%20fixed%20in%202.12.5>?

Thanks
Michael



Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com<mailto:michael.hebenstr...@intel.com>


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] problem after upgrading 2.10.4 to 2.12.4

[Hebenstreit, Michael]

I would not plan a direct upgrade until Whamcloud fixes the underlying issue. 
Currently the only viable way seem to be a step by step upgrade. I imagine 
you'd first upgrade to 2.10.8, and then copy all old file to a new place 
(something like: mkdir .new_copy; rsync -a  * .new_copy; rm -rf *; mv 
.new_copy/* .; rmdir .new_copy) so that all files have been re-created with 
correct information. Knut's script is a hack and last minute resort. 

-Original Message-
From: lustre-discuss  On Behalf Of 
Patrick Shopbell
Sent: Wednesday, June 24, 2020 12:36
To: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] problem after upgrading 2.10.4 to 2.12.4


Hello all,
I have been following this discussion with interest, as we are in the process 
of a long-overdue upgrade of our small Lustre system. We are moving everything 
from

RHEL 6 + Lustre 2.5.2

to

RHEL 7 + Lustre 2.8.0

We are taking this route merely because 2.8.0 supported both RHEL 6 and 7, and 
so we could keep running, to some extent. (In reality, we have found that v2.8 
clients crash our v2.5 MGS on a pretty regular basis.)

Once our OS upgrades are done, the plan is to then take everything to

RHEL 7 + Lustre 2.12.x

 From what I gather on this thread, however... I should expect to have some 
difficulty reading most of my files, since we have been running 2.5 for a long 
time. And so I should plan on running Knut's 'update_25_objects' on all of my 
OSTs? Is that correct? Should I need to do that at Lustre 2.8.0, or not until I 
get to v2.12? Also, I assume this issue is irrelevant of underlying filesystem 
- we are still running lustrefs on our 12 OSTs, rather than ZFS.

Thanks so much. This list is always very helpful and interesting.
--
Patrick


On 6/24/20 1:16 AM, Franke, Knut wrote:
> Am Dienstag, den 23.06.2020, 20:03 + schrieb Hebenstreit, Michael:
>> Is there any way to stop the scans on the OSTs?
> Yes, by re-mounting them with -o noscrub. This doesn't fix the issue 
> though.
>
>> Is there any way to force the file system checks?
> As shown in your second mail, the scrubs are already running.
> Unfortunately, they don't (as of Lustre 2.12.4) fix the issue.
>
>> Has anyone found a workaround for the FID sequence errors?
> Yes, see the script attached to LU-13392. In short:
>
> 0. Make sure you have a backup. This might eat your lunch and fry your 
> cat for afters.
> 1. Enable the canmount property on the backend filesystem. For example:
> [oss]# zfs set canmount=on mountpoint=/mnt/ostX ${fsname}-ost/ost 
> 2. Mount the target as 'zfs'. For example:
> [oss]# zfs mount ${fsname}-ost/ost 3. update_25_objects /mnt/ostX 
> 4. unmount and remount the OST as 'lustre'
>
> This will rewrite the extended attributes of OST objects created by 
> Lustre 2.4/2.5 to a format compatible with 2.12.
>
>> Can I downgrade from 2.12.4 to 2.10.8 without destroying the FS?
> We've done this successfully, but again - no guarantees.
>
>> Has the error described in https://jira.whamcloud.com/browse/LU-13392
>>   been fixed in 2.12.5?
> I don't think so.
>
> Cheers,
> Knut


-- 

**
| Patrick Shopbell   Department of Astronomy |
| p...@astro.caltech.edu  Mail Code 249-17|
| (626) 395-4097 California Institute of Technology  |
| (626) 568-9352  (FAX)  Pasadena, CA  91125 |
| WWW: http://www.astro.caltech.edu/~pls/|
**

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] problem after upgrading 2.10.4 to 2.12.4

[Hebenstreit, Michael]

Sorry, my Lustre started out as 2.10.(4)? I suspect it was the ZFS integration.

-Original Message-
From: lustre-discuss  On Behalf Of 
Thomas Roth
Sent: Thursday, June 25, 2020 02:24
To: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] problem after upgrading 2.10.4 to 2.12.4

Hi all,

from the discussion here and LU-13392 I take it that this is happening to files 
written under 2.5 ?
Because we are planning to upgrade 2.10.6 -> 2.12, but our Lustre has only ever 
been on 2.10 (.5 at first, I think, now .6). So in theory we should be safe 
here?

Regards,
Thomas

On 24.06.20 20:43, Hebenstreit, Michael wrote:
> I would not plan a direct upgrade until Whamcloud fixes the underlying issue. 
> Currently the only viable way seem to be a step by step upgrade. I imagine 
> you'd first upgrade to 2.10.8, and then copy all old file to a new place 
> (something like: mkdir .new_copy; rsync -a  * .new_copy; rm -rf *; mv 
> .new_copy/* .; rmdir .new_copy) so that all files have been re-created with 
> correct information. Knut's script is a hack and last minute resort. 
> 
> -Original Message-
> From: lustre-discuss  On 
> Behalf Of Patrick Shopbell
> Sent: Wednesday, June 24, 2020 12:36
> To: lustre-discuss@lists.lustre.org
> Subject: Re: [lustre-discuss] problem after upgrading 2.10.4 to 2.12.4
> 
> 
> Hello all,
> I have been following this discussion with interest, as we are in the 
> process of a long-overdue upgrade of our small Lustre system. We are 
> moving everything from
> 
> RHEL 6 + Lustre 2.5.2
> 
> to
> 
> RHEL 7 + Lustre 2.8.0
> 
> We are taking this route merely because 2.8.0 supported both RHEL 6 
> and 7, and so we could keep running, to some extent. (In reality, we 
> have found that v2.8 clients crash our v2.5 MGS on a pretty regular 
> basis.)
> 
> Once our OS upgrades are done, the plan is to then take everything to
> 
> RHEL 7 + Lustre 2.12.x
> 
>  From what I gather on this thread, however... I should expect to have some 
> difficulty reading most of my files, since we have been running 2.5 for a 
> long time. And so I should plan on running Knut's 'update_25_objects' on all 
> of my OSTs? Is that correct? Should I need to do that at Lustre 2.8.0, or not 
> until I get to v2.12? Also, I assume this issue is irrelevant of underlying 
> filesystem - we are still running lustrefs on our 12 OSTs, rather than ZFS.
> 
> Thanks so much. This list is always very helpful and interesting.
> --
> Patrick
> 
> 
> On 6/24/20 1:16 AM, Franke, Knut wrote:
>> Am Dienstag, den 23.06.2020, 20:03 + schrieb Hebenstreit, Michael:
>>> Is there any way to stop the scans on the OSTs?
>> Yes, by re-mounting them with -o noscrub. This doesn't fix the issue 
>> though.
>>
>>> Is there any way to force the file system checks?
>> As shown in your second mail, the scrubs are already running.
>> Unfortunately, they don't (as of Lustre 2.12.4) fix the issue.
>>
>>> Has anyone found a workaround for the FID sequence errors?
>> Yes, see the script attached to LU-13392. In short:
>>
>> 0. Make sure you have a backup. This might eat your lunch and fry 
>> your cat for afters.
>> 1. Enable the canmount property on the backend filesystem. For example:
>> [oss]# zfs set canmount=on mountpoint=/mnt/ostX ${fsname}-ost/ost 
>> 2. Mount the target as 'zfs'. For example:
>> [oss]# zfs mount ${fsname}-ost/ost 3. update_25_objects /mnt/ostX 
>> 4. unmount and remount the OST as 'lustre'
>>
>> This will rewrite the extended attributes of OST objects created by 
>> Lustre 2.4/2.5 to a format compatible with 2.12.
>>
>>> Can I downgrade from 2.12.4 to 2.10.8 without destroying the FS?
>> We've done this successfully, but again - no guarantees.
>>
>>> Has the error described in https://jira.whamcloud.com/browse/LU-13392
>>>   been fixed in 2.12.5?
>> I don't think so.
>>
>> Cheers,
>> Knut
> 
> 

--

Thomas Roth
Department: IT
Location: SB3 2.291
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum für Schwerionenforschung GmbH Planckstraße 1, 64291 
Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528 Managing 
Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock Chairman of 
the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
State Secretary / Staatssekretär Dr. Volkmar Dietz 
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] bios spray passwd change

[Hebenstreit, Michael]

For Intel server motherboards the tool is called "syscfg".

From: lustre-discuss  On Behalf Of 
Einar Næss Jensen
Sent: Thursday, June 25, 2020 06:53
To: Hopper, Edward - CTR ; 
lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] bios spray passwd change


Hello.



Don't know what system you have, but for all our commodity systems (Lenovo, 
Dell, HPE) we have specific vendor tools for that. Usually ipmitool also works.





Best Regards

Einar Næss Jensen




--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technology
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no


From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 on behalf of Hopper, Edward - CTR 
mailto:edward.hop...@rocket.com>>
Sent: Thursday, June 25, 2020 14:15
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] bios spray passwd change


I know this is a bit off topic but since we deal with a huge amount of compute 
nodes

I will shoot to the group.  Looking for a script to change the bios passwords 
all at once.

Any help would be appreciated.



Edward Hopper





Anyone can build a fast CPU. The trick is to build a fast system ~ Seymour Cray


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] build help with 5.15 kernel

[Hebenstreit, Michael via lustre-discuss]

Hello

I'm trying to build the Lustre 2.12.8 client on a 5.15 kernel and already 
failing in the configure step. Looks to me like something in the build process 
has changed. The failure occurs in configure line 14390. From the log:

configure:14390: cp conftest.c build && make -d 
_module_/tmp/lustre-2.12.8/build LUSTRE_KERNEL_TEST=conftest.i LDFLAGS= 
LD=/usr/bin/ld -m elf_x86_64 CC=gcc -f
make: *** No rule to make target '_module_/tmp/lustre-2.12.8/build'.  Stop.
configure:14393: $? = 2

For some reasons the construct "make -d _module_/${PWD} .." does no longer work 
(it works on same OS with 4.18 RH8.3 kernel). If I replace it with something 
like "make -d modules .." It works, modules are build, but the subsequent test 
for ${PWD}/conftest.i fails

Any help appreciated
Happy new year
Michael


Michael Hebenstreit Senior Cluster Architect
Intel Corporation, MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] build help with 5.15 kernel

[Hebenstreit, Michael via lustre-discuss]

Some additional info. I extracted the make command and ran it against the 2 
kernel versions. Old kernel works, new kernel fails

$ make _module_/tmp/lustre-2.12.5/build LUSTRE_KERNEL_TEST=conftest.i LDFLAGS=' 
LD=/usr/bin/ld -m elf_x86_64' CC=gcc   -f /tmp/lustre-2.12.5/build/Makefile 
LUSTRE_LINUX_CONFIG=/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//.config
 
LINUXINCLUDE='-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//arch/x86/include
 -Iinclude -Iarch/x86/include/generated 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include -Iinclude2 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/uapi 
-Iinclude/generated 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//arch/x86/include/uapi 
-Iarch/x86/include/generated/uapi 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/uapi 
-Iinclude/generated/uapi -include 
/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/linux/kconfig.h' 
-o tmp_include_depends -o scripts -o include/config/MARKER -C 
/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib/ 
EXTRA_CFLAGS='-Werror-implicit-function-declaration -g 
-I/tmp/lustre-2.12.5/libcfs/include -I/tmp/lustre-2.12.5/lnet/include 
-I/tmp/lustre-2.12.5/lustre/include/uapi -I/tmp/lustre-2.12.5/lustre/include 
-Wno-format-truncation -Wno-stringop-truncation -Wno-stringop-overflow' 
M=/tmp/lustre-2.12.5/build
make: Entering directory 
'/global/panfs01/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib'
  CC [M]  /tmp/lustre-2.12.5/build/conftest.o
  CPP [M] /tmp/lustre-2.12.5/build/conftest.i
make: Leaving directory 
'/global/panfs01/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib'

$ touch /tmp/lustre-2.12.5/build/conftest.c
$ make _module_/tmp/lustre-2.12.5/build LUSTRE_KERNEL_TEST=conftest.i LDFLAGS=' 
LD=/usr/bin/ld -m elf_x86_64' CC=gcc   -f /tmp/lustre-2.12.5/build/Makefile 
LUSTRE_LINUX_CONFIG=/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/.config
 
LINUXINCLUDE='-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/arch/x86/include
 -Iinclude -Iarch/x86/include/generated 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include -Iinclude2 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/uapi 
-Iinclude/generated 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/arch/x86/include/uapi
 -Iarch/x86/include/generated/uapi 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/uapi 
-Iinclude/generated/uapi -include 
/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/linux/kconfig.h'
 -o tmp_include_depends -o scripts -o include/config/MARKER -C 
/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64 
EXTRA_CFLAGS='-Werror-implicit-function-declaration -g 
-I/tmp/lustre-2.12.5/libcfs/include -I/tmp/lustre-2.12.5/lnet/include 
-I/tmp/lustre-2.12.5/lustre/include/uapi -I/tmp/lustre-2.12.5/lustre/include 
-Wno-format-truncation -Wno-stringop-truncation -Wno-stringop-overflow' 
M=/tmp/lustre-2.12.5/build
make: Entering directory 
'/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64'
make: *** No rule to make target '_module_/tmp/lustre-2.12.5/build'.  Stop.
make: Leaving directory 
'/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64'

-Original Message-
From: lustre-discuss  On Behalf Of 
Hebenstreit, Michael via lustre-discuss
Sent: Thursday, December 30, 2021 5:07 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] build help with 5.15 kernel

Hello

I'm trying to build the Lustre 2.12.8 client on a 5.15 kernel and already 
failing in the configure step. Looks to me like something in the build process 
has changed. The failure occurs in configure line 14390. From the log:

configure:14390: cp conftest.c build && make -d 
_module_/tmp/lustre-2.12.8/build LUSTRE_KERNEL_TEST=conftest.i LDFLAGS= 
LD=/usr/bin/ld -m elf_x86_64 CC=gcc -f
make: *** No rule to make target '_module_/tmp/lustre-2.12.8/build'.  Stop.
configure:14393: $? = 2

For some reasons the construct "make -d _module_/${PWD} .." does no longer work 
(it works on same OS with 4.18 RH8.3 kernel). If I replace it with something 
like "make -d modules .." It works, modules are build, but the subsequent test 
for ${PWD}/conftest.i fails

Any help appreciated
Happy new year
Michael


Michael Hebenstreit Senior Cluster Architect Intel Corporation, 
MS: RR1-105/H14  TSACG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144 
Rio Rancho, NM 87124
UNITED STATES   E-mail: michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] build help with 5.15 kernel

[Hebenstreit, Michael via lustre-discuss]

Thanks for the pointers.

-Original Message-
From: Degremont, Aurelien  
Sent: Monday, January 3, 2022 3:35 AM
To: Hebenstreit, Michael ; 
lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] build help with 5.15 kernel

Hello Michael

Lustre 2.12.8 does not support Linux 5.15 More recent Lustre versions support 
up to Linux 5.11 but not further.
See these tickets for 5.12 and 5.14 support.
https://jira.whamcloud.com/browse/LU-14651
https://jira.whamcloud.com/browse/LU-15220

It is possible to manually backport patches to support some 5.x kernels with 
2.12 but this is not trivial. 

I don't know what is your current project but it will be much easier for you if 
you can target an older Linux kernel and focus on kernel used by major distro. 
Ubuntu 20.04 is using 5.11 by example.


Aurélien


Le 31/12/2021 01:34, « lustre-discuss au nom de Hebenstreit, Michael via 
lustre-discuss »  a écrit :

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



Some additional info. I extracted the make command and ran it against the 2 
kernel versions. Old kernel works, new kernel fails

$ make _module_/tmp/lustre-2.12.5/build LUSTRE_KERNEL_TEST=conftest.i 
LDFLAGS=' LD=/usr/bin/ld -m elf_x86_64' CC=gcc   -f 
/tmp/lustre-2.12.5/build/Makefile 
LUSTRE_LINUX_CONFIG=/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//.config
 
LINUXINCLUDE='-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//arch/x86/include
 -Iinclude -Iarch/x86/include/generated 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include -Iinclude2 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/uapi 
-Iinclude/generated 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//arch/x86/include/uapi 
-Iarch/x86/include/generated/uapi 
-I/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/uapi 
-Iinclude/generated/uapi -include 
/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib//include/linux/kconfig.h' 
-o tmp_include_depends -o scripts -o include/config/MARKER -C 
/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib/ 
EXTRA_CFLAGS='-Werror-implicit-function-declaration -g 
-I/tmp/lustre-2.12.5/libcfs/include -I/tmp/lustre-2.12.5/lnet/include 
-I/tmp/lustre-2.12.5/lustre/include/uapi -I/tmp/lustre-2.12.5/lustre/include 
-Wno-format-truncation -Wno-stringop-truncation -Wno-stringop-overflow' 
M=/tmp/lustre-2.12.5/build
make: Entering directory 
'/global/panfs01/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib'
  CC [M]  /tmp/lustre-2.12.5/build/conftest.o
  CPP [M] /tmp/lustre-2.12.5/build/conftest.i
make: Leaving directory 
'/global/panfs01/admin/src/4.18.0-240.22.1.el8_3.crt6.x86_64.withib'

$ touch /tmp/lustre-2.12.5/build/conftest.c
$ make _module_/tmp/lustre-2.12.5/build LUSTRE_KERNEL_TEST=conftest.i 
LDFLAGS=' LD=/usr/bin/ld -m elf_x86_64' CC=gcc   -f 
/tmp/lustre-2.12.5/build/Makefile 
LUSTRE_LINUX_CONFIG=/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/.config
 
LINUXINCLUDE='-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/arch/x86/include
 -Iinclude -Iarch/x86/include/generated 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include -Iinclude2 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/uapi 
-Iinclude/generated 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/arch/x86/include/uapi
 -Iarch/x86/include/generated/uapi 
-I/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/uapi 
-Iinclude/generated/uapi -include 
/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64/include/linux/kconfig.h'
 -o tmp_include_depends -o scripts -o include/config/MARKER -C 
/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64 
EXTRA_CFLAGS='-Werror-implicit-function-declaration -g 
-I/tmp/lustre-2.12.5/libcfs/include -I/tmp/lustre-2.12.5/lnet/include 
-I/tmp/lustre-2.12.5/lustre/include/uapi -I/tmp/lustre-2.12.5/lustre/include 
-Wno-format-truncation -Wno-stringop-truncation -Wno-stringop-overflow' 
M=/tmp/lustre-2.12.5/build
make: Entering directory 
'/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64'
make: *** No rule to make target '_module_/tmp/lustre-2.12.5/build'.  Stop.
make: Leaving directory 
'/global/panfs01/admin/src/5.15.0-spr.bkc.pc.1.21.0.x86_64'

-Original Message-
From: lustre-discuss  On Behalf Of 
Hebenstreit, Michael via lustre-discuss
Sent: Thursday, December 30, 2021 5:07 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] build help with 5.15 kernel

Hello

I'm trying to build the Lustre 2.12.8 client on a 5.15 kernel and already 
failing in the configure step. Looks to me like something in the build process 
has changed. The failure occurs in configure line 14390.

[lustre-discuss] OST failure

[Hebenstreit, Michael via lustre-discuss]

We had an OST failure. Is there a way to


  1.  Make the whole Lustre FS read only?
  2.  Take the failing OST offline to try and recover what's possible?

Thanks
Michael


Michael Hebenstreit Senior Cluster Application Engineer
Intel Corporation, MS: RR1-105/H14  AXG
1600 Rio Rancho Blvd SE Tel.:   +1 505-794-3144
Rio Rancho, NM 87124
UNITED STATES   E-mail: 
michael.hebenstr...@intel.com


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org