Re: [lustre-discuss] Lustre rpm install creating a file that breaks lustre

[Andreas Dilger]

The "alias ko2iblnd-opa" line in ko2iblnd.conf doesn't do anything unless OPA 
(or older QLogic) interfaces are detected on your system via the 
/usr/sbin/ko2iblnd-probe script.

This indirection is used to allow better default module parameters between OPA 
and MLX devices, which can't easily be determined inside the kernel, and would 
be hard to change after the fact.  In theory, if there were substantially 
better ko2iblnd module parameters for new MLX or RoCE devices, then this same 
mechanism could be used to do similar interface-specific tunings in userspace.

You could run that script with the last line commented out (or replace 
"exec->echo" on the last line) to see what it is doing.

Cheers, Andreas

On Oct 2, 2019, at 12:55, Kurt Strosahl 
mailto:stros...@jlab.org>> wrote:



Good Afternoon,



While getting lustre 2.10.8 running on a RHEL 7.7 system I found that the 
RPM install was putting a file in /etc/modprobe.d that was preventing lnet from 
starting properly.



the file is ko2iblnd.conf, which contains the following...



alias ko2iblnd-opa ko2iblnd
options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 
concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 
fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4



install ko2iblnd /usr/sbin/ko2iblnd-probe



Our system is running infiniband, not omnipath.  So I'm mot sure why this file 
is being put in place.  Removing the file allows lnet to start properly.

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud






___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lustre on SLES15

[Cory Spitz]

Hello.

You’ll have to compile it yourself.  SLES 15 isn’t included in the matrix scope 
for L2.13.0 (http://wiki.lustre.org/Release_2.13.0).  Maybe it’ll be included 
for L2.14.0.

-Cory

--


On 9/27/19, 6:49 AM, "lustre-discuss on behalf of Tauferner, Andrew T" 
mailto:lustre-discuss-boun...@lists.lustre.org>
 on behalf of 
andrew.t.taufer...@intel.com> wrote:

When will Lustre RPMs be available for SLES15 and SLES15 SP1?  Thank you.

Andrew Tauferner

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Limit to number of OSS?

[Degremont, Aurelien]

Thanks for this info. But actually I was really looking at the number of OSS, 
not OSTs :)
This is really more how Lustre client nodes and MDT will cope with very large 
number of OSSes.

De : Andreas Dilger 
Date : vendredi 4 octobre 2019 à 04:54
À : "Degremont, Aurelien" 
Cc : "lustre-discuss@lists.lustre.org" 
Objet : Re: [lustre-discuss] Limit to number of OSS?

On Oct 3, 2019, at 07:55, Degremont, Aurelien 
mailto:degre...@amazon.com>> wrote:

Hello all!

This doc from the wiki says "Lustre can support up to 2000 OSS per file system" 
(http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines).

I'm a bit surprised by this statement. Does somebody has information about the 
upper limit to the number of OSSes?
Or what could be the scaling limitator for this number of OSS? Network limit? 
Memory consumption? Other?

That's likely a combination of a bit of confusion and a bit of safety on the 
part of Intel writing that document.

The Lustre Operations Manual writes:
Although a single file can only be striped over 2000 objects, Lustre file 
systems can have thousands of OSTs. The I/O bandwidth to access a single file 
is the aggregated I/O bandwidth to the objects in a file, which can be as much 
as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, 
clients can do I/O using multiple files to utilize the full file system 
bandwidth.
I think PNNL once tested up to 4000 OSTs, and I think the compile-time limit 
is/was 8000 OSTs (maybe it was made dynamic, I don't recall offhand), but the 
current code could _probably_ handle up to 65000 OSTs without significant 
problems.  Beyond that, there is the 16-bit OST index limit in the filesystem 
device labels and the __u16 lov_user_md_v1->lmm_stripe_offset to specify the 
starting OST index for "lfs setstripe", but that could be overcome with some 
changes.

Given OSTs are starting to approach 1PB with large drives and 
declustered-parity RAID, this would get us in the range 8-65EB, which is over 
2^64 bytes (16EB), so I don't think it is an immediate concern.  Let me know if 
you have any trouble with a 9000-OST filesystem... :-)

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org