[lustre-discuss] overstriping
Just a note that the man page for lfs setstripe seems a bit misleading for the -C --overstripe-count option. The man page states "creating > 1 stripe per OST if count exceeds the number of OSTs in the file system". It appears that lustre creates more than one stripe on an OST if the number of stripes exceeds the number of OSTs in the pool being used. In my example below I selected a pool with 8 OSTs, and striped 16 wide. The file system has 64 OSTs. 16 is less than 64, but I still got 2 stripes per OST. -C, --overstripe-count The number of stripes to create, creating > 1 stripe per OST if count exceeds the number of OSTs in the file system. 0 means to use the filesystem-wide default stripe count (default 1), and -1 means to stripe over all available OSTs. pfe22.jbauer2 908> lfs setstripe --pool 1_hdd -C 16 this.dat pfe22.jbauer2 909> lfs getstripe this.dat this.dat lmm_stripe_count: 16 lmm_stripe_size: 1048576 lmm_pattern: raid0,overstriped lmm_layout_gen: 0 lmm_stripe_offset: 7 lmm_pool: 1_hdd obdidx objid objid group 7 23665469 0x1691b3d 0 0 22497141 0x1574775 0 4 23541897 0x1673889 0 2 23308778 0x163a9ea 0 1 23033000 0x15f74a8 0 6 23687573 0x1697195 0 5 23635915 0x168a7cb 0 3 23446533 0x165c405 0 7 23665470 0x1691b3e 0 0 22497142 0x1574776 0 4 23541898 0x167388a 0 2 23308779 0x163a9eb 0 1 23033001 0x15f74a9 0 6 23687574 0x1697196 0 5 23635916 0x168a7cc 0 3 23446534 0x165c406 0 ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] llapi documentation
While sitting stuck in traffic in a torrential downpour I figured out 3 of my 4 questions. The only remain question: Can something be done about the prints coming from the llapi_get_pool...() functions? llapi_get_poollist() generates a print of:*Pools from nbp17:* llapi_get_poolmembers() generates a print of:*Pool: nbp17.1_ssd* John On 6/15/2022 7:53 AM, lustre-discuss-requ...@lists.lustre.org wrote: Send lustre-discuss mailing list submissions to lustre-discuss@lists.lustre.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org or, via email, send a message with subject or body 'help' to lustre-discuss-requ...@lists.lustre.org You can reach the person managing the list at lustre-discuss-ow...@lists.lustre.org When replying, please edit your Subject line so it is more specific than "Re: Contents of lustre-discuss digest..." Today's Topics: 1. Re: llapi documentation (Andreas Dilger) 2. Re: llapi documentation (John Bauer) -- Message: 1 Date: Wed, 15 Jun 2022 07:08:58 + From: Andreas Dilger To: John Bauer Cc:"lustre-discuss@lists.lustre.org" Subject: Re: [lustre-discuss] llapi documentation Message-ID:<19c08e40-b54a-485d-99e8-925886608...@ddn.com> Content-Type: text/plain; charset="us-ascii" On Jun 14, 2022, at 05:32, John Bauer mailto:bau...@iodoctors.com>> wrote: I have had little success in my search for documentation on pool functions in llapi. I've looked in: https://wiki.lustre.org/PFL2_High_Level_Design https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace I'm looking for info on llapi_get_poollist() and llapi_get_poolmembers(). I've found the prototype in /usr/include/lustre/lustreapi.h, but that's about it. Can anyone point me to some documentation? it looks like there aren't man pages for these functions, just the function comment blocks in the code, and their usage internally: /** * Get the list of pools in a filesystem. * \param namefilesystem name or path * \param poollistcaller-allocated array of char* * \param list_size size of the poollist array * \param buffer caller-allocated buffer for storing pool names * \param buffer_size size of the buffer * * \return number of pools retrieved for this filesystem * \retval -error failure */ int llapi_get_poollist(const char *name, char **poollist, int list_size, char *buffer, int buffer_size) /** * Get the list of pool members. * \param poolnamestring of format \.\ * \param members caller-allocated array of char* * \param list_size size of the members array * \param buffer caller-allocated buffer for storing OST names * \param buffer_size size of the buffer * * \return number of members retrieved for this pool * \retval -error failure */ int llapi_get_poolmembers(const char *poolname, char **members, int list_size, char *buffer, int buffer_size) Patches to add llapi_get_poollist.3 and llapi_get_poolmembers.3 (and related) man pages welcome. The pool related functions should probably be moved into a new liblustreapi_pool.c file to reduce the size of liblustreapi.c. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud -- next part -- An HTML attachment was scrubbed... URL:<http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220615/ef3c7324/attachment-0001.htm> -- Message: 2 Date: Wed, 15 Jun 2022 07:56:40 -0500 From: John Bauer To: Andreas Dilger Cc:"lustre-discuss@lists.lustre.org" Subject: Re: [lustre-discuss] llapi documentation Message-ID:<8b437470-1f49-1535-94f2-8492956a4...@iodoctors.com> Content-Type: text/plain; charset="utf-8"; Format="flowed" Andreas, Thanks for the info.? It got me a lot farther down the road. A few comments: 1)? It appears that the values returned in the poollist argument to llapi_get_poollist() are temporary.? I used the values in poollist in the call to llapi_get_poolmembers( poollist[i],...). Works great on the first call to get_poolmembers(), but subsequent calls fail and it appears the poollist has been overwritten.? If I make my own copy of the strings in poollist before calling get_poolmembers, everythings works well. If this is indeed the case, it should be noted. 2)? There are prints to stdout or stderr resulting from calls to these llapi functions. llapi_get_poollist generates a print of:Pools from nbp17: llapi_get_poolmembers generates a print of:Pool: nbp17.1_ssd Seems undesirable to have llapi doing unrequested prints. 3)? What is the buffer argument good for ( last argument for each of the functions )?? It appears to be populated with the f
[lustre-discuss] unclear language in Operations manual
Hi folks, I've recently come across this snippet in the ops manual (section 13.8. Running Multiple Lustre File Systems, page 111 in the current pdf) > Note > If a client(s) will be mounted on several file systems, add the following > line to /etc/ xattr.conf file to avoid problems when files are moved between > the file systems: lustre.* skip Is this describing the case where a single client mounts more than one lustre filesystem simultaneously? ie mount -t lustre mgsnode:/foo /mnt/foo AND mount -t lustre mgsnode:/bar /mnt/bar? I suspect I should file a LUDOC if so, as the language doesn't flow. As (ahem) we've never done this, what's it likely to screw up when a user's copying from /mnt/foo to /mnt/bar ? Many thanks Andrew ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] llapi documentation
Andreas, Thanks for the info. It got me a lot farther down the road. A few comments: 1) It appears that the values returned in the poollist argument to llapi_get_poollist() are temporary. I used the values in poollist in the call to llapi_get_poolmembers( poollist[i],...). Works great on the first call to get_poolmembers(), but subsequent calls fail and it appears the poollist has been overwritten. If I make my own copy of the strings in poollist before calling get_poolmembers, everythings works well. If this is indeed the case, it should be noted. 2) There are prints to stdout or stderr resulting from calls to these llapi functions. llapi_get_poollist generates a print of:Pools from nbp17: llapi_get_poolmembers generates a print of:Pool: nbp17.1_ssd Seems undesirable to have llapi doing unrequested prints. 3) What is the buffer argument good for ( last argument for each of the functions )? It appears to be populated with the first value in in the array. 4) Here is output from my test program. There are 10 poolNames. All works great until I get to the pool that has 64 members ( which I know from other paths ). The 2nd argument to llapi_get_poolmembers() is a 'char *poolMembers[256]'. Plenty of room, but the call fails with a return value of -75. Why? Pools from nbp17: poollist_get() nPool=10 buffer=nbp17.1_hdd Pool: nbp17.1_hdd poollist_get() poolNames[0]=nbp17.1_hdd 8 members buffer=nbp17-OST_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[0]=nbp17-OST_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[1]=nbp17-OST0001_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[2]=nbp17-OST0002_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[3]=nbp17-OST0003_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[4]=nbp17-OST0004_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[5]=nbp17-OST0005_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[6]=nbp17-OST0006_UUID poollist_get() poolNames[0]=nbp17.1_hdd members[7]=nbp17-OST0007_UUID Pool: nbp17.1_ssd poollist_get() poolNames[1]=nbp17.1_ssd 8 members buffer=nbp17-OST0064_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[0]=nbp17-OST0064_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[1]=nbp17-OST0065_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[2]=nbp17-OST0066_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[3]=nbp17-OST0067_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[4]=nbp17-OST0068_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[5]=nbp17-OST0069_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[6]=nbp17-OST006a_UUID poollist_get() poolNames[1]=nbp17.1_ssd members[7]=nbp17-OST006b_UUID Pool: nbp17.2_hdd poollist_get() poolNames[2]=nbp17.2_hdd 8 members buffer=nbp17-OST0008_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[0]=nbp17-OST0008_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[1]=nbp17-OST0009_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[2]=nbp17-OST000a_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[3]=nbp17-OST000b_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[4]=nbp17-OST000c_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[5]=nbp17-OST000d_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[6]=nbp17-OST000e_UUID poollist_get() poolNames[2]=nbp17.2_hdd members[7]=nbp17-OST000f_UUID Pool: nbp17.2_ssd poollist_get() poolNames[3]=nbp17.2_ssd 8 members buffer=nbp17-OST006c_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[0]=nbp17-OST006c_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[1]=nbp17-OST006d_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[2]=nbp17-OST006e_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[3]=nbp17-OST006f_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[4]=nbp17-OST0070_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[5]=nbp17-OST0071_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[6]=nbp17-OST0072_UUID poollist_get() poolNames[3]=nbp17.2_ssd members[7]=nbp17-OST0073_UUID Pool: nbp17.3_hdd poollist_get() poolNames[4]=nbp17.3_hdd 8 members buffer=nbp17-OST0010_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[0]=nbp17-OST0010_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[1]=nbp17-OST0011_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[2]=nbp17-OST0012_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[3]=nbp17-OST0013_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[4]=nbp17-OST0014_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[5]=nbp17-OST0015_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[6]=nbp17-OST0016_UUID poollist_get() poolNames[4]=nbp17.3_hdd members[7]=nbp17-OST0017_UUID Pool: nbp17.3_ssd poollist_get() poolNames[5]=nbp17.3_ssd 8 members buffer=nbp17-OST0074_UUID poollist_get()
Re: [lustre-discuss] llapi documentation
On Jun 14, 2022, at 05:32, John Bauer mailto:bau...@iodoctors.com>> wrote: I have had little success in my search for documentation on pool functions in llapi. I've looked in: https://wiki.lustre.org/PFL2_High_Level_Design https://doc.lustre.org/lustre_manual.xhtml#managingstripingfreespace I'm looking for info on llapi_get_poollist() and llapi_get_poolmembers(). I've found the prototype in /usr/include/lustre/lustreapi.h, but that's about it. Can anyone point me to some documentation? it looks like there aren't man pages for these functions, just the function comment blocks in the code, and their usage internally: /** * Get the list of pools in a filesystem. * \param namefilesystem name or path * \param poollistcaller-allocated array of char* * \param list_size size of the poollist array * \param buffer caller-allocated buffer for storing pool names * \param buffer_size size of the buffer * * \return number of pools retrieved for this filesystem * \retval -error failure */ int llapi_get_poollist(const char *name, char **poollist, int list_size, char *buffer, int buffer_size) /** * Get the list of pool members. * \param poolnamestring of format \.\ * \param members caller-allocated array of char* * \param list_size size of the members array * \param buffer caller-allocated buffer for storing OST names * \param buffer_size size of the buffer * * \return number of members retrieved for this pool * \retval -error failure */ int llapi_get_poolmembers(const char *poolname, char **members, int list_size, char *buffer, int buffer_size) Patches to add llapi_get_poollist.3 and llapi_get_poolmembers.3 (and related) man pages welcome. The pool related functions should probably be moved into a new liblustreapi_pool.c file to reduce the size of liblustreapi.c. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
