Hi Kit, Thanks much! This is super helpful, I just tested your patch (on master) and it seems to work as expected, at least with map_mode = gid_only.
I will do more tests on this system and let you know if I notice any issues. Thanks again, Stephane > On Nov 18, 2016, at 7:52 AM, Kit Westneat <kit.westn...@gmail.com> wrote: > > Hi Stephane, > > I wrote a patch that adds a couple of flags to allow GID only mapping: > http://review.whamcloud.com/23853 > > How to activate it: > lctl nodemap_modify --name my_nodemap --property map_mode --value gid_only > > > > On Mon, Nov 7, 2016 at 5:59 PM, Stephane Thiell <sthi...@stanford.edu> wrote: > > > On Nov 4, 2016, at 11:05 PM, Dilger, Andreas <andreas.dil...@intel.com> > > wrote: > > > > Actually, the nodemap feature will work with any client, since it is only > > affecting lookups on the MDS and quota on the OSS. > > Great! :-) > > > > > It probably would take less time for you to implement the flag feature than > > the time it is taking to create the thousands of UID entries. While I think > > it should scale very large, I don't think we have tested the 1M or so > > entries you are creating. The good news is that since this is using a hash > > table on the server it shouldn't hurt performance too much. > > > > Let us know how many you finally create, and how it is working with so many > > entries. > > I was able to add ~820k idmaps in 48h [with all servers up and running, see > my note below]… but I can’t go much beyond that point, as I am seeing less > than 1 idmap being added per second. Even with such a low rate, I noticed a > cpu-bound thread on all MDS and OSS: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 30258 root 20 0 0 0 0 R 100.0 0.0 1798:38 > ll_cfg_requeue > > I have two vmcore dumps that show backtraces for this running thread that can > be found at the end of this email. > > I don’t know why the nodemap_idx_insert is called so often, as the apparent > idmap creation rate is very slow. > > Last minute note: I was able to quickly add a more idmaps from the MGS much > faster by disconnecting the lustre servers first. Once the idmaps were added > to the MGS, I started all MDS and OSS servers again, and the idmap > synchronization from the MGS was immediate. So for large uid ranges, I would > recommend to add all the idmaps with only the MGS up and running. That can be > very useful in case of writeconf until a uid range-based or identity mapping > solution is available. > > Also, to finish with a good news, I just completed a few small scale metadata > benchmarks using files with mapped uig/gid, but I didn't notice any > difference in terms of md performance like create or delete/sec rates when > the nodemap feature is enable or disabled. Tested with 820k loaded idmaps. > > All the best, > Stephane > > > > bt running ll_cfg_requeue #1: > --- <NMI exception stack> --- > #4 [ffff881f3e93fae8] memset at ffffffff813017d9 > #5 [ffff881f3e93fae8] kmem_cache_alloc_trace at ffffffff811c117e > #6 [ffff881f3e93fb30] lu_context_init at ffffffffa0a72d06 [obdclass] > #7 [ffff881f3e93fb50] osd_trans_start at ffffffffa0de0151 [osd_ldiskfs] > #8 [ffff881f3e93fb88] nodemap_idx_insert at ffffffffa0f28af4 [ptlrpc] > #9 [ffff881f3e93fbd0] nodemap_save_config_cache at ffffffffa0f2c5e0 [ptlrpc] > #10 [ffff881f3e93fc78] nodemap_config_set_active_mgc at ffffffffa0f2c9ad > [ptlrpc] > #11 [ffff881f3e93fce0] mgc_process_recover_nodemap_log at ffffffffa0a0ce6b > [mgc] > #12 [ffff881f3e93fd70] mgc_process_log at ffffffffa0a0f894 [mgc] > #13 [ffff881f3e93fe30] mgc_requeue_thread at ffffffffa0a11908 [mgc] > #14 [ffff881f3e93fec8] kthread at ffffffff810a5aef > #15 [ffff881f3e93ff50] ret_from_fork at ffffffff81645a58 > > bt running ll_cfg_requeue #2: > --- <NMI exception stack> --- > #4 [ffff883f5f0239a8] iam_it_init at ffffffffa1168563 [osd_ldiskfs] > #5 [ffff883f5f0239b0] iam_insert at ffffffffa116a2f3 [osd_ldiskfs] > #6 [ffff883f5f023b20] osd_index_iam_insert at ffffffffa1157027 [osd_ldiskfs] > #7 [ffff883f5f023b88] nodemap_idx_insert at ffffffffa0d41c0c [ptlrpc] > #8 [ffff883f5f023bd0] nodemap_save_config_cache at ffffffffa0d455e0 [ptlrpc] > #9 [ffff883f5f023c78] nodemap_config_set_active_mgc at ffffffffa0d459ad > [ptlrpc] > #10 [ffff883f5f023ce0] mgc_process_recover_nodemap_log at ffffffffa0a30e6b > [mgc] > #11 [ffff883f5f023d70] mgc_process_log at ffffffffa0a33894 [mgc] > #12 [ffff883f5f023e30] mgc_requeue_thread at ffffffffa0a35908 [mgc] > #13 [ffff883f5f023ec8] kthread at ffffffff810a5aef > #14 [ffff883f5f023f50] ret_from_fork at ffffffff81645a58 > > > _______________________________________________ > Iudev mailing list > iu...@lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/iudev-opensfs.org > _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org