If re-mounting lustre temporarily improves GET_INFO_FS, you may be hitting client cache or parameter limit. A few suggestions to try on the lustre client: - check ulimit settings before starting robinhood, especially number of open files & file locks - set sysctl vfs_cache pressure for more aggressive reclaim of dentries & inodes - on your lustre client, check mdc parameters: max_rpcs_in_flight, lru_size & lru_max_age. FYI, its common to tune the lustre osc parameters but rbh is more sensitive to mdc parameters
regards, chris hunter yale hpc group > Date: Wed, 6 May 2015 15:34:57 +0200 > From: "Carmelo Ponti (CSCS)" <[email protected]> > Subject: Re: [robinhood-support] Robinhood: Changelog read speed > expected > To: LEIBOVICI Thomas <[email protected]> > Cc: "[email protected]" > <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset="utf-8" > > Thomas > > >> try increasing EntryProcessor :: nb_threads to 12, 16... > > > I tried it already and doesn't help much. The only trick I found (by > chance and I don't know the real reason) to reduce GET_INFO_FS is to > stop robinhood, umount and mount lustre again and restart robinhood. I > did it and this is the result: > > Stage | Wait | Curr | Done | Total | ms/op | > 0: GET_FID | 0 | 0 | 0 | 0 | 0.00 | > 1: GET_INFO_DB |99997 | 3 | 0 | 1370749 | 0.25 | > 2: GET_INFO_FS | 0 | 0 | 0 | 501077 | 0.18 | > 3: REPORTING | 0 | 0 | 0 | 2520 | 0.00 | > 4: PRE_APPLY | 0 | 0 | 0 | 374336 | 0.00 | > 5: DB_APPLY | 0 | 0 | 0 | 374336 | 0.18 | 0.01% > batched (avg batch size: 2.0) > 6: CHGLOG_CLR | 0 | 0 | 0 | 1022160 | 0.03 | > 7: RM_OLD_ENTRIES | 0 | 0 | 0 | 0 | 0.00 | > > 2015/05/06 15:24:07 robinhood@daintrbh01[22301/1] STATS | read speed > = 3288.12 record/sec > 2015/05/06 15:25:07 robinhood@daintrbh01[22301/1] STATS | read speed > = 2289.67 record/sec > 2015/05/06 15:26:07 robinhood@daintrbh01[22301/1] STATS | read speed > = 1683.97 record/sec > 2015/05/06 15:27:08 robinhood@daintrbh01[22301/1] STATS | read speed > = 1745.38 record/sec > 2015/05/06 15:28:08 robinhood@daintrbh01[22301/1] STATS | read speed > = 2177.25 record/sec > 2015/05/06 15:29:08 robinhood@daintrbh01[22301/1] STATS | read speed > = 2171.17 record/sec > 2015/05/06 15:30:08 robinhood@daintrbh01[22301/1] STATS | read speed > = 2260.57 record/sec > 2015/05/06 15:31:08 robinhood@daintrbh01[22301/1] STATS | read speed > = 2277.25 record/sec > > Unfortunately this is only temporary (~ 1 day). > > Thank you very much for all. > Carmelo > > -- ---------------------------------------------------------------------- > Carmelo Ponti System Engineer CSCS Swiss Center for Scientific Computing Via > Trevano 131 Email: [email protected] CH-6900 Lugano > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cscs.ch&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=d_G2h_sZYG4xtHMeKo8QgjDmOcMVdQvYgM-5Dri1AOY&m=NpFy8pB3-DaPD4DunoW-PookT_N7YKWyr3RaaO_nREg&s=iJVDHgDgn0UDaGPn6yWtShvCJTh1B_n6dNirdTd6Z3c&e= > Phone: +41 91 610 82 15/Fax: +41 91 610 82 82 ------- ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
