Hi Chris and Thomas for completeness I can tell you that since my last email (~ mount ago) robinhood is working well. I cannot tell you if this it's the result of the tunning you suggested (now vfs_cache_pressure is 125) or if our users didn't stress the file systems anymore (it's not so usual somebody create or remove million of files in few hours), but now I'm getting a very good read speed average (~ 10000 record/sec).
Thank you again, Carmelo On Mon, 2015-05-11 at 18:36 +0200, Carmelo Ponti (CSCS) wrote: > Hi Chris > > I read something about the dangers to use vm.drop_caches so I was in > doubt. During the weekend our users didn't stress the system, the number > of changelog went down to ca. 0 and for this reason I cannot give you > any useful feedback about last settings. I took advantage of the fact > thee machine was empty to perform a "rbh-config optimize_db" which gave > me some benefits too (now the read speed varies between 2500 to 10000 > record/sec). Anyway I'm waiting users start stressing the machine again > to see what it will happen. I will let you know. > > regards > Carmelo > > On Mon, 2015-05-11 at 11:06 -0400, Chris Hunter wrote: > > Hi Carmelo, > > > > Glad you to hear you are making progress. > > vm.drop_caches is disruptive, it will likely dump the cache for your > > local filesystem, mysql buffer, etc. Too aggressive vfs_cache_pressure > > values can cause similar problems. > > I found running the lru_size=clear command hourly gave about the same > > results as setting lru_max_age to one hour, YMMV. > > > > regards, > > chris hunter > > yale hpc group > > > > On 05/08/2015 10:47 AM, Carmelo Ponti (CSCS) wrote: > > > Chris > > > > > > On Thu, 2015-05-07 at 11:56 -0400, Chris Hunter wrote: > > >> Hi Carmelo, > > >> > > >> What I found is once the local vfs cache is full, robinhood GET_INFO_FS > > >> speed drops. You can monitor (via /proc/meminfo) buffer, cached & slab > > >> memory usage when starting robinhood, you will see these values reach > > >> threshold after a time. Once this state is reached, parameters > > >> lru_max_age and vfs_cache_pressure are important. > > > > > > I see your point now. I increased vfs_cache_pressure to 125 and I > > > monitoring buffer, cached & slab as you suggested. I will check on > > > Monday and let you know. > > > > > >> You can also "flush" lustre cached ldlm locks by echoing "clear" to > > >> lru_size parameter. > > > > > > I scheduled this every hour in cron. > > > In addition on > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.kernel.org_doc_Documentation_sysctl_vm.txt&d=AwICaQ&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=d_G2h_sZYG4xtHMeKo8QgjDmOcMVdQvYgM-5Dri1AOY&m=2e-uiK2hYQaNO4gSQJmS-sbkcCSotxt3gGa7oYx5vds&s=j5zzS6pU4jPUsa4HvMfpFTrBqUXcffJvk08rLyK0sNg&e= > > > I'm reading about /proc/sys/vm/drop_caches. Do you think it could help > > > dropping slab objects and pagecache? (echo 3 > > >> /proc/sys/vm/drop_caches). > > > > > > Carmelo > > > > -- ---------------------------------------------------------------------- Carmelo Ponti System Engineer CSCS Swiss Center for Scientific Computing Via Trevano 131 Email: [email protected] CH-6900 Lugano http://www.cscs.ch Phone: +41 91 610 82 15/Fax: +41 91 610 82 82 ---------------------------------------------------------------------- ------------------------------------------------------------------------------ _______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
