Adam, I tried adding innodb=disabled;
To ListManager but 2.5.0 beta barfs on it: Config Check | WARNING: unknown parameter 'innodb' in block 'ListManager' line 74 I sadly found out that due to a mistake on my part 2.5.0 was built without lustre support so no stripe info was being collected, When I rebuilt performance was only 50entires/ms :-( Scans never finish, you can't run qurries. My solution for now is to roll all the way back to 2.3.x The last version of robinhood that worked at full speed for us. We will look at thi sall again when we have lustre + changelogs. It is sad because we really wanted the post scan script hook, we are working on a system where we use squop to pull all the data from robinhood into hive/pig and then run a bunch of stats on our system over time. Problem is robinhood isn't fast enough on the hardware I have available (7200RPM sata). Thanks for the help. Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion [email protected] (734)936-1985 On Dec 3, 2013, at 12:21 PM, Adam Brenner <[email protected]> wrote: > Brock, > > What version of RBH are you running? What is your commit_behavior set > to? I believe if you go ahead and add the following line to your RBH > configuration file within the ListManager and then within the MySQL > section, this should get you covered: > > innodb = disabled; > > You will then need to convert your database structure to MyISAM. It > might be easier to just to delete the database and start from fresh. > RBH should create the DB structure with MyISAM engine. Someone can > correct me on this if I am wrong. > > > However, what I am guessing is happening (and this is a big guess) is > that MySQL is not tuned for InnoDB. When I first setup RBH on our HPC > cluster, I noticed the same issue. Writes are extremely slow - double > digits insert records being reported by RBH. Here are a few things I > did to increase that to three/four digits record inserts (and > hopefully this information will help others). > > > Run MySQL Tuner[1]. Make sure this is ran after an initial full scan. > This will help you spot areas within the my.cnf configuration file to > update. If your system has plenty of memory, take advantage of that. > Setting > > innodb_file_per_table=1 > > would be a good idea, since each table will get their own namespace > for InnoDB rather then sharing the default with every other database. > Ideally, you want to keep the entire Innodb index in memory, > innodb_buffer_pool_size will do that. > > > Another thing you should do is read the guide Thomas posted earlier > about optimizing your RBH workflow[2]. Ideally, you want your bottle > neck to be the insert operation (STAGE_DB_APPLY) and not anything > else. You will need to fine tune the parameters to make this happen. > You might even want to consider lowering (or increasing) > max_pending_operations within EntryProcess. > > Hope that helps, > -Adam > > [1]: http://mysqltuner.pl > [2]: http://sourceforge.net/apps/trac/robinhood/wiki/TunePipeline > > > -- > Adam Brenner > Computer Science, Undergraduate Student > Donald Bren School of Information and Computer Sciences > > Research Computing Support > Office of Information Technology > http://www.oit.uci.edu/rcs/ > > University of California, Irvine > www.ics.uci.edu/~aebrenne/ > [email protected] > > > On Tue, Dec 3, 2013 at 7:52 AM, Brock Palen <[email protected]> wrote: >> Is there a way in newer versions of robinhood to go back to the old MyISAM >> tables and settings used in pervious versions? >> >> We, as many sites do, still run lustre 1.8 and thus must to full scans, in >> pervious versions of robinhood on the same hardware we had 1500 updates/s >> now we are in a few hundred, the database is so slammed (note we run on >> cheap spindles) it makes robinhood useless, scans take multiple days and >> because the disk is so busy we can't run any meaningful queries. >> >> Any thoughts on this would be helpful, I have been on the list before and we >> have messed with the mysql settings a lot to try to speed up the scan to no >> success. >> >> Brock Palen >> www.umich.edu/~brockp >> CAEN Advanced Computing >> XSEDE Campus Champion >> [email protected] >> (734)936-1985 >> >> >> >> >> ------------------------------------------------------------------------------ >> Rapidly troubleshoot problems before they affect your business. Most IT >> organizations don't have a clear picture of how application performance >> affects their revenue. With AppDynamics, you get 100% visibility into your >> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics >> Pro! >> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk >> _______________________________________________ >> robinhood-support mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/robinhood-support >>
signature.asc
Description: Message signed with OpenPGP using GPGMail
------------------------------------------------------------------------------ CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments & Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________ robinhood-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/robinhood-support
