Ryan Davis wrote:
> I just committed a change that seems to cut memory use by about 40ish
> %. There was a phase that was munging the output to deal with an old
> yaml bug and was creating quite a bit of memory. Removing it dropped
> my memory consumption from 112meg to 73meg. We could drop a tad more
> if we nuked the part of the cache for the source we were currently
> bulk loading, but it only equated to about 10 meg in my experiments.
I've taken Ryan's changes (removing a copy step in the bulk download)
and added a few more for a beta release of RubyGems.
Here are the changes:
* Bulk downloads no longer copy the YAML string before loading it.
* Bulk downloads don't kick in until more than 500 gems need resyncing
(the old limit was 50).
* Users may set the bulk download threshhold in the command line (-B)
or in the .gemrc file.
Here are the memory stats using Ruby 1.8.5 on Mac OSX:
Old Version Bulk: 107Mb Real, 146Mb Virtual
New Version Bulk: 98Mb Real, 130Mb Virtual
New Version Incremental 36Mb Real, 63Mb Virtual
It looks like the bulk download memory footprint is rather modest. I
don't know if it will be enough for those who are currently feeling
the hurt.
However, by specifying a bulk download threshhold of 100000, you can
force gems to use the incremental method all the time. The downside to
using incremental downloads is that it is *much* slower for large
updates (however, if bulk downloads make you swap, then who knows).
On my machine, a bulk update takes about 20 seconds. An
incremental update takes about 0.2 seconds per out-of-date gem. If you
clobber the cache and update everything (currently 6600 some odd
gems), the incremental update takes about 20 minutes. My calculations
(based on my home network connection, the current size of the gem
index and many other things too numerous to mention) puts the break
even point at around 100 gems.
I would like those who are experiencing memory problems to give the
beta version a try and report back. You can get the beta version using
(using sudo if necessary on your system):
gem update --system --source http://onestepback.org/betagems
Of course, if you are having problems due to memory, the above might
not work. You can download the update explicitly using the following
URL:
http://onestepback.org/betagems/gems/rubygems-update-0.9.4.1.gem
and then install locally.
To change the threshhold value in the beta, use the -B option. For
example, the following command will incrementally synchronize up to a
million out of date gems:
gem list -r -B 1000000
I have one more idea on memory footprint issues, but would like to get
reports back on the beta before proceeding.
Thanks to everyone for their feedback.
--
-- Jim Weirich [EMAIL PROTECTED] http://onestepback.org
-- In theory, practice and theory are the same.
-- In practice, they are different.
_______________________________________________
Rubygems-developers mailing list
Rubygems-developers@rubyforge.org
http://rubyforge.org/mailman/listinfo/rubygems-developers
