On 02/15/12 09:45, Shawn Walker wrote:
On 02/08/12 18:09, Brock Pytlik wrote:
Webrev:
https://cr.opensolaris.org/action/browse/pkg/bpytlik/7139651-v1
Bug:
7139651 pkgrecv doesn't scale with large repos
With this change, memory use doesn't scale with the number of packages
being downloaded. It does read the manifests twice, but only downloads
them from the server once. This does introduce a small performance
penalty, roughly 2-3 more seconds of manifest processing when processing
4000 packages, but that's in the noise when considering the 23-24
minutes a full pkgrecv of those 4000 packages takes.
I'll assume this significantly reduces memory usage? Did you measure
how much?
Measurement was a bit difficult because we don't process manifests in a
particular order, so here are some example numbers which, to me,
demonstrate the improvement:
To get these numbers I modified both the gate and my workspace so that
it processed manifests but didn't download files as well as pausing
after every 1000 packages so that I could check what prstat was reporting:
Command: pkgrecv -s http://ipkg.us.oracle.com/opensolaris/dev -d ./tmp '*'
Packages Processed Gate Memory (MB) Workspace memory (MB)
1000 267 267
2000 303 267
3000 379 267
4000 483 267
Command: pkgrecv -m all-timestamps -s
http://ipkg.us.oracle.com/opensolaris/dev -d ./tmp '*'
Packages Processed Gate Memory (MB) Workspace memory (MB)
1000 392 267
2000 484 267
3000 606 267
4000 673 267
5000 1027 267
6000 1091 267
7000 1432 267
So, the amount of memory saved depends on how many packages are
processed, the more that are processed, the bigger the savings.
Brock
LGTM,
-Shawn
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
