Re: [osol-discuss] very long wait in the end of pkg install
On 03/ 1/10 01:51 AM, casper@sun.com wrote: ... I think you're confused about what I've said here. And no, the data is intentionally stored by digest and not by its original name. It would neither be practical, nor efficient to store the files in the repository by their delivered name. The size of the /dev repository is currently around 63.6G and growing... A hardlink doesn't cost (much) of additional storage. To be clear, the cache stores compressed file content named after a digest used for the file. The structure of the client's download cache is intentionally the same as that of the repository. In addition, the structure of the cache has been optimised with ZFS and client/server operations in mind. If you look at /var/pkg/download on your own system, it will become clear why linking is not practical. The exact number of directories and expected number of files per-directory in the cache is all the result of careful, thorough performance and storage analysis. I would reiterate that if a user needs to restore or fix various packages on a system, the correct, supported interface to do that is to use the pkg command, preferably via LiveCD or other mechanism. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
For example, on my system, when I did an image-update from build 110 to build 111, with all of the data to be installed downloaded already, it only took about five minutes to upgrade 666 packages and move around 480 megabytes worth of data. That's not too shabby if you ask me. Compare upgrading from build 128 to later :-) Sorry, I meant from build 118; the continuous rewriting of the contents file was a bug which was easily fixed. I don't think you should benchmark against a completely broken system; to use that then your bar would by very low. What was interesting is that zfs improved the old contents file issue on systems with more than 1GB of memory; with ufs each update of the contents file was completely written to disk; this one done at least twice for each package. With zfs much of the contents file was never written to disk. On laptops, with slow disks, the difference would be greater. During the development I noticed how much I/O was done before/after when installing a zone: 15m06 sd1 read 59M write 34G (ufs) 6m30 sd1 read 66M write 583M (zfs eats 33GB of I/O) but after it sis more like: 4m02 sd1 read 83M write 319M (ufs) 2m59 sd1 read 76M write 129M A full install used to write 150GB. The use of pkzip7 further penalizes the old install; uncompressing takes around 1/3rd of the install time. Casper ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
It only *looks* like it downloads everything again -- this is purely a misleading display issue that we're hoping to improve at a future point. It did take a reasonable amount of time. Not less than downloading the first time, but I can't be sure. The ips server is local, though. What happens is that the transport system iterates through all the content needed for the operation, verifies that it's valid, and then skips the download for the file because it already has the content. If you actually snoop network traffic, you'll see it doesn't actually download all the files again. I think you're confused about what I've said here. And no, the data is intentionally stored by digest and not by its original name. It would neither be practical, nor efficient to store the files in the repository by their delivered name. The size of the /dev repository is currently around 63.6G and growing... A hardlink doesn't cost (much) of additional storage. Since files are retrieved by digest, there's no reason to cache them by any other means. But it would be useful. Casper ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
[...] Except that pkg uses elfcmp/wsdiff type mechanisms to not rev ELF binaries that don't differ in the important bits, but only differ in the timestamps or other metadata. [...] Are there standalone tools like that, able to ignore trivial differences when comparing ELF binaries? The only elfcmp I've found so far is meant to compare a binary to a running process, to ensure the process hasn't been tampered with. -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
Richard L. Hamilton wrote: [...] Except that pkg uses elfcmp/wsdiff type mechanisms to not rev ELF binaries that don't differ in the important bits, but only differ in the timestamps or other metadata. [...] Are there standalone tools like that, able to ignore trivial differences when comparing ELF binaries? The only elfcmp I've found so far is meant to compare a binary to a running process, to ensure the process hasn't been tampered with. See /opt/onbld/bin/elfcmp in the SUNWonbld package. -- -Alan Coopersmith- alan.coopersm...@sun.com Oracle Solaris Platform Engineering: X Window System ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 03/ 1/10 03:22 AM, casper@sun.com wrote: ... It only *looks* like it downloads everything again -- this is purely a misleading display issue that we're hoping to improve at a future point. It did take a reasonable amount of time. Not less than downloading the first time, but I can't be sure. The ips server is local, though. If you can actually prove it's happening, then file a bug or email pkg-discuss about it. My guess is that there's more than meets the eye to the issue you're seeing. Since files are retrieved by digest, there's no reason to cache them by any other means. But it would be useful. The manifest files for the packages have provide all the information you need to map names to digests; adding links would do nothing to improve the performance of the system. Again, the structure of the cache is not conducive to the sort of linking you're talking about. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 03/ 1/10 03:20 AM, casper@sun.com wrote: ... For example, on my system, when I did an image-update from build 110 to build 111, with all of the data to be installed downloaded already, it only took about five minutes to upgrade 666 packages and move around 480 megabytes worth of data. That's not too shabby if you ask me. Compare upgrading from build 128 to later :-) Sorry, I meant from build 118; the continuous rewriting of the contents file was a bug which was easily fixed. I don't think you should benchmark against a completely broken system; to use that then your bar would by very low. So I checked back into this. Significant performance regressions and improvements both occurred during builds 118-132 for pkg(5). So depending on what build you were upgrading from and to, you could have seen wildly varying performance during image-update. On my laptop (with a 4200 RPM hard drive I might add, but 4GB memory), it took around 7 minutes on average to upgrade from build 131 to build 133, using the gate version of pkg (which will be delivered in 134), which was comprised of: * updates to 1,528 packages (comprised of 13,504 files) * 423.4MB of data retrieved (not already cached) * 13,414 removals; 48,509 installs; 19,245 updates I'd say that's pretty good, and the good news is that time should go down even further in the future... What was interesting is that zfs improved the old contents file issue on systems with more than 1GB of memory; with ufs each update of the contents file was completely written to disk; this one done at least twice for each package. With zfs much of the contents file was never written to disk. ... Yep, the old SVR4 pkg system is much better than it used to be. However, it also only has to manage a few thousand package versions (at most). pkg(5) has to manage information for over 68,000 unique package versions currently for builds all the way from 86-133. I greatly appreciate the time you've spent to provide feedback and insight into your own forays into this area. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
Alan Coopersmith wrote: Richard L. Hamilton wrote: [...] Except that pkg uses elfcmp/wsdiff type mechanisms to not rev ELF binaries that don't differ in the important bits, but only differ in the timestamps or other metadata. [...] Are there standalone tools like that, able to ignore trivial differences when comparing ELF binaries? The only elfcmp I've found so far is meant to compare a binary to a running process, to ensure the process hasn't been tampered with. See /opt/onbld/bin/elfcmp in the SUNWonbld package. Elfcmp is a useful tool to determine if two binaries are essentially equivalent. However, I believe there are minor differences between what elfcmp considers the same and what IPS does. That may be fine for your purposes; just don't expect matching results 100% of the time. Scott -- Scott Rotondo Senior Principal Engineer, Solaris Engineering President, Trusted Computing Group Phone/FAX: +1 408 850 3655 (Internal x68278) ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 03/ 1/10 07:19 PM, Scott Rotondo wrote: Alan Coopersmith wrote: Richard L. Hamilton wrote: [...] Except that pkg uses elfcmp/wsdiff type mechanisms to not rev ELF binaries that don't differ in the important bits, but only differ in the timestamps or other metadata. [...] Are there standalone tools like that, able to ignore trivial differences when comparing ELF binaries? The only elfcmp I've found so far is meant to compare a binary to a running process, to ensure the process hasn't been tampered with. See /opt/onbld/bin/elfcmp in the SUNWonbld package. Elfcmp is a useful tool to determine if two binaries are essentially equivalent. However, I believe there are minor differences between what elfcmp considers the same and what IPS does. That may be fine for your purposes; just don't expect matching results 100% of the time. Indeed; and the algorithms that pkg(5) uses will change (and have changed) over time. -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/28/10 06:10 AM, Shawn Walker wrote: On 02/27/10 10:29 PM, Anon Y Mous wrote: Ok, install OpenSolaris on a server, create a zone, and then in your first zone just running this very basic IPS command: pkg install SUNWman takes forever because it downloads each man page file individually. Downloading a large number of small files via FTP or whatever will always be slower than downloading one really big file over a TCP/IP network, and this is the achilles heel of IPS. The fact that a majority of man pages are delivered in a single package is a package bug: http://defect.opensolaris.org/bz/show_bug.cgi?id=1964 With that said, yes, the individual file retrieval system currently used by pkg(5) does have tradeoffs in transfer performance depending on the scenario. In the initial install case, a single pre-generated bundle would be better for transfers, although not nearly as much as some might believe. However, a per-file based retrieval system has advantages over one that relies on pre-generated bundles of package content. Specifically, it enables additional functionality that would otherwise be costly (resource-wise) or not as practical, such as: * multi-variant (and facet) packages -- Allows package creators to determine package boundaries using delivered functionality (content) instead of being forced to use architecture or zone variations. -- Enhances the user experience by simplifying the management of packages on a system. -- Makes it possible for a user to change between or add additional sets of functionality to their system efficiently. For example, change from x86 to SPARC, downloading only the files that have changed. * efficient update operations -- Because pkg(5) only retrieves the files that have changed between package versions between updates, the client only downloads exactly what it needs to perform update operations. Other systems have chosen to implement this by pre-generating deltas between package versions, but that also means that their users have to rely on a pre-generated delta being available for every possible origin point from which they need to update from and to. * efficient repair operations -- Because pkg(5) only has to retrieve the individual files it needs to perform an operation, this greatly reduces the time needed to restore missing package content on a system. * greatly reduced publication resource costs -- Because pkg(5) stores package content as individual files (by content hash), files that are identical between packages are shared on the server. In addition, files that don't change between package versions are also shared. This can greatly reduce the resource-cost of publication storage and publication time. apt-get on Nexenta is really kicking OpenSolaris IPS's butt right now in terms of performance and this is a real apples to apples comparison because both Nexenta and Oracle OpenSolaris both use the same SunOS / unix / genunix kernel, the same ZFS file system, etc. so the problem is obviously with the way IPS is implemented and not with the Solaris kernel itself (which has blazing fast performance as seen on Nexenta and Milax). At the moment, I'm not aware of any posteriors that feel harmed. With that said, apt and pkg(5) do not have completely equivalent functionality, so it's difficult to perform an apples-to-apples comparison. As an in-devleopment project that has been around for far less years (decades?) than apt-get, I believe pkg(5) is doing pretty well. Performance work remains on-going. Where are all the dtrace performance improvement people hiding? Perhaps they can use dtrace to explain why apt-get on Nexenta's apt-get is so much faster, since Nexenta also has dtrace? Might be a good topic for a sun.com blog or paper write-up. When you look for problems, you'll find them. But I'd also say, if you don't look for *improvements*, you won't find them either. Quite frankly, when you say things like is so much faster without exactly quantifying what you're talking about, it tends to look a little vague and hand-wavy. If you look back at the bug entries for pkg(5) or the mailing list discussions, you'll find that a significant amount of time during the last release cycle has been spent on improving performance. In addition, every time a major change was made, performance has always been part of the discussion. I personally have spent easily a few weeks worth of time dtrace profiling parts of the package system and many more using other methods of profiling. And I know other team members have spent a lot of time doing performance analysis as well. For example, the performance of pkg info and pkg list has greatly improved: * pkg info on 2009.06: 3 seconds - 2010.x: 0.3 seconds * pkg list on 2009.06: 5.4 seconds - 2010.x: 0.57 seconds * pkg list -as on 2009.06: 33 seconds - 2010.x: 3.56 seconds Likewise, Johansen replaced pkg(5)'s
Re: [osol-discuss] very long wait in the end of pkg install
* efficient update operations -- Because pkg(5) only retrieves the files that have changed between package versions between updates, the client only downloads exactly what it needs to perform update operations. Other systems have chosen to implement this by pre-generating deltas between package versions, but that also means that their users have to rely on a pre-generated delta being available for every possible origin point from which they need to update from and to. Do we actually implement that? Currently, every new release revs all the binaries; how much of the OS is not a binary? (Upgrading using pkg is much slower than liveupgrade; in both cases I use a local server). * efficient repair operations -- Because pkg(5) only has to retrieve the individual files it needs to perform an operation, this greatly reduces the time needed to restore missing package content on a system. The latency will nearly always kill you; I believe that the Windows method is superior: you do keep all the package spooled locally but repairing is than simple and can happen off-line. * greatly reduced publication resource costs -- Because pkg(5) stores package content as individual files (by content hash), files that are identical between packages are shared on the server. In addition, files that don't change between package versions are also shared. This can greatly reduce the resource-cost of publication storage and publication time. While I'm fine with that, I would suggest that you also use hardlinks and also keep the data installed linked to the proper name. E.g., I have broken python and now I need to copy the data from the server. Where is it? Casper ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
casper@sun.com wrote: * efficient update operations -- Because pkg(5) only retrieves the files that have changed between package versions between updates, the client only downloads exactly what it needs to perform update operations. Other systems have chosen to implement this by pre-generating deltas between package versions, but that also means that their users have to rely on a pre-generated delta being available for every possible origin point from which they need to update from and to. Do we actually implement that? Currently, every new release revs all the binaries; how much of the OS is not a binary? Except that pkg uses elfcmp/wsdiff type mechanisms to not rev ELF binaries that don't differ in the important bits, but only differ in the timestamps or other metadata. Outside of ELF binaries, the OS is full of man pages, header files, shell/perl/python scripts, image files, fonts, configuration files, etc. An image-update is often noticably much much smaller than the DVD image downloads used to be. -- -Alan Coopersmith- alan.coopersm...@sun.com Oracle Solaris Platform Engineering: X Window System ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/28/10 03:01 PM, casper@sun.com wrote: * efficient update operations -- Because pkg(5) only retrieves the files that have changed between package versions between updates, the client only downloads exactly what it needs to perform update operations. Other systems have chosen to implement this by pre-generating deltas between package versions, but that also means that their users have to rely on a pre-generated delta being available for every possible origin point from which they need to update from and to. Do we actually implement that? Currently, every new release revs all the binaries; how much of the OS is not a binary? Yes, it was implemented. I assume you're talking about ON because not every binary changes between releases. Even then, pkg(5) uses a custom content digest to perform this comparison based on file type. Which means that in the case of elf files, only if the file meaningfully changed will it get upgraded. pkg(5) team members have also been working with members of other consolidations or projects to avoid re-delivery where possible. I did an example comparison of this between dev builds sometime ago, and if I remember correctly, this mechanism reduced total transfer size by about 30%. (Upgrading using pkg is much slower than liveupgrade; in both cases I use a local server). That doesn't match the general feedback I've seen (on x86). My personal experience has been that image-update is way faster than liveupgrade. For example, on my system, when I did an image-update from build 110 to build 111, with all of the data to be installed downloaded already, it only took about five minutes to upgrade 666 packages and move around 480 megabytes worth of data. That's not too shabby if you ask me. * efficient repair operations -- Because pkg(5) only has to retrieve the individual files it needs to perform an operation, this greatly reduces the time needed to restore missing package content on a system. The latency will nearly always kill you; I believe that the Windows method is superior: you do keep all the package spooled locally but repairing is than simple and can happen off-line. Except that pkg(5) actually does both. It keeps the package data spooled locally, and if the user has chosen to purge the download cache, it can retrieve just the data it needs. * greatly reduced publication resource costs -- Because pkg(5) stores package content as individual files (by content hash), files that are identical between packages are shared on the server. In addition, files that don't change between package versions are also shared. This can greatly reduce the resource-cost of publication storage and publication time. While I'm fine with that, I would suggest that you also use hardlinks and also keep the data installed linked to the proper name. I think you're confused about what I've said here. And no, the data is intentionally stored by digest and not by its original name. It would neither be practical, nor efficient to store the files in the repository by their delivered name. The size of the /dev repository is currently around 63.6G and growing... E.g., I have broken python and now I need to copy the data from the server. Where is it? If python wasn't broken, you'd use pkg fix to fix it. If python was unavailable, you'd go to the BUI for the repository, and search for /usr/bin/python under advanced search. That would list the action (along with its hash and target path) that matched the delivered file. You could then use the hash to retrieve the compressed source file and restore it. The BUI could change of course to simplify this a little bit, but you get the idea. If you were off-line and couldn't use the BUI to search, you could grep through the package manifests in /var/pkg/pkg for usr/bin/python and do the same thing. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/28/10 03:22 PM, Shawn Walker wrote: On 02/28/10 03:01 PM, casper@sun.com wrote: ... E.g., I have broken python and now I need to copy the data from the server. Where is it? If python wasn't broken, you'd use pkg fix to fix it. If python was unavailable, you'd go to the BUI for the repository, and search for /usr/bin/python under advanced search. That would list the action (along with its hash and target path) that matched the delivered file. You could then use the hash to retrieve the compressed source file and restore it. The BUI could change of course to simplify this a little bit, but you get the idea. If you were off-line and couldn't use the BUI to search, you could grep through the package manifests in /var/pkg/pkg for usr/bin/python and do the same thing. And before I forget, you could also boot from a LiveCD that matched the build you were running and: * mount the target boot environment somewhere * execute pkg -R /mntpoint fix ...which would use the download cache if present, and if the content needed was missing from the download cache, it would retrieve it if possible. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
For example, on my system, when I did an image-update from build 110 to build 111, with all of the data to be installed downloaded already, it only took about five minutes to upgrade 666 packages and move around 480 megabytes worth of data. That's not too shabby if you ask me. Compare upgrading from build 128 to later :-) Except that pkg(5) actually does both. It keeps the package data spooled locally, and if the user has chosen to purge the download cache, it can retrieve just the data it needs. So when upgrade image dies, e.g., because there's a symlink or directory in the wrong place and you remove the offending file, why does it download everything again? I think you're confused about what I've said here. And no, the data is intentionally stored by digest and not by its original name. It would neither be practical, nor efficient to store the files in the repository by their delivered name. The size of the /dev repository is currently around 63.6G and growing... A hardlink doesn't cost (much) of additional storage. Casper ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 03/ 1/10 01:51 AM, casper@sun.com wrote: For example, on my system, when I did an image-update from build 110 to build 111, with all of the data to be installed downloaded already, it only took about five minutes to upgrade 666 packages and move around 480 megabytes worth of data. That's not too shabby if you ask me. Compare upgrading from build 128 to later :-) From what I recall, it was still only a few minutes if all the content is cached. Again, x86 only. Except that pkg(5) actually does both. It keeps the package data spooled locally, and if the user has chosen to purge the download cache, it can retrieve just the data it needs. So when upgrade image dies, e.g., because there's a symlink or directory in the wrong place and you remove the offending file, why does it download everything again? It only *looks* like it downloads everything again -- this is purely a misleading display issue that we're hoping to improve at a future point. What happens is that the transport system iterates through all the content needed for the operation, verifies that it's valid, and then skips the download for the file because it already has the content. If you actually snoop network traffic, you'll see it doesn't actually download all the files again. I think you're confused about what I've said here. And no, the data is intentionally stored by digest and not by its original name. It would neither be practical, nor efficient to store the files in the repository by their delivered name. The size of the /dev repository is currently around 63.6G and growing... A hardlink doesn't cost (much) of additional storage. Since files are retrieved by digest, there's no reason to cache them by any other means. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). iostat -D 2 shows my hd busy at around 100% iosnoop reports a lot of access by pkg : UID PID DBLOCK SIZE COMM PATHNAME [...] 0 1666 R 35943541 4096pkg none 0 1666 R 35689921 4096pkg none 0 1666 R 4440824 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35921758 4096pkg none 0 1666 R 4622752 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35914394 4096pkg none 0 1666 R 4619760 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35927313 4096pkg none 0 1666 R 4628296 4096pkg none 1 465 R 14771611512 idmapd none 65535 626 R 4616648 4096 Xorg none 0 1666 R 4628016 4096pkg none 0 1666 R 4639352 4096pkg none 0 1666 R 35668129 4096pkg none 0 1666 R 4419792 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35745803 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35827415 4096pkg none 0 1666 R 4560008 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 30094080 65536pkg none 1 465 R 14771611512 idmapd none 0 1666 R 30321619 4096pkg none 0 1666 R 4325904 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35841146 6144pkg none 0 1666 R 35908772 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35855403 4096pkg none 0 1666 R 35680068 4096pkg none 0 1666 R 4431488 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35841146 6144pkg none 0 1666 R 35925193 4096pkg none 0 1666 R 4626176 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35924929 4096pkg none 0 1666 R 4625912 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 4494072 4096pkg none 0 1666 R 4615440 4096pkg none 0 1666 R 4581384 4096pkg none 0 1666 R 30091646 6144pkg none 1 465 R 14771611512 idmapd none 0 1666 R 30320827 4096pkg none 0 1666 R 4325112 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35687936 65536pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35687571 4096pkg none 0 1666 R 4438632 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35693271 4096pkg none 0 1666 R 000 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35825142 6144pkg none 0 1666 R 35833326 4096pkg none 0 1666 R 4565544 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35812864 65536pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35812610 4096pkg none 0 1666 R 4553536 4096pkg none 0 1666 R 35841146 6144pkg none 0 1666 R 35972910 4096pkg none 0 1666 R 4667736 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35925153 4096pkg none 0 1666 R 4626136 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35958985 4096pkg none 0 1666 R 4654040 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35928401 4096pkg none 0 1666 R 4629384 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35927377 4096pkg none 0 1666 R 4628360 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35709858 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35945429 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35825142 6144pkg none 0 1666 R 35833702 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35841146 6144pkg none 0 1666 R 35921280 65536pkg none 1 465 R 14771611512 idmapd none [...] I have 4GB of swap, of which 3.6G free Any ideas someone ? pkg is barely usable on this system -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
Hi, Performance was also pretty terrible on my old Core 2 Quad with 4GB RAM. That was one of the main reasons I switched to Nexenta (http://nexenta.org). Andy On 27 February 2010 10:52, Bruno Damour br...@ruomad.net wrote: well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). iostat -D 2 shows my hd busy at around 100% iosnoop reports a lot of access by pkg : UID PID DBLOCK SIZE COMM PATHNAME [...] 0 1666 R 35943541 4096pkg none 0 1666 R 35689921 4096pkg none 0 1666 R 4440824 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35921758 4096pkg none 0 1666 R 4622752 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35914394 4096pkg none 0 1666 R 4619760 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35927313 4096pkg none 0 1666 R 4628296 4096pkg none 1 465 R 14771611512 idmapd none 65535 626 R 4616648 4096 Xorg none 0 1666 R 4628016 4096pkg none 0 1666 R 4639352 4096pkg none 0 1666 R 35668129 4096pkg none 0 1666 R 4419792 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35745803 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35827415 4096pkg none 0 1666 R 4560008 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 30094080 65536pkg none 1 465 R 14771611512 idmapd none 0 1666 R 30321619 4096pkg none 0 1666 R 4325904 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35841146 6144pkg none 0 1666 R 35908772 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35855403 4096pkg none 0 1666 R 35680068 4096pkg none 0 1666 R 4431488 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35841146 6144pkg none 0 1666 R 35925193 4096pkg none 0 1666 R 4626176 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35924929 4096pkg none 0 1666 R 4625912 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 4494072 4096pkg none 0 1666 R 4615440 4096pkg none 0 1666 R 4581384 4096pkg none 0 1666 R 30091646 6144pkg none 1 465 R 14771611512 idmapd none 0 1666 R 30320827 4096pkg none 0 1666 R 4325112 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35687936 65536pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35687571 4096pkg none 0 1666 R 4438632 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35693271 4096pkg none 0 1666 R 000 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35825142 6144pkg none 0 1666 R 35833326 4096pkg none 0 1666 R 4565544 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35812864 65536pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35812610 4096pkg none 0 1666 R 4553536 4096pkg none 0 1666 R 35841146 6144pkg none 0 1666 R 35972910 4096pkg none 0 1666 R 4667736 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35925153 4096pkg none 0 1666 R 4626136 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35958985 4096pkg none 0 1666 R 4654040 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35928401 4096pkg none 0 1666 R 4629384 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35927377 4096pkg none 0 1666 R 4628360 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35709858 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35945429 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35825142 6144pkg none 0 1666 R 35833702 4096pkg none 1 465 R 14771611512 idmapd none 0 1666 R 35841146 6144pkg none 0 1666 R 35921280 65536pkg none 1 465 R 14771611512 idmapd none [...] I have 4GB of swap, of which 3.6G free Any ideas someone ? pkg is barely usable on this system -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list
Re: [osol-discuss] very long wait in the end of pkg install
well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). iostat -D 2 shows my hd busy at around 100% iosnoop reports a lot of access by pkg : How much memory? It is possible that you're paging to death. Casper ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
No hope for IPS. It kills OpenSolaris´s future. This is not an attack. Just a sad finding. Yes, I bring this to the table again. But let´s face it, what is better? A convenient slimer or an inconvenient person who reports the truth at least once per year? A mistake is only then a mistake, if it doesn´t get corrected. IPS is such a case. From day 1 it was a bad implementation. The paradigm is good. The python code looks nice and sophisticated. But it is un-usable in the current form. Plus the status console output is cryptic and meaningless (while also not being verbose). Conary - on the other hand - had been available under a free license plus usable 3 years before IPS was started. It is a sad fact. Am I a lazy person who always complains without ever offering alternatives? Judge yourself, but together with Brian Gupta (who significantly sponsored me financially) we ported conary to OpenSolaris. Src and test-bins were there. Nobody ever wanted it. So do what you want. But be aware that it kills your distro and associated market share. Yesterday I wanted to install openoffice. After 2 hours I killed it, because it already ^^proceeded^^ till about 80MB of 2800MB. I needed to edit my word document yesterday, not today. During these first 2 hours the box was completely out of mem and out of CPU cycles. Totally inresponsive (3 to 10 seconds per click). After I killed pkg, I could continue to do something, just as normal. This box is a 2GHz Pentium Dual Core Notebook manufactured in December 2008. Two Intel cores running at 2GHz each. Plus the RAM is 4GB. How many engineers have been occupied by doing this IPS? How many more have been busy to adopt their projects from SVR4-pkgadd to IPS? I bet a certain person will now show up and tell me, that it is complete dieren etc. And that it was a wanted community project (in 2007/08 he always claimed, it was just simply a trial). And that it is of course unfinished alpha software and that - finally after its completion - all bottlenecks will have been removed. And that OpenSolaris´ special requirements made it impossible to base the work on a functioning system (such as conary). And that it is bad practice to complain about the work of others. And that this is the essence of DEVELopment and PROGRESS, to throw all other things away and to start from 0. Employing 100 engineers for a few years and throwing away money that nobody had (while other fronts, such as SPARC-gfx) had to die out due to a ^lack of money and resources^ . And it is not alone Python´s fault (interpreted vs. compiled), because conary is also in Python. Ok, if you want to jump, feel free to freely fall. I tried to help and got burned for it (in 2007). -- Your Choice -- And worse for you: Customers also choose. %martin bochnig Old page was: http://hub.opensolaris.org/bin/view/Project+conary-eval (most things seem to have been deleted after website migration) On Sat, Feb 27, 2010 at 12:17 PM, Andrew Stormont andyjstorm...@googlemail.com wrote: Hi, Performance was also pretty terrible on my old Core 2 Quad with 4GB RAM. That was one of the main reasons I switched to Nexenta (http://nexenta.org). Andy On 27 February 2010 10:52, Bruno Damour br...@ruomad.net wrote: well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). iostat -D 2 shows my hd busy at around 100% iosnoop reports a lot of access by pkg : UID PID D BLOCK SIZE COMM PATHNAME [...] 0 1666 R 35943541 4096 pkg none 0 1666 R 35689921 4096 pkg none 0 1666 R 4440824 4096 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 35921758 4096 pkg none 0 1666 R 4622752 4096 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 35914394 4096 pkg none 0 1666 R 4619760 4096 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 35927313 4096 pkg none 0 1666 R 4628296 4096 pkg none 1 465 R 14771611 512 idmapd none 65535 626 R 4616648 4096 Xorg none 0 1666 R 4628016 4096 pkg none 0 1666 R 4639352 4096 pkg none 0 1666 R 35668129 4096 pkg none 0 1666 R 4419792 4096 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 35745803 4096 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 35827415 4096 pkg none 0 1666 R 4560008 4096 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 30094080 65536 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 30321619 4096 pkg none 0 1666 R 4325904 4096 pkg none 1 465 R 14771611 512 idmapd none 0 1666 R 35841146 6144 pkg none 0 1666 R 35908772 4096 pkg none 1 465 R 14771611 512
Re: [osol-discuss] very long wait in the end of pkg install
On Sat, Feb 27, 2010 at 12:48 PM, casper@sun.com wrote: well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). iostat -D 2 shows my hd busy at around 100% iosnoop reports a lot of access by pkg : How much memory? It is possible that you're paging to death. Hi Casper, I am subscribed to pkg and caiman. Maybe his system is swapping to death. But why??? If you compare memory- and CPU- usage of IPS to those values of *every* other system on the market, it is unbelievable at the first look. ^Horrific^ - to find the right term. %martin ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
Ok I have ONLY 1G mem, not much nowadays but anyway. This laptop has been tested on different systems, first of all it runs WinXP (at work) but I have also run gentoo (from stage 1 !, all self-compiled) and I never felt it unresponsive or slow. even running emerge -uD world was OK compared to what it feels withy pkg. At least you had some output to watch ;-) and I never had to stop browsing the web because I was updating my system. My main grief is not the time it takes, but that you cannot do anything during this time. Even shell windows go grey. Bruno -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On Sat, Feb 27, 2010 at 1:19 PM, Bruno Damour br...@ruomad.net wrote: Ok I have ONLY 1G mem, not much nowadays but anyway. This laptop has been tested on different systems, first of all it runs WinXP (at work) but I have also run gentoo (from stage 1 !, all self-compiled) and I never felt it unresponsive or slow. even running emerge -uD world was OK compared to what it feels withy pkg. At least you had some output to watch ;-) and I never had to stop browsing the web because I was updating my system. My main grief is not the time it takes, but that you cannot do anything during this time. Even shell windows go grey. Bruno Good description. It is completely impossible to work with or on anything, while pkg is doing his day-long trip. Not even running two big builds would bring the a given box in such a state (such as compiling OS/Net and X11 at once). That´s the main point. Just keeping it running (like wget) would not be that bad. -- %mab ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On Sat, Feb 27, 2010 at 12:48 PM, casper@sun.com wrote:
well the whole install took 2 hours
for 1 pkg !!
it seems that disk read/write here is the bottleneck (?).
iostat -D 2 shows my hd busy at around 100%
iosnoop reports a lot of access by pkg :
How much memory? =A0It is possible that you're paging to death.
Hi Casper,
I am subscribed to pkg and caiman.
Maybe his system is swapping to death.
But why??? If you compare memory- and CPU- usage of IPS to those
values of *every* other system on the market, it is unbelievable at
the first look. ^Horrific^ - to find the right term.
Yes, that is true.
I'm not sure why it needs so much memory; it is possible that using
python contributes to the memory use. Both perl and python of wonderful
primitives but you pay for how the primitives are implemented: they don't
give you a choice.
A long time ago, we re-wrote the contents file database in a (SQL)
database; that was a bad idea for several reasons: you have again no choice
on how the data is stored and this again added to huge memory pressure
and apart from other issues such as not properly sorting the contents file,
it made installing much slower and not faster. The contents file was then
around 15-20MB, the database used 512MB or more.
It took me around 10 days to re-write the svr4 package tools using some
form of home-grown database: but this database only needed indexing on one key,
the pathname, to me it was important to keep the memory use down as I
wanted to run it on small systems (upgrade with a shrunk miniroot on a
256MB system)
I used avl trees and stored the following:
typedef struct pkgentry {
char *line; /* The contents line for the file */
avl_node_t avl; /* The avl header */
int pkgoff; /* Where the packages live; start with SP */
int pathlen;/* The length of the pathname */
int len;/* Length of the line (incl NUL) */
} pkgentry_t;
I keep the whole line and I don't actually parse it and most of the work is
still done by the client programs.
The contents file was at the end of SXDE around 40MB(!); the daemon needs
only 60MB.
I don't know how much data the IPS needs to keep in memory, but I'm sure
that the penalty is much larger than the 50%.
It gets worse when you want to install/upgrade zones: (memory + cpu)* #zones.
You can scale if your operation is O(1) or the constant is small.
Casper
___
opensolaris-discuss mailing list
opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
I ran pkg install with truss, in the hope of discovering why it takes so long to complete, especially AFTER it has reported that every thing is installed. You are running a fresh install of b133, correct? Has pkg been always that slow? I suspect that pkg has been reasonably fast for the first few packages that you installed, but has become slower and slower the more packages have been installed? I ran into this (for me) mysterious result : # truss -o pkg.log -D pkg install diagnostic/wireshark # tail pkg.log 1.7255schedctl() = 0xFEC69000 0.0002sigaction(SIGINT, 0x08047650, 0x080476D0) = 0 421.4744 open(/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/messages.mo, O_RDONLY) Err#2 ENOENT Yep, a truss with options -m all -t all shows that there are lots of FLTPAGE pagefaults before the failed LC_MESSAGES/messages open. -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
# tail pkg.log 1.7255 schedctl() = 0xFEC69000 0.0002sigaction(SIGINT, 0x08047650, 0x080476D0) = 0 421.4744 open(/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/messages.mo, O_RDONLY) Err#2 ENOENT stack backtrace for the pkg process at the time after the sigaction() system call: core 'core.3048' of 3048: /usr/bin/python2.6 /usr/bin/pkg install diagnostic/wireshark fed85a88 update_refs (fedf3a50) + c fed8644c collect (2, fedcc63c, 80476d8, fed86e72) + 238 fed86e9c PyGC_Collect (0, fedcc63c, 80476f8, fed78c98, 0, fedcc63c) + 34 fed766e7 Py_Finalize (0, fedcc63c, 8047738, fed7791b, 0, fedcc63c) + 47 fed78c98 Py_Exit (0, fedcc63c, 8047738, fed77912) + 18 fed7791b handle_system_exit (80a3a40, fedcc63c, 8047778, fed78152, 80a3a40, 80a3a40) + 13f fed77959 PyErr_PrintEx (1, 8047af7, 101, fed77608) + 31 fed77619 PyErr_Print (80477c0, fefc47a4, 8047aff, 1, fed76c46, 0) + 1d fed772b2 PyRun_SimpleFileExFlags (fef867e8, 8047af7, 1, 804789c) + 1c2 fed76c46 PyRun_AnyFileExFlags (fef867e8, 8047af7, 1, 804789c) + 6e fed85694 Py_Main (4, 80479a8, feef6f25, 8050b88, 1, 0) + a94 08050bdf main (4, 80479a8, 80479bc, 804799c) + 63 08050aed _start (4, 8047ae4, 8047af7, 8047b04, 8047b0c, 0) + 7d Python seems to be running a garbage collection at system exit, and this touches lots of memory. And I suspect that the pkg process on your machine has a RSS available free memory. -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 03:50 PM, Jürgen Keil wrote: I ran pkg install with truss, in the hope of discovering why it takes so long to complete, especially AFTER it has reported that every thing is installed. You are running a fresh install of b133, correct? yes, install from livecd as I couldn't upgrade b132 Has pkg been always that slow? I think maybe not that slow before, say b130, especially not in the post-install phase (cretaing plan as alxways been a bit slow but after it went rather well AFAIK I suspect that pkg has been reasonably fast for the first few packages that you installed, but has become slower and slower the more packages have been installed? Not that much packages installed after fress install I ran into this (for me) mysterious result : # truss -o pkg.log -D pkg install diagnostic/wireshark # tail pkg.log 1.7255 schedctl() = 0xFEC69000 0.0002 sigaction(SIGINT, 0x08047650, 0x080476D0) = 0 421.4744open(/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/messages.mo, O_RDONLY) Err#2 ENOENT Yep, a truss with options -m all -t all shows that there are lots of FLTPAGE pagefaults before the failed LC_MESSAGES/messages open. ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
I suspect that pkg has been reasonably fast for the first few packages that you installed, but has become slower and slower the more packages have been installed? Not that much packages installed after fress install That is, on your fresh b133 install from livecd, the first package install pkg install diagnostic/wireshark is already slow in the post install phase (slow: more than 10 minutes) ? What process size is listed by prstat or top for the pkg process during the post install phase? (In top I observe 180M during the install phase, and 390M post install, on a fresh b133 install, when I install wireshark as the first package) -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 05:31 PM, Rich Burridge wrote: Bruno Damour wrote: well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). Maybe you are seeing the effects of: http://defect.opensolaris.org/bz/show_bug.cgi?id=14507 In other words, you installed 1 package, but pkg then removed half of the packages you had on your machine (the ones with the old names that are no longer required). No, I don't think so, because that wasn't the first pkg I added. The upgrade from 132-133 is very memory intensive. Once you are past it, it should be much better. If you now install another package, does it still take 2 hours? As a matter of fact, I installed it (took about 3/4h), uninstalled (much faster) and installed again (2h). Bruno ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 04:09 PM, Jürgen Keil wrote: # tail pkg.log 1.7255schedctl() = 0xFEC69000 0.0002 sigaction(SIGINT, 0x08047650, 0x080476D0) = 0 421.4744open(/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/messages.mo, O_RDONLY) Err#2 ENOENT stack backtrace for the pkg process at the time after the sigaction() system call: core 'core.3048' of 3048: /usr/bin/python2.6 /usr/bin/pkg install diagnostic/wireshark fed85a88 update_refs (fedf3a50) + c fed8644c collect (2, fedcc63c, 80476d8, fed86e72) + 238 fed86e9c PyGC_Collect (0, fedcc63c, 80476f8, fed78c98, 0, fedcc63c) + 34 fed766e7 Py_Finalize (0, fedcc63c, 8047738, fed7791b, 0, fedcc63c) + 47 fed78c98 Py_Exit (0, fedcc63c, 8047738, fed77912) + 18 fed7791b handle_system_exit (80a3a40, fedcc63c, 8047778, fed78152, 80a3a40, 80a3a40) + 13f fed77959 PyErr_PrintEx (1, 8047af7, 101, fed77608) + 31 fed77619 PyErr_Print (80477c0, fefc47a4, 8047aff, 1, fed76c46, 0) + 1d fed772b2 PyRun_SimpleFileExFlags (fef867e8, 8047af7, 1, 804789c) + 1c2 fed76c46 PyRun_AnyFileExFlags (fef867e8, 8047af7, 1, 804789c) + 6e fed85694 Py_Main (4, 80479a8, feef6f25, 8050b88, 1, 0) + a94 08050bdf main (4, 80479a8, 80479bc, 804799c) + 63 08050aed _start (4, 8047ae4, 8047af7, 8047b04, 8047b0c, 0) + 7d Python seems to be running a garbage collection at system exit, and this touches lots of memory. And I suspect that the pkg process on your machine has a RSS available free memory. BTW when did pkg switch to python 2.6 ? ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 04:09 PM, Jürgen Keil wrote: # tail pkg.log 1.7255schedctl() = 0xFEC69000 0.0002 sigaction(SIGINT, 0x08047650, 0x080476D0) = 0 421.4744open(/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/messages.mo, O_RDONLY) Err#2 ENOENT stack backtrace for the pkg process at the time after the sigaction() system call: core 'core.3048' of 3048: /usr/bin/python2.6 /usr/bin/pkg install diagnostic/wireshark fed85a88 update_refs (fedf3a50) + c fed8644c collect (2, fedcc63c, 80476d8, fed86e72) + 238 fed86e9c PyGC_Collect (0, fedcc63c, 80476f8, fed78c98, 0, fedcc63c) + 34 fed766e7 Py_Finalize (0, fedcc63c, 8047738, fed7791b, 0, fedcc63c) + 47 fed78c98 Py_Exit (0, fedcc63c, 8047738, fed77912) + 18 fed7791b handle_system_exit (80a3a40, fedcc63c, 8047778, fed78152, 80a3a40, 80a3a40) + 13f fed77959 PyErr_PrintEx (1, 8047af7, 101, fed77608) + 31 fed77619 PyErr_Print (80477c0, fefc47a4, 8047aff, 1, fed76c46, 0) + 1d fed772b2 PyRun_SimpleFileExFlags (fef867e8, 8047af7, 1, 804789c) + 1c2 fed76c46 PyRun_AnyFileExFlags (fef867e8, 8047af7, 1, 804789c) + 6e fed85694 Py_Main (4, 80479a8, feef6f25, 8050b88, 1, 0) + a94 08050bdf main (4, 80479a8, 80479bc, 804799c) + 63 08050aed _start (4, 8047ae4, 8047af7, 8047b04, 8047b0c, 0) + 7d Python seems to be running a garbage collection at system exit, and this touches lots of memory. And I suspect that the pkg process on your machine has a RSS available free memory. very probably, I see rss of pkg growing over 300M... so with Xorg + gnome... ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
BTW when did pkg switch to python 2.6 ? It has been using python 2.6 for several builds now. I have an old b129 installation and that is using python 2.6 for /usr/bin/pkg. -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On Sat, Feb 27, 2010 at 4:50 PM, Jürgen Keil jrgn.k...@googlemail.com wrote: I ran pkg install with truss, in the hope of discovering why it takes so long to complete, especially AFTER it has reported that every thing is installed. You are running a fresh install of b133, correct? Has pkg been always that slow? I suspect that pkg has been reasonably fast for the first few packages that you installed, but has become slower and slower the more packages have been installed? Hello Mr. Keil: If you suspect a standard bin/pkg to run ^reasonably fast ^, then all your systems must be Quad core i7´s with 8GB ram. Is this the new minimum requirement for 2010.03? If you are a regular follower of pkg-discuss and caiman-discuss, then you should know the memory consumption peak values and associated problems. What you meant in this context was probably ^RELAtively fast ^, but that is a difference. The rela- then only refers to other IPS pkg- testruns themselves, rather than to any truly ^reasonably fast ^ packaging system. It seems you only run OpenSolaris on state-of-the-art hardware. If you ever tried it on a Pentium III with 1024MB ram recently, then you would have chose a different wording in the first place. Let alone trying to use it on UltraSPARC II/IIi/IIe/IIe+ workstations such as on the Blade 150. Which other packaging systems did you compare pkg to? Did you compare it to e.g. Yum, Apt, Smart or Conary? Or more directly, did you try BeleniX or Nexenta? I recommend you Package management meta-tools: survey and state of the art http://www.mancoosi.org/edos/manager/ and http://moinakg.wordpress.com/2008/11/22/the-belenix-package-manager/ But most of all: To simply try a few alternative packaging frameworks. You should enjoy the little difference :) IPS is an anti-investment. Like a black hole. ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 05:54 AM, casper@sun.com wrote: On Sat, Feb 27, 2010 at 12:48 PM,casper@sun.com wrote: well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). iostat -D 2 shows my hd busy at around 100% iosnoop reports a lot of access by pkg : How much memory? =A0It is possible that you're paging to death. Hi Casper, I am subscribed to pkg and caiman. Maybe his system is swapping to death. But why??? If you compare memory- and CPU- usage of IPS to those values of *every* other system on the market, it is unbelievable at the first look. ^Horrific^ - to find the right term. Yes, that is true. I'm not sure why it needs so much memory; it is possible that using python contributes to the memory use. Both perl and python of wonderful primitives but you pay for how the primitives are implemented: they don't give you a choice. A long time ago, we re-wrote the contents file database in a (SQL) database; that was a bad idea for several reasons: you have again no choice on how the data is stored and this again added to huge memory pressure and apart from other issues such as not properly sorting the contents file, it made installing much slower and not faster. The contents file was then around 15-20MB, the database used 512MB or more. As a gentle reminder, there's a reason why the attributes section of man pkg.1 indicates that it is In Development... The memory usage issues are well known and they are being worked on. As an in-development project, the memory usage has gone up and down repeatedly over many months. Memory usage is actually down overall in recent history and will continue to go down as additional changes are made. With that said, recent changes in the package graph (due to renaming) have unfortunately significantly increased memory pressure in some scenarios. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 05:57 PM, Jürgen Keil wrote: I suspect that pkg has been reasonably fast for the first few packages that you installed, but has become slower and slower the more packages have been installed? Not that much packages installed after fress install That is, on your fresh b133 install from livecd, the first package install pkg install diagnostic/wireshark is already slow in the post install phase (slow: more than 10 minutes) ? well, it was not the first package I installed on my fresh install, but it was the first time I installed THIS package. What process size is listed by prstat or top for the pkg process during the post install phase? (In top I observe 180M during the install phase, and 390M post install, on a fresh b133 install, when I install wireshark as the first package) I did not check but I suspect the same figures more or less from what I saw on other packages I installed. As a matter of fact I had also thunderbird+firefox so the memory usage was a bit high. I just retried it now without any other heavy app other than X+gnome, and it went much much faster (7-8mn) I noticed as well that arc cache was much lower than the last time (100M vs 180M). I suspect an interaction between pkg memory consumption and zfs arc_cache. Bruno ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On Sat, Feb 27, 2010 at 7:47 PM, Shawn Walker swal...@opensolaris.org wrote: On 02/27/10 05:54 AM, casper@sun.com wrote: On Sat, Feb 27, 2010 at 12:48 PM,casper@sun.com wrote: well the whole install took 2 hours for 1 pkg !! it seems that disk read/write here is the bottleneck (?). iostat -D 2 shows my hd busy at around 100% iosnoop reports a lot of access by pkg : How much memory? =A0It is possible that you're paging to death. Hi Casper, I am subscribed to pkg and caiman. Maybe his system is swapping to death. But why??? If you compare memory- and CPU- usage of IPS to those values of *every* other system on the market, it is unbelievable at the first look. ^Horrific^ - to find the right term. Yes, that is true. I'm not sure why it needs so much memory; it is possible that using python contributes to the memory use. Both perl and python of wonderful primitives but you pay for how the primitives are implemented: they don't give you a choice. A long time ago, we re-wrote the contents file database in a (SQL) database; that was a bad idea for several reasons: you have again no choice on how the data is stored and this again added to huge memory pressure and apart from other issues such as not properly sorting the contents file, it made installing much slower and not faster. The contents file was then around 15-20MB, the database used 512MB or more. As a gentle reminder, there's a reason why the attributes section of man pkg.1 indicates that it is In Development... Cheers, -- Shawn Walker Forgive me the *lol* I knew that this comes (I even publicly predicted it). But then you should not ship this in a commercially supported product. Just a thought. ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 11:39 AM, Martin Bochnig wrote: ... Package management meta-tools: survey and state of the art http://www.mancoosi.org/edos/manager/ Which is from a survey dated March 2006. I would note though that the results of this survey have been considered during pkg(5) implementation. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
Has pkg been always that slow? I suspect that pkg has been reasonably fast for the first few packages that you installed, but has become slower and slower the more packages have been installed? Hello Mr. Keil: If you suspect a standard bin/pkg to run ^reasonably fast ^, then all your systems must be Quad core i7´s with 8GB ram. If you are a regular follower of pkg-discuss and caiman-discuss, then you should know the memory consumption peak values and associated problems. What you meant in this context was probably ^RELAtively fast ^, but that is a difference. The rela- then only refers to other IPS pkg- testruns themselves, rather than to any truly ^reasonably fast ^ packaging system. The worst install time I've been able to reproduce so far with pkg install wireshark was 3 - 4 minutes. I tried a virtualbox guest with 768 MB of memory (single core); and I tried on a core2 duo laptop with grub uppermem 1024000, The worst time I've seen to far is an order of magnitude better than the 2 hours reported in this thread. In my tests the pkg process consumes 340 MB (which is a *lot* for installing a single package) but I expect that such a process size is ok for a system with 1GB of ram. So I think some important piece of information is missing for reproducing the bug. For example the system might use part of system ram for video ram so that less than 1GB of ram was available; or lots of other applications where running consuming memory; or the process size of pkg is much bigger than 340 MB (because lots of packages have already been installed ?) ... -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
As a matter of fact I had also thunderbird+firefox so the memory usage was a bit high. I just retried it now without any other heavy app other than X+gnome, and it went much much faster (7-8mn) Since b132 there is a java vm process running on the gnome desktop. It displays an icon in the gnome panel showing a tooltip text that all system resources are healty. You may want to stop that java process to some save 50+ MB of memory. -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 02:24 PM, Jürgen Keil wrote: Has pkg been always that slow? I suspect that pkg has been reasonably fast for the first few packages that you installed, but has become slower and slower the more packages have been installed? Hello Mr. Keil: If you suspect a standard bin/pkg to run ^reasonably fast ^, then all your systems must be Quad core i7´s with 8GB ram. If you are a regular follower of pkg-discuss and caiman-discuss, then you should know the memory consumption peak values and associated problems. What you meant in this context was probably ^RELAtively fast ^, but that is a difference. The rela- then only refers to other IPS pkg- testruns themselves, rather than to any truly ^reasonably fast ^ packaging system. The worst install time I've been able to reproduce so far with pkg install wireshark was 3 - 4 minutes. I tried a virtualbox guest with 768 MB of memory (single core); and I tried on a core2 duo laptop with grub uppermem 1024000, The worst time I've seen to far is an order of magnitude better than the 2 hours reported in this thread. In my tests the pkg process consumes 340 MB (which is a *lot* for installing a single package) but I expect that such a process size is ok for a system with 1GB of ram. And for the record, it's not supposed to take that much. Right now, the amount of memory taking is a function of the number of packages installed on your system + configured publishers + available packages. Cheers, -- Shawn Walker ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
Ok, install OpenSolaris on a server, create a zone, and then in your first zone just running this very basic IPS command: pkg install SUNWman takes forever because it downloads each man page file individually. Downloading a large number of small files via FTP or whatever will always be slower than downloading one really big file over a TCP/IP network, and this is the achilles heel of IPS. apt-get on Nexenta is really kicking OpenSolaris IPS's butt right now in terms of performance and this is a real apples to apples comparison because both Nexenta and Oracle OpenSolaris both use the same SunOS / unix / genunix kernel, the same ZFS file system, etc. so the problem is obviously with the way IPS is implemented and not with the Solaris kernel itself (which has blazing fast performance as seen on Nexenta and Milax). Where are all the dtrace performance improvement people hiding? Perhaps they can use dtrace to explain why apt-get on Nexenta's apt-get is so much faster, since Nexenta also has dtrace? Might be a good topic for a sun.com blog or paper write-up. -- This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] very long wait in the end of pkg install
On 02/27/10 10:29 PM, Anon Y Mous wrote: Ok, install OpenSolaris on a server, create a zone, and then in your first zone just running this very basic IPS command: pkg install SUNWman takes forever because it downloads each man page file individually. Downloading a large number of small files via FTP or whatever will always be slower than downloading one really big file over a TCP/IP network, and this is the achilles heel of IPS. The fact that a majority of man pages are delivered in a single package is a package bug: http://defect.opensolaris.org/bz/show_bug.cgi?id=1964 With that said, yes, the individual file retrieval system currently used by pkg(5) does have tradeoffs in transfer performance depending on the scenario. In the initial install case, a single pre-generated bundle would be better for transfers, although not nearly as much as some might believe. However, a per-file based retrieval system has advantages over one that relies on pre-generated bundles of package content. Specifically, it enables additional functionality that would otherwise be costly (resource-wise) or not as practical, such as: * multi-variant (and facet) packages -- Allows package creators to determine package boundaries using delivered functionality (content) instead of being forced to use architecture or zone variations. -- Enhances the user experience by simplifying the management of packages on a system. -- Makes it possible for a user to change between or add additional sets of functionality to their system efficiently. For example, change from x86 to SPARC, downloading only the files that have changed. * efficient update operations -- Because pkg(5) only retrieves the files that have changed between package versions between updates, the client only downloads exactly what it needs to perform update operations. Other systems have chosen to implement this by pre-generating deltas between package versions, but that also means that their users have to rely on a pre-generated delta being available for every possible origin point from which they need to update from and to. * efficient repair operations -- Because pkg(5) only has to retrieve the individual files it needs to perform an operation, this greatly reduces the time needed to restore missing package content on a system. * greatly reduced publication resource costs -- Because pkg(5) stores package content as individual files (by content hash), files that are identical between packages are shared on the server. In addition, files that don't change between package versions are also shared. This can greatly reduce the resource-cost of publication storage and publication time. apt-get on Nexenta is really kicking OpenSolaris IPS's butt right now in terms of performance and this is a real apples to apples comparison because both Nexenta and Oracle OpenSolaris both use the same SunOS / unix / genunix kernel, the same ZFS file system, etc. so the problem is obviously with the way IPS is implemented and not with the Solaris kernel itself (which has blazing fast performance as seen on Nexenta and Milax). At the moment, I'm not aware of any posteriors that feel harmed. With that said, apt and pkg(5) do not have completely equivalent functionality, so it's difficult to perform an apples-to-apples comparison. As an in-devleopment project that has been around for far less years (decades?) than apt-get, I believe pkg(5) is doing pretty well. Performance work remains on-going. Where are all the dtrace performance improvement people hiding? Perhaps they can use dtrace to explain why apt-get on Nexenta's apt-get is so much faster, since Nexenta also has dtrace? Might be a good topic for a sun.com blog or paper write-up. When you look for problems, you'll find them. But I'd also say, if you don't look for *improvements*, you won't find them either. Quite frankly, when you say things like is so much faster without exactly quantifying what you're talking about, it tends to look a little vague and hand-wavy. If you look back at the bug entries for pkg(5) or the mailing list discussions, you'll find that a significant amount of time during the last release cycle has been spent on improving performance. In addition, every time a major change was made, performance has always been part of the discussion. I personally have spent easily a few weeks worth of time dtrace profiling parts of the package system and many more using other methods of profiling. And I know other team members have spent a lot of time doing performance analysis as well. For example, the performance of pkg info and pkg list has greatly improved: * pkg info on 2009.06: 3 seconds - 2010.x: 0.3 seconds * pkg list on 2009.06: 5.4 seconds - 2010.x: 0.57 seconds * pkg list -as on 2009.06: 33 seconds - 2010.x: 3.56 seconds Likewise, Johansen replaced pkg(5)'s existing transport mechanism that relied on core
