Hi Shawn, thank you for clarifying those points. Since it seems that the installer will need to consume directly pkg(5) Python API at least for some pkg(5) functionality (e.g. bug 4546, 7736), as it is to be exposed only by Python API, I think it makes sense to switch from pkg(1) CLI to Python API (in order to use consistent approach). I have filed bug 13851 for this.
Thank you, Jan On 01/12/10 04:41 PM, Shawn Walker wrote: > On 01/12/10 02:10 AM, Jan Damborsky wrote: >> Hi Shawn, >> >> >> On 01/11/10 05:39 PM, Shawn Walker wrote: >>> On 01/11/10 09:41 AM, Fr?d?ric VANNIERE wrote: >>>>> Another thing that might be contributing to this is how AI is >>>>> performing the installs. >>>>> Can you provide the full install log? >>>> >>>> It's here : http://pastebin.com/m3c26ca80 >>>> >>>> Note: The PWKinstall-opensolaris package is downloaded from a custom >>>> IPS repository and is post-install my customization script (I'll post >>>> about this latter) . >>> >>> That helps quite a bit; AI appears to be executing the pkg command to >>> install each package separately and for other tasks (over 90 times). >> >> Yep - that is how AI currently installs packages listed in AI manifest. >> >>> >>> That's going to be much slower than simply executing the pkg(1) >>> command once for all packages to be installed. Not only that, AI >>> should really be using the python pkg.client.api that's provided >>> instead as that would further reduce the overhead of the operations it >>> is performing. >> >> >> That is interesting - thank you for pointing this out. >> Do you happen to have more data about what the difference is >> with respect to the overhead of particular approaches you mentioned: >> >> * calling pkg(1) for each package > > If you call pkg(1) for each package, you are significantly increasing > install times due to the following: > > * search indices update cannot be coalesced for the set of packages > being installed > > * dependency solver overhead means you pay a higher price for > individual install instead of a single install > > * transport system depends upon observed performance metrics per > session to determine optimum path and relies on persistent connections > to optimise file transfer > > * python interpreter startup cost > >> * calling pkg(1) for all packages at once > > This is the best option if you cannot use the API. > >> * consuming directly pkg.client.api > > Specific benefits of using the API include: > > * access to additional information that is not easily > available/parseable via the CLI > > * ability to control various transport retry/timeout errors > > * ability to perform granular error-case handling (far more expressive > than the limited exit code failures provided by cli) > > * ability to retrieve package architecture and zone variant information > > * in the future, ability to retrieve package size information before > plan preparation or execution > >> We are mostly interested in memory consumption as well as well >> time execution. >> >> As far as stability of those APIs is concerned, are there any >> significant differences between pkg(1) and pkg.client.api ? >> I mean if we switch to pkg.client.api, should we assume >> to be hit by API changes more often than we currently are >> with pkg(1) ? > > More often, yes, but the changes should all be documented in > doc/client_api_versions.txt in the pkg gate. And even though changes > might affect you more often, it is usually a span of at least weeks > and sometimes months before it changes incompatibly. > > The pkg.client.api interface is the intended interface for external > consumers. > > The interface is also versioned, so if it changes incompatibly, your > program should fail as soon as it tries to load the API; not > mysteriously somewhere in the middle. > >> Also, pkg(1) man page is useful source of information and often >> used as API documentation, is pkg.client.api API documented >> somewhere as well ? > > See 'pydoc pkg.client.api' and 'pydoc pkg.api_common'. Note that you > need to use the python2.6 version of pydoc as of b129. > > For example: > > /usr/lib/python2.6/pydoc.py pkg.client.api >
