Hi all,
Catching up on the conversation...impressive results coming out of apk.
Haven't used apk enough to have an educated opinion, but by quickly
glancing at the code, I believe it uses an ad hoc solver, which might
struggle with corner cases (same issues we had with opkg, before
switching to libsolv). For building images without package managers, its
likely that this isn't a serious concern.
Regarding opkg, I haven't focus on build time so I get the feeling there
might be some low hanging fruit. I plan to spend some time investigating
and will report back. The mechanism used by apk (1 read, 1 write) can be
added to opkg too...in general the idea with opkg is that you set it up
with the trade off that make sense for you. So I can envision having a
config option that enables apk-like behavior.
Regardless, its exciting to see changes that expand the ecosystem!
Cheers,
Alejandro
On 6/30/20 5:12 PM, Fredrik Gustafsson wrote:
Hi Martin, apk is quite different compared to other package managers.
A traditional package manager download a package, saves it to a local
cache, verifies it and then extract it. That's at least 3 reads and 2
writes of the data. Apk does this on the fly, so 1 read and 1 write
of the data. I believe that it's a so fundamental design choice that
opkg would have it very hard to compete for speed. The speed
advantage is even bigger when installing over network since the
extraction and verification is done wile the network is waiting for
data. That means that apk has usually installed the package when an
ordinary package manager just has downloaded it and should start to
verify and install it.
So there's no secret sauce or awesome optimizations, it's "just" an
other architectural design.
With that said, there's still some low hanging fruit for apk, for
example package generation (my patch is doing 2 reads and 2 writes of
all data) and index generation (duplicate reads and one extraction
too much) that could have significant speedup. But let's make it
possible to use apk at all before starting to tune it.
More information:
https://urldefense.com/v3/__https://www.youtube.com/watch?v=sIG2P9k6EjA__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMCfLL158w$
BR Fredrik ________________________________________ From: Martin
Jansa <martin.ja...@gmail.com> Sent: Tuesday, June 30, 2020 11:54 PM
To: Fredrik Gustafsson Cc: Ross Burton; OE-core;
tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers
as a plugin
On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
Hi Ross, those 5 seconds will increase to minutes for my use case
and we build a lot hence I hope this will save us a lot of computer
power and engineering time. For example I've sent numbers on
building a bigger image (core-image-sato-sdk-ptest) to Alex and
Alex: out.apk.1: 1:13.35 out.apk.2: 1:13.51 out.apk.3: 1:13.23
out.apk.4: 1:14.07 out.apk.5: 1:13.00 out.deb.1: 3:49.37 out.deb.2:
3:50.77 out.deb.3: 3:51.39 out.deb.4: 3:53.40 out.deb.5: 3:53.99
out.ipk.1: 2:38.99 out.ipk.2: 2:39.07 out.ipk.3: 2:35.34 out.ipk.4:
2:36.15 out.ipk.5: 2:34.55 out.rpm.1: 1:58.61 out.rpm.2: 1:59.42
out.rpm.3: 1:59.70 out.rpm.4: 1:58.96 out.rpm.5: 1:58.11
Also consider that if we've a target without python with limited
space, rpm is out of the question. So the real comparison would be
between ipk and apk. Let's say we can save 80 seconds in each
build. Now multiply that with the number of builds, and we do a lot
(I guess we might approach 100 000 builds/week in the next year).
Then this will save 92.5 days build time each week.
This is impressive.
Have you tried to profile where opkg spends most of the time compared
to apk? It would be nice to know if there is something fundamentally
different in how they handle packages or if apk is just better
optimized.
Now with opkg better maintained (thanks Alejandro!) there might be
some relatively low hanging optimalizations which could be
implemented there as well (for people who didn't implement the rest
to switch to apk yet :)).
I know at LGE we have some patches (some related to performance as
well) which unfortunately weren't ever upstreamed :/, some are at:
https://urldefense.com/v3/__https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMDh3PHxpA$
Cheers,
--
Cheers,
Alejandro
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#140197):
https://lists.openembedded.org/g/openembedded-core/message/140197
Mute This Topic: https://lists.openembedded.org/mt/75057633/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
