Re: New tool to (quickly) check for available package upgrades
On Wed, Jun 17, 2020 at 09:12:08PM -, Stuart Henderson wrote: > On 2020-06-17, Marc Espie wrote: > > The only way you end up with broken installations is when porters don't do > > their jobs, that is they fail to bump a shared library or something like > > that. > > They do still break in some cases: > > libA depends on libB > someapp depends on libA, libB > > libB has a major bump. libA gets updated but someapp is missed > due to a mirror with an incomplete update. Now someapp wants old libB, > but libA wants new libB, resulting in breakage. Ouchie... that one I can't do much about. :( > (There are also situations where some installed packages are broken > for part of a pkg_add -u run, though they do sort themselves out later - > I forget the details but I think it was something to do with the timing > of ldconfig runs, updates to things like glib2/pango often do this). ldconfig is run just before actually exec'ing anything that might depend on it. Have you run into the problem lately ? I think it's mostly gone with @tags, since it delays running most things until the end of pkg_add -u, so that things have actually settled down.
Re: New tool to (quickly) check for available package upgrades
On Wed, Jun 17, 2020 at 09:12:08PM -, Stuart Henderson wrote: > This is already a problem when pkg_add fetches the directory listing > (though a smaller one because the filenames don't change as often). > > Firstly the contents of the mirror can change during the pkg_add run > so the listing becomes invalid, that can happen on any mirror. > > Secondly if the mirror involves a cache (CDNs) they will often cache > the directory listing as an object - the other files in the directory > can be out of sync with regard to the served index page. As the index is > not too big and is hit on every pkg_add run against that mirror, it's > highly likely to be cached, more so than probably anything else in the > directory except the quirks package. Yes, I've run into this one as well. One more reason to hate CDNs I have on my list to check about ignoring the index in some cases. Namely: packages have the actual names of the dependencies they were built against built-in. It should be possible and not that hard to try grabbing that package, even if it didn't appear in the index, if it appears to be more recent than what the index says.
Re: New tool to (quickly) check for available package upgrades
On 2020-06-17, Jeremy O'Brien wrote: >> What if a new batch of amd64/i386 files appears while one of the ongoing >> syncs run, do you restart over and hope yet another new one doesn't appear >> while that one is running? > > Is this something that actually happens? These 4 arches all take <3 days or so for a package build: 39.9G i386 46.4G amd64 34.5G aarch64 25.4G sparc64 146Gtotal Then there are files from base plus the other slower package arches, and stable packages and releases every now and again. Not particularly carefully calculated but if a mirror can't sustain an average of somewhere around 6Mb/s for fetches (bearing in mind for many mirrors these are fairly high latency links and the tuning of OpenBSD's tcp stack isn't the best in the world for coping with this) they aren't going to keep up. And of course even if they normally cope then a network problem somewhere can drop it below the speed needed to keep in sync.
Re: New tool to (quickly) check for available package upgrades
On 2020-06-17, Marc Espie wrote: > The only way you end up with broken installations is when porters don't do > their jobs, that is they fail to bump a shared library or something like > that. They do still break in some cases: libA depends on libB someapp depends on libA, libB libB has a major bump. libA gets updated but someapp is missed due to a mirror with an incomplete update. Now someapp wants old libB, but libA wants new libB, resulting in breakage. (There are also situations where some installed packages are broken for part of a pkg_add -u run, though they do sort themselves out later - I forget the details but I think it was something to do with the timing of ldconfig runs, updates to things like glib2/pango often do this). >> Even with snapshot shearing though, having this index file could provide a >> substantial speed upgrade. Instead of having to check *all* installed >> package's header for updates, you could use the index to know the subset of >> packages that you expect to have actually changed, and only download *those* >> packages' headers. If the expected "combined" sha of a given package doesn't >> match the index's version, then the mirror is clearly out of sync and we >> could abort an update as usual. > > > The problem is the multiple RTT...! if you manage to keep one single > connection open, you get a substantial speed upgrade. > > Generating a single index is more problematic than it would seem at first o > glance. This is already a problem when pkg_add fetches the directory listing (though a smaller one because the filenames don't change as often). Firstly the contents of the mirror can change during the pkg_add run so the listing becomes invalid, that can happen on any mirror. Secondly if the mirror involves a cache (CDNs) they will often cache the directory listing as an object - the other files in the directory can be out of sync with regard to the served index page. As the index is not too big and is hit on every pkg_add run against that mirror, it's highly likely to be cached, more so than probably anything else in the directory except the quirks package. > When everything goes fine, you're happy. > > If anything is out-of-synch, you're in deep shit. Sometimes things get in a state where a mirror can't get a complete sync at all, we've had times where it's been many days / week+ on fanout->L2s so nobody gets updates. One question is what to do in that situation. When it's like that, depending on what changed, updating any packages at all can cause quite bad breakage. > This would mean having several possible modes to cope with that, we don't > have enough resources to do that. > >
Re: New tool to (quickly) check for available package upgrades
Jeremy O'Brien wrote: > > What if a new batch of amd64/i386 files appears while one of the > > ongoing syncs run, do you restart over and hope yet another new one > > doesn't appear while that one is running? > > Is this something that actually happens? Yes. Apparently you don't like reading.
Re: New tool to (quickly) check for available package upgrades
On Wed, Jun 17, 2020, at 13:45, Janne Johansson wrote: > Do think of what you call "the index file" in terms of "I check/replace some > 100+G of snapshots and packages every 24h", at which point do you replace > that single file, before, under or after none,most,all packages for your arch > are replaced? Will it be synced when the copying passes "i" for "index.txt" > in that packages folder? My gut is it doesn't matter when the file is sync'd. The file would only be used to prevent pkg_add from checking *every* installed package file *every* time. The effect would be basically the same as how my tool works, which is, it gives you a subset of installed packages to update, except with the added benefit of also detecting same-version rebuilds. pkg_add would still go through its normal recursive dependency checking, just with a (presumably) *much* shorter package list. > What happens if a sync gets cut off, restarted and/or if two syncs suddenly > run into eachother and replace files as they go? > > What if a new batch of amd64/i386 files appears while one of the ongoing > syncs run, do you restart over and hope yet another new one doesn't appear > while that one is running? Is this something that actually happens?
Re: New tool to (quickly) check for available package upgrades
On 2020-06-17, Jeremy O'Brien wrote:
> On Wed, Jun 17, 2020, at 13:45, Janne Johansson wrote:
>>
>> Now if someone invents a decent piece of code to use http connection
>> pooling, quic/http3/rsync or whatever to speed up getting the required info,
>> I'm sure we mirror admins would be happy to add/edit our server programs to
>> serve it.
>>
>
> Welp, I got a nice taste of what taking advantage of http-keepalives can do,
> and it is indeed quite a bit faster. Here's all I did (a bit overkill, but a
> neat proof-of-concept):
>
> # pkg_add squid
> # rcctl start squid
> # export http_server=http://localhost:3128/
> # pkg_add -u
>
> squid by default uses keepalives on both the client and server connections.
> It seems that the server closes our connections several times during the
> transfers anyway (I assume either a max request limit or keepalive timeout),
> but it's much better than starting a new connection every time. I'd be really
> interested to see what something like quic can do.
>
Or a bit lighter than squid, you can do something like this in nginx as a
reverse-proxy:
upstream live_pkgcdn {
server cdn.openbsd.org:80;
keepalive 2;
}
server {
server_name xxx;
listen 80;
location ~ /pub/OpenBSD {
proxy_pass http://live_pkgcdn;
proxy_set_header Host "cdn.openbsd.org";
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
Re: New tool to (quickly) check for available package upgrades
On Wed, Jun 17, 2020, at 13:45, Janne Johansson wrote: > > Now if someone invents a decent piece of code to use http connection pooling, > quic/http3/rsync or whatever to speed up getting the required info, I'm sure > we mirror admins would be happy to add/edit our server programs to serve it. > Welp, I got a nice taste of what taking advantage of http-keepalives can do, and it is indeed quite a bit faster. Here's all I did (a bit overkill, but a neat proof-of-concept): # pkg_add squid # rcctl start squid # export http_server=http://localhost:3128/ # pkg_add -u squid by default uses keepalives on both the client and server connections. It seems that the server closes our connections several times during the transfers anyway (I assume either a max request limit or keepalive timeout), but it's much better than starting a new connection every time. I'd be really interested to see what something like quic can do.
Re: New tool to (quickly) check for available package upgrades
Den ons 17 juni 2020 kl 17:04 skrev Marc Espie : > > > > > > The concept you need to understand is snapshot shearing. > > > > > A full package snapshot is large enough that it's hard to > guarantee that > > > > > you will have a full snapshot on a mirror at any point in time. > > > > > In fact, you will sometimes encounter a mix of two snapshots (not > that often, > > > > > recently, but still) > > > > > Hence, the decision to not have a central index for all packages, > but to > > > > > keep (and trust) the actual meta-info within the packages proper. > > > > > > > > Sorry, I guess I should've responded to this as well. Isn't snapshot > shearing going to be a problem regardless of the existence of a single > central-index? For instance, pkg_add notices a chromium update, which > requires a newer version of a dependency that hasn't been propagated to the > mirror yet. > > > Even with snapshot shearing though, having this index file could provide > a substantial speed upgrade. Instead of having to check *all* installed > package's header for updates, you could use the index to know the subset of > packages that you expect to have actually changed, and only download > *those* packages' headers. If the expected "combined" sha of a given > package doesn't match the index's version, then the mirror is clearly out > of sync and we could abort an update as usual. > > Do think of what you call "the index file" in terms of "I check/replace some 100+G of snapshots and packages every 24h", at which point do you replace that single file, before, under or after none,most,all packages for your arch are replaced? Will it be synced when the copying passes "i" for "index.txt" in that packages folder? What happens if a sync gets cut off, restarted and/or if two syncs suddenly run into eachother and replace files as they go? What if a new batch of amd64/i386 files appears while one of the ongoing syncs run, do you restart over and hope yet another new one doesn't appear while that one is running? This is the reality of snapshot package today: du -sh snapshots/packages/* 34.5G snapshots/packages/aarch64 52.4G snapshots/packages/amd64 18.4G snapshots/packages/arm 44.1G snapshots/packages/i386 24.1G snapshots/packages/mips64 10.2G snapshots/packages/mips64el 26.7G snapshots/packages/powerpc 25.4G snapshots/packages/sparc64 Whatever limitations Marcs design has, it makes it possible for us mirror admins to sync with some kind of best-effort while still giving most openbsd users the ability to have pkg_add -u leave you with a working package eco-system on a daily basis. If the cost is that it takes 40 minutes at night from crontab, then I would not trade a greppable file for losing some or a lot of the above-mentioned gotchas that the current system somehow actually handles. Now if someone invents a decent piece of code to use http connection pooling, quic/http3/rsync or whatever to speed up getting the required info, I'm sure we mirror admins would be happy to add/edit our server programs to serve it. -- May the most significant bit of your life be positive.
Re: New tool to (quickly) check for available package upgrades
On Wed, Jun 17, 2020 at 09:44:32AM -0400, Jeremy O'Brien wrote: > On Wed, Jun 17, 2020, at 08:47, Marc Espie wrote: > > On Wed, Jun 17, 2020 at 08:28:02AM -0400, Jeremy O'Brien wrote: > > > On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > > > > > > > The concept you need to understand is snapshot shearing. > > > > > > > > A full package snapshot is large enough that it's hard to guarantee that > > > > you will have a full snapshot on a mirror at any point in time. > > > > > > > > In fact, you will sometimes encounter a mix of two snapshots (not that > > > > often, > > > > recently, but still) > > > > > > > > Hence, the decision to not have a central index for all packages, but to > > > > keep (and trust) the actual meta-info within the packages proper. > > > > > > > > > > Sorry, I guess I should've responded to this as well. Isn't snapshot > > > shearing going to be a problem regardless of the existence of a single > > > central-index? For instance, pkg_add notices a chromium update, which > > > requires a newer version of a dependency that hasn't been propagated to > > > the mirror yet. > > > > > That's not a big problem, it just stops before updating... at least > > your installation doesn't get hosed. > > > > Sorry if I'm being dense here. I'm just trying to understand this completely. > > In the current implementation, does pkg_add recursively verify every > dependency of a package before installing *any* of them? Or does it install > dependency updates as it finds them? Because if it's the latter, I can still > envision scenarios that break a given package, such as: Package A depends on > B and C. B is sync'd, C is not yet. pkg_add installs the update to B, then > encounters the missing C update, and bails the upgrade of A. Now you have A > installed which doesn't work with the new version of B. Is this something > that can currently happen? Nope. That's why we have shared libraries numbers. We keep the old libraries around, so packages will keep working. In the rare case where some packages have tight dependencies (see cups and cups-libs for instance), pkg_add will update both at the same time. An update is done in two steps: first extract the new package(s) to temporary files, THEN delete the old package and move the temporary files in final location. The only way you end up with broken installations is when porters don't do their jobs, that is they fail to bump a shared library or something like that. > Even with snapshot shearing though, having this index file could provide a > substantial speed upgrade. Instead of having to check *all* installed > package's header for updates, you could use the index to know the subset of > packages that you expect to have actually changed, and only download *those* > packages' headers. If the expected "combined" sha of a given package doesn't > match the index's version, then the mirror is clearly out of sync and we > could abort an update as usual. The problem is the multiple RTT...! if you manage to keep one single connection open, you get a substantial speed upgrade. Generating a single index is more problematic than it would seem at first o glance. When everything goes fine, you're happy. If anything is out-of-synch, you're in deep shit. This would mean having several possible modes to cope with that, we don't have enough resources to do that.
Re: New tool to (quickly) check for available package upgrades
On Wed, Jun 17, 2020, at 08:47, Marc Espie wrote: > On Wed, Jun 17, 2020 at 08:28:02AM -0400, Jeremy O'Brien wrote: > > On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > > > > > The concept you need to understand is snapshot shearing. > > > > > > A full package snapshot is large enough that it's hard to guarantee that > > > you will have a full snapshot on a mirror at any point in time. > > > > > > In fact, you will sometimes encounter a mix of two snapshots (not that > > > often, > > > recently, but still) > > > > > > Hence, the decision to not have a central index for all packages, but to > > > keep (and trust) the actual meta-info within the packages proper. > > > > > > > Sorry, I guess I should've responded to this as well. Isn't snapshot > > shearing going to be a problem regardless of the existence of a single > > central-index? For instance, pkg_add notices a chromium update, which > > requires a newer version of a dependency that hasn't been propagated to the > > mirror yet. > > > That's not a big problem, it just stops before updating... at least > your installation doesn't get hosed. > Sorry if I'm being dense here. I'm just trying to understand this completely. In the current implementation, does pkg_add recursively verify every dependency of a package before installing *any* of them? Or does it install dependency updates as it finds them? Because if it's the latter, I can still envision scenarios that break a given package, such as: Package A depends on B and C. B is sync'd, C is not yet. pkg_add installs the update to B, then encounters the missing C update, and bails the upgrade of A. Now you have A installed which doesn't work with the new version of B. Is this something that can currently happen? I'm just trying to understand how the index causes the possibility of *more* breakage when compared to the current system. Even with snapshot shearing though, having this index file could provide a substantial speed upgrade. Instead of having to check *all* installed package's header for updates, you could use the index to know the subset of packages that you expect to have actually changed, and only download *those* packages' headers. If the expected "combined" sha of a given package doesn't match the index's version, then the mirror is clearly out of sync and we could abort an update as usual.
Re: New tool to (quickly) check for available package upgrades
On Wed, Jun 17, 2020 at 08:28:02AM -0400, Jeremy O'Brien wrote: > On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > > > The concept you need to understand is snapshot shearing. > > > > A full package snapshot is large enough that it's hard to guarantee that > > you will have a full snapshot on a mirror at any point in time. > > > > In fact, you will sometimes encounter a mix of two snapshots (not that > > often, > > recently, but still) > > > > Hence, the decision to not have a central index for all packages, but to > > keep (and trust) the actual meta-info within the packages proper. > > > > Sorry, I guess I should've responded to this as well. Isn't snapshot shearing > going to be a problem regardless of the existence of a single central-index? > For instance, pkg_add notices a chromium update, which requires a newer > version of a dependency that hasn't been propagated to the mirror yet. > That's not a big problem, it just stops before updating... at least your installation doesn't get hosed.
Re: New tool to (quickly) check for available package upgrades
On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > The concept you need to understand is snapshot shearing. > > A full package snapshot is large enough that it's hard to guarantee that > you will have a full snapshot on a mirror at any point in time. > > In fact, you will sometimes encounter a mix of two snapshots (not that often, > recently, but still) > > Hence, the decision to not have a central index for all packages, but to > keep (and trust) the actual meta-info within the packages proper. > Sorry, I guess I should've responded to this as well. Isn't snapshot shearing going to be a problem regardless of the existence of a single central-index? For instance, pkg_add notices a chromium update, which requires a newer version of a dependency that hasn't been propagated to the mirror yet.
Re: New tool to (quickly) check for available package upgrades
On Tue, Jun 16, 2020, at 21:02, Marc Espie wrote: > > The concept you need to understand is snapshot shearing. > > A full package snapshot is large enough that it's hard to guarantee that > you will have a full snapshot on a mirror at any point in time. > > In fact, you will sometimes encounter a mix of two snapshots (not that often, > recently, but still) > > Hence, the decision to not have a central index for all packages, but to > keep (and trust) the actual meta-info within the packages proper. > > The only way to avoid that would be to have specific tools for mirroring, > and to ask mirror sites to set aside twice as much room so that they > can rotate snapshots. > > Now, each package has all dependency information in the packing-list... > which allows pkg_add to know whether to update or not. We do a somewhat > strict check, we have tried to relax the check at some point in the past, > but this was causing more trouble than it was worth. > > The amount of data transmitted during pkg_add -u mostly depends on signify: > you have to grab at least a full signed block, which is slightly over 64KB. > > On most modern systems, the bandwidth is not an issue, the number of RTT > is more problematic. The way to speed pkg_add -u up would be to keep the > connection alive, which means ditching ftp(1) and switching to specific code > that talks modern http together with byte-ranges. > Thank you for this information. It was very helpful! So currently with my 433 packages installed, I need to transfer ~28MB of data and make the associated RTTs every time I want to know if I have updates. Multiply this by the number of people using OpenBSD that also check for package updates periodically (a completely unknown number to me), and it seems like there could be benefits to both the mirror providers and the users to optimize this a bit :) I'm spit-balling ideas to optimize this in my head, and I came up with something that seems simple and feasible to me. Please let me know if I'm way off-base here with my proposal, as I may be making false assumptions here. I noticed that there doesn't seem to be one "master" sha that hashes *all* of the files of a given package, but rather one sha per file that the package installs. I'm assuming that this is what causes us to have to download the +CONTENTS file from every installed package. Here is my proposal: maintain a single separate file in the mirrors (similar to the index.txt) where each line looks like this: got-0.36.tgzN9IEajlcv8snEv9clqcnsZZodggAR8VnFwCdu8I19WY= where the package name is business as usual, but the hash is actually a reduction of all the existing hashes from the +CONTENTS into a single hash (through concatenation, xor, probably doesn't matter that much; just pick something). Then, 'pkg_add -u' only downloads this single file as the source of truth for the current package state. The file can of course be signed + compressed as well. Does this idea seem to have any basis in reality?
Re: New tool to (quickly) check for available package upgrades
On Tue, Jun 16, 2020 at 04:59:07PM -0400, Jeremy O'Brien wrote: > Hey misc@, > > I wrote a quick little tool here: > https://github.com/neutralinsomniac/obsdpkgup in Go to show available package > upgrades from your configured mirror. > > It takes no more than a few seconds (the time it takes to download index.txt > from the package repo) to show you all packages that have received a version > bump. This tool *won't* show same-version package-rebuild upgrades, so it > shouldn't be used as a complete replacement to running 'pkg_add -u', but > rather as a companion to show when actual newer versions of packages are > released. I just noticed that in my 99% case, I was waiting anywhere from > 5-10 minutes for 'pkg_add -u' to complete checking all ~400 of my installed > packages, and it uses a considerable amount of bandwidth while doing so. > > As I understand it, the pkgtools detect same-version rebuilds by downloading > enough of every installed package tgz to check the metadata contained within > to determine if an upgrade is needed. If anyone knows of an alternative way > to determine when a same-version package install is required, I would love to > know of it. In the meantime, I hope someone else can make use of this tool as > well. The concept you need to understand is snapshot shearing. A full package snapshot is large enough that it's hard to guarantee that you will have a full snapshot on a mirror at any point in time. In fact, you will sometimes encounter a mix of two snapshots (not that often, recently, but still) Hence, the decision to not have a central index for all packages, but to keep (and trust) the actual meta-info within the packages proper. The only way to avoid that would be to have specific tools for mirroring, and to ask mirror sites to set aside twice as much room so that they can rotate snapshots. Now, each package has all dependency information in the packing-list... which allows pkg_add to know whether to update or not. We do a somewhat strict check, we have tried to relax the check at some point in the past, but this was causing more trouble than it was worth. The amount of data transmitted during pkg_add -u mostly depends on signify: you have to grab at least a full signed block, which is slightly over 64KB. On most modern systems, the bandwidth is not an issue, the number of RTT is more problematic. The way to speed pkg_add -u up would be to keep the connection alive, which means ditching ftp(1) and switching to specific code that talks modern http together with byte-ranges.
Re: New tool to (quickly) check for available package upgrades
On 2020-06-16, Jeremy O'Brien wrote: > On Tue, Jun 16, 2020, at 17:19, Daniel Jakots wrote: >> I think if I wanted to compare packages between a machine of mine and a >> mirror, I would compare the quirks package signature timestamps. On >> your machine you can find it with >> $ grep digital-signature /var/db/pkg/quirks*/+CONTENTS >> and on the mirror, you need to fetch the quirks-XXX.tgz (I guess you >> can find the XXX with the index.txt) and then look for the +CONTENTS >> file. I have this which I use sometimes. #!/bin/sh old=`pkg_info -f quirks | sed -En '/@digital-sig/ s/(.*signify2:|:external)//gp'` new=`PKG_DBDIR=/var/empty pkg_info -f quirks | sed -En '/@digital-sig/ s/(.*signify2:|:external)//gp'` if [[ $old == $new ]]; then echo "Already up-to-date: $old" else echo "Updating: $old -> $new" pkg_add -u fi > I'm not sure I'm following. What helpful information does this convey, > exactly? It's my understanding that the quirks package specifies mainly > package obsolescence and stem-changes. It's a quick way to check if a new package build is available. But it does rely on you not installing new packages without doing a pkg_add -u (because that will cause the local quirks package to be updated). > As I understand it, the pkgtools detect same-version rebuilds by > downloading enough of every installed package tgz to check the metadata > contained within to determine if an upgrade is needed. Every package is rebuilt in *every* ports build so it's not so much detecting whether there was a same-version rebuild, but whether you need to install it. > If anyone knows > of an alternative way to determine when a same-version package install > is required, I would love to know of it. That is the only way to check. The packing list at the start of each package includes the information about library versions it was built against (this is called the "signature" - unrelated to crypto signature - see it with pkg_info -S $somepkg). This is needed to decide whether you need to update that package.
Re: New tool to (quickly) check for available package upgrades
On Tue, Jun 16, 2020, at 17:19, Daniel Jakots wrote: > I think if I wanted to compare packages between a machine of mine and a > mirror, I would compare the quirks package signature timestamps. On > your machine you can find it with > $ grep digital-signature /var/db/pkg/quirks*/+CONTENTS > and on the mirror, you need to fetch the quirks-XXX.tgz (I guess you > can find the XXX with the index.txt) and then look for the +CONTENTS > file. > > Cheers, > Daniel > I'm not sure I'm following. What helpful information does this convey, exactly? It's my understanding that the quirks package specifies mainly package obsolescence and stem-changes.
Re: New tool to (quickly) check for available package upgrades
On Tue, 16 Jun 2020 16:59:07 -0400, "Jeremy O'Brien" wrote: > I wrote a quick little tool here: > https://github.com/neutralinsomniac/obsdpkgup in Go to show available > package upgrades from your configured mirror. > > It takes no more than a few seconds (the time it takes to download > index.txt from the package repo) to show you all packages that have > received a version bump. This tool *won't* show same-version > package-rebuild upgrades, so it shouldn't be used as a complete > replacement to running 'pkg_add -u', but rather as a companion to > show when actual newer versions of packages are released. I just > noticed that in my 99% case, I was waiting anywhere from 5-10 minutes > for 'pkg_add -u' to complete checking all ~400 of my installed > packages, and it uses a considerable amount of bandwidth while doing > so. > > As I understand it, the pkgtools detect same-version rebuilds by > downloading enough of every installed package tgz to check the > metadata contained within to determine if an upgrade is needed. If > anyone knows of an alternative way to determine when a same-version > package install is required, I would love to know of it. In the > meantime, I hope someone else can make use of this tool as well. I think if I wanted to compare packages between a machine of mine and a mirror, I would compare the quirks package signature timestamps. On your machine you can find it with $ grep digital-signature /var/db/pkg/quirks*/+CONTENTS and on the mirror, you need to fetch the quirks-XXX.tgz (I guess you can find the XXX with the index.txt) and then look for the +CONTENTS file. Cheers, Daniel
New tool to (quickly) check for available package upgrades
Hey misc@, I wrote a quick little tool here: https://github.com/neutralinsomniac/obsdpkgup in Go to show available package upgrades from your configured mirror. It takes no more than a few seconds (the time it takes to download index.txt from the package repo) to show you all packages that have received a version bump. This tool *won't* show same-version package-rebuild upgrades, so it shouldn't be used as a complete replacement to running 'pkg_add -u', but rather as a companion to show when actual newer versions of packages are released. I just noticed that in my 99% case, I was waiting anywhere from 5-10 minutes for 'pkg_add -u' to complete checking all ~400 of my installed packages, and it uses a considerable amount of bandwidth while doing so. As I understand it, the pkgtools detect same-version rebuilds by downloading enough of every installed package tgz to check the metadata contained within to determine if an upgrade is needed. If anyone knows of an alternative way to determine when a same-version package install is required, I would love to know of it. In the meantime, I hope someone else can make use of this tool as well. Thanks! Jeremy
