Hi
On Thu, Oct 27, 2016 at 03:45:21PM +, rbraun204 . wrote:
> I have a couple of debian boxes in very remote areas that are connected back
> to
> our wan via a 56kbps satellite link. Most of the time we have a constant
> stream of data coming/going to that machine so the link is saturated quite a
> bit.
Ouch. I guess that the data isn't for looking at cat videos then!
> I'm having all sorts of trouble getting apt to play nicely with the extremely
> slow link. When I try to do an apt-get update, it seems to work for a while,
> then will start to download whichever list it's currently on all over
> again.
> I tried running apt-get update for about 24h and it would never completely
> download the amd64 main Packages.gz (around 7.5M). It would just keep trying
> to start over and over again. Maybe sometimes it will work, but +50% of the
> time it will crap out. Apt is configured to use a proxy server, aswell as
> http::timeout is set to 300 via apt.conf
>
> FWIW, I can reliably rsync files over the sat link without issue. It takes a
> while for sure, getting about .75 - 1.5KB/s. So the files do get there. So
> it seems like whatever magic is baked into the rsync protocol to handle these
> slow links is working alot more reliably for me then the http gets that apt is
> using. Running rsync with bwlimit will work all day I've found.
That's very odd. So TCP connections stay alive then.
> I'm currently trying to build a list of debs that the system wants using
> something like
>
> apt-get dist-upgrade --allow-unauthenticated -y --print-uris | grep -o '\
> 'http.*\' | tr "\'" " " > downloads
>
> then wget'ing them locally and rsyncing them up the remote. Seems to be
> working so far, but the last failed apt-get update seemed to blow away the
> lists on the remote and I can no longer see any pending package upgrades on
> the
> system.
>
> I've also tried tarring up /var/lib/apt/lists/* from a known working system
> and
> rsyncing that up to the remote, to try and update the lists manually I
> guess.
> But that didn't seem to work either. After dropping the list files in
> /var/lib
> /apt/lists and running apt-get dist-upgrade, still showed no pending
> updates.
> So not sure why that would be.
>
> So after all that, here are my questions :)
>
> 1. Is there some crappy link tweaks I can use in apt to help apt with
> transferring data over a 1.5KB link?
>
> 2. In theory, if I wanted to transfer the apt-get update data via rsync,
> should I be able to tar up /var/lib/apt/lists/* and send that manually? It
> didn't seem to work, but I would imagine there's more going on behind the
> scenes.
if the sources.list are identical, yes: I believe that should work.
> 3. Generally just curious what others have done when trying to keep systems
> up
> to date in very remote places with limited pipes.
>
>
> Worst case scenario, If we had to burn a cd full of debs monthly and ship it
> out to the remote I guess that would work. We also have our own custom repos
> with software that gets updated aswell. But sometimes we would need to push
> those updates out asap. Also, there is only 1 machine at each remote, so
> it's not an issue of running approx to save X machines all updating over the
> network at once.
Normally, I'd suggest looking into running a private mirror and
rsyncing it, but with only one machine at each location, that's
overkill.
I think you may want to look into apt-zip: This will decouple the
download from apt, allowing you to get the data xferred by whatever
method works (rsync?), and picking up the xferred file on the remote
location with apt...
Unfortunately, apt-zip has been discontinued - last appears in wheezy,
but it may still work or be made to work? It seems aimed at your
exact use case...
Failing that I can imagine other (hand-crafted) solutions with these
components:
- make sure /etc/sources.list (and /etc/sources.list.d) are identical
- rsync /var/lib/apt/* across
- on the remote end: run:
apt-get upgrade --print-uris # or similar
- grab the URLs
- download the *.debs and rsync them into /var/cache/apt/archives/
- on the remote end:
apt-get upgrade
This is entirely off the top of my head, but with a bit more thought
and scripting, it _should_ work...
Hope this helps
--
Karl
