rsync

[Richard Bostrom]

rsync works perfectly well i apologize for whining it was a simple permissions 
issue. oh what you have to put up with ...

Yours sincerely
Richardh Bostrom

Sent with [Proton Mail](https://proton.me/) secure email.

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Roy J. Tellason, Sr.]

On Friday 09 February 2024 04:41:37 pm hw wrote:
> On Fri, 2024-02-09 at 11:34 -0500, Roy J. Tellason, Sr. wrote:
> > On Friday 09 February 2024 06:07:16 am hw wrote:
> > > What other manufacturers could we buy UPSs from?
> >  
> > I have a Tripp-Lite sitting next to me here that replaced an APC and
> > has 2-1/2 times the capabiliity.  Been in service several weeks and
> > so far I'm pretty happy with it...
> 
> They seem to be extremely rare here.  

Where's "here"?  I ordered mine from Home Depot,  online.  The wait until it 
arrived didn't seem excessive.

> Are they any good, and how's the battery availability?

It seems okay,  and I haven't checked on the battery availability,  no need at 
this point and if I did it'd probably change by the time I needed them anyhow. 



-- 
Member of the toughest, meanest, deadliest, most unrelenting -- and
ablest -- form of life in this section of space,  a critter that can
be killed but can't be tamed.  --Robert A. Heinlein, "The Puppet Masters"
-
Information is more dangerous than cannon to a society ruled by lies. --James 
M Dakin

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Dan Ritter]

hw wrote: 
> On Fri, 2024-02-09 at 06:44 -0500, Dan Ritter wrote:
> > hw wrote: 
> > > On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote:
> > > > [...]
> > > That sucks.  I didn't know that they don't stand behind their
> > > products, and it makes APC not recommendable any longer.
> > > 
> > > What other manufacturers could we buy UPSs from?
> > 
> > Liebert at the high end, CyberPower at the low end. 
> 
> I've never heard of Liebert, they are rather expensive.  Cyberpower
> seems to be cheap.
> 
> Are they any good, and how is the battery availability?  Can they even
> be monitored?

Liebert is very good, and -- as you said -- expensive. If you
are outfitting a datacenter, they are usually on the list.

Cyberpower is reasonably reliable; the batteries can be found
online. They are USB connected devices readable by NUT.

Some selected stats:

battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 20
battery.runtime: 3060
battery.runtime.low: 300
battery.type: PbAcid
battery.voltage: 24.0
battery.voltage.nominal: 24
device.mfr: CPS
device.model: CST135XLU
device.type: ups
driver.name: usbhid-ups
driver.version: 2.8.0
driver.version.data: CyberPower HID 0.6
driver.version.internal: 0.47
driver.version.usb: libusb-1.0.26 (API: 0x1000109)
input.voltage: 121.0
input.voltage.nominal: 120
output.voltage: 121.0
ups.beeper.status: enabled
ups.load: 16
ups.mfr: CPS
ups.productid: 0501
ups.realpower.nominal: 810
ups.serial: CDQHX2004035
ups.vendorid: 0764

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[hw]

On Fri, 2024-02-09 at 11:34 -0500, Roy J. Tellason, Sr. wrote:
> On Friday 09 February 2024 06:07:16 am hw wrote:
> > What other manufacturers could we buy UPSs from?
>  
> I have a Tripp-Lite sitting next to me here that replaced an APC and
> has 2-1/2 times the capabiliity.  Been in service several weeks and
> so far I'm pretty happy with it...

They seem to be extremely rare here.  Are they any good, and how's the
battery availability?

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[hw]

On Fri, 2024-02-09 at 06:44 -0500, Dan Ritter wrote:
> hw wrote: 
> > On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote:
> > > [...]
> > That sucks.  I didn't know that they don't stand behind their
> > products, and it makes APC not recommendable any longer.
> > 
> > What other manufacturers could we buy UPSs from?
> 
> Liebert at the high end, CyberPower at the low end. 

I've never heard of Liebert, they are rather expensive.  Cyberpower
seems to be cheap.

Are they any good, and how is the battery availability?  Can they even
be monitored?

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Stefan Monnier]

>> What other manufacturers could we buy UPSs from?
> I have a Tripp-Lite sitting next to me here that replaced an APC and has
> 2-1/2 times the capabiliity.  Been in service several weeks and so far I'm
> pretty happy with it...

Would they accept a warranty claim without having to run some
proprietary software (diagnostic and/or OS)?


Stefan

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Roy J. Tellason, Sr.]

On Friday 09 February 2024 06:07:16 am hw wrote:
> What other manufacturers could we buy UPSs from?
 
I have a Tripp-Lite sitting next to me here that replaced an APC and has 2-1/2 
times the capabiliity.  Been in service several weeks and so far I'm pretty 
happy with it...


-- 
Member of the toughest, meanest, deadliest, most unrelenting -- and
ablest -- form of life in this section of space,  a critter that can
be killed but can't be tamed.  --Robert A. Heinlein, "The Puppet Masters"
-
Information is more dangerous than cannon to a society ruled by lies. --James 
M Dakin

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Dan Ritter]

hw wrote: 
> On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote:
> > [...]
> That sucks.  I didn't know that they don't stand behind their
> products, and it makes APC not recommendable any longer.
> 
> What other manufacturers could we buy UPSs from?

Liebert at the high end, CyberPower at the low end. 

-dsr-

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[hw]

On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote:
> [...]
> Someone on the apcupsd mailing list thinks I have a faulty UPS or
> battery and should get a replacement.
> 
> APC refuses to proceed with a warranty claim because they don't
> support apcupsd or nut, only their own proprietary Powerchute. They
> won't proceed unless I can get Powerchute to show these events or a
> failed self-test.

That sucks.  I didn't know that they don't stand behind their
products, and it makes APC not recommendable any longer.

What other manufacturers could we buy UPSs from?

> [...]
> Having said that, I don't need to do a warranty claim. As it was
> only purchased a couple of weeks ago, consumer law allows me to
> return it to the seller as faulty whether they accept that or not,
> so I'll likely do that. It's just disappointing and a lot more
> hassle.

That seems like the best option.  You can then buy from a better
manufacturer which may avoid having trouble with APC later if there's
a problem.

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Curt]

On 2024-02-08, Charles Curley  wrote:
> On Thu, 8 Feb 2024 15:29:21 +
> Andy Smith  wrote:
>
>> I do not overly want to buy a Windows licence, run it
>> in a VM and pass USB through to that VM just to try this.
>
> You could try wine. You might need the more recent crossover-office,
> which is proprietary (but contributes greatly to wine).
>

I wonder if he could run the app on one of these virtual machines for
evaluation purposes:

https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Charles Curley]

On Thu, 8 Feb 2024 15:29:21 +
Andy Smith  wrote:

> I do not overly want to buy a Windows licence, run it
> in a VM and pass USB through to that VM just to try this.

You could try wine. You might need the more recent crossover-office,
which is proprietary (but contributes greatly to wine).

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Andy Smith]

Hello,

On Sun, Jan 28, 2024 at 06:55:04PM +, Andy Smith wrote:
> So, I must admit, I am quite tempted by BX1600MI which would cost me
> about £183. The equivalent spec in the Pro range is more than twice
> this price.

[ TL;DR: While free software like apcupsd or nut support all APC
  models that you can buy today, APC (Schneider Electric) the
  company only supports its own Windows-only Powerchute and won't do
  a warranty claim unless you can run that. I therefore question the
  device's suitability to a Linux environment. ]

Just as an update, I bought the APC Back-UPS BX1600MI and while
superficially it seems fine, using "apcupsd" and/or "nut" it reports
a constant stream of short-lived (less than 1 second) battery
detach/re-attach and powerfail/restore events.

The unit itself doesn't show any audible or visual alarm but as
these events are sub-second in duration I don't know if they are
just too quick for that.

Someone on the apcupsd mailing list thinks I have a faulty UPS or
battery and should get a replacement.

APC refuses to proceed with a warranty claim because they don't
support apcupsd or nut, only their own proprietary Powerchute. They
won't proceed unless I can get Powerchute to show these events or a
failed self-test. I can't do that because I don't have any Windows
machines. I do not overly want to buy a Windows licence, run it
in a VM and pass USB through to that VM just to try this.

While in theory if I had heeded the warnings about Back-UPS being of
lesser quality I might have bought a more expensive model that
wasn't faulty (or at least did not have this problem, whatever it
is), I am disappointed to learn that APC will not proceed with
warranty claims unless you can run some Windows software, which puts
me off the entire product range.

Having said that, I don't need to do a warranty claim. As it was
only purchased a couple of weeks ago, consumer law allows me to
return it to the seller as faulty whether they accept that or not,
so I'll likely do that. It's just disappointing and a lot more
hassle.

Thanks,
Andy

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync--delete-after)

[gene heskett]

On 1/28/24 13:55, Andy Smith wrote:

Hi,

Thanks, this is very useful.

On Sun, Jan 28, 2024 at 06:58:08PM +0100, hw wrote:

However, stay away from their cheap models as seen on this[1] picture
(Back UPS).  They work and you can replace the batteries yourself even
though you're not supposed to.  It's a minimum basic device.  It may
be on ok option if you're on a budget.  Their batteries last about 3
years.


So, I must admit, I am quite tempted by BX1600MI which would cost me
about £183. The equivalent spec in the Pro range is more than twice
this price.

Although the battery is not strictly user-replaceable, I watched
some videos on the task and it seems pretty easily doable.

Something for me to think on.

Thanks,
Andy


I'm a fan of APC, but the consumer versions. and I don't worry about 
batteries until they won't last the 6 or 7 seconds it takes to spin up 
the 20kw kohler in the back yard. My now deceased wife was on an oxy 
concentrator the last 15 years of her life, and a power failure of 20 
minutes might have finished her, so I bought a standby just a few months 
before the direcho that took power down for 3 days in June 2010. Very 
handy since.


I have an APC-1600 that been begging for a battery for a couple years, 
Still works fine for those few seconds.

Take care, stay well, Andy.

Cheers, Gene Heskett, CET
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after

[Brad Rogers]

On Sun, 28 Jan 2024 19:19:55 +0100
hw  wrote:

Hello hw,

>How do you know in advance when the battery will have failed?

Even my very basic UPS (APC Backup 1400) has a light on the front
labelled "Replace Battery".  That, combined with a very annoying high
pitch scream, are pretty good motivators to do the job.

I know the Backup 1400 was mentioned in this thread as "probably avoid"
(or something similar), but it's served me well thus far.  Had to replace
the battery pack only once.  That was after ten years, not the three to
five that people have been talking about.  APC no longer sell that
model, but battery packs are still available.  Just as an FYI, the
battery packs are sealed Lead-Acid.

Where I live (UK), it's possible to sell lead-acid batteries to scrap
merchants.  Amount paid is variable and subject to massive market forces
that are best described as 'volatile'.

Like others have mentioned with some of the more basic APC devices, this
particular model isn't designed with user replaceable batteries in mind,
but it's not an overly difficult task.  It can't easily (if at all) be
done leaving connected devices powered up, though.

-- 
 Regards  _   "Valid sig separator is {dash}{dash}{space}"
 / )  "The blindingly obvious is never immediately apparent"
/ _)rad   "Is it only me that has a working delete key?"
They take away our freedom in the name of liberty
Suspect Device - Stiff Little Fingers


pgpqAXSSxvoLF.pgp
Description: OpenPGP digital signature

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Andy Smith]

Hi,

Thanks, this is very useful.

On Sun, Jan 28, 2024 at 06:58:08PM +0100, hw wrote:
> However, stay away from their cheap models as seen on this[1] picture
> (Back UPS).  They work and you can replace the batteries yourself even
> though you're not supposed to.  It's a minimum basic device.  It may
> be on ok option if you're on a budget.  Their batteries last about 3
> years.

So, I must admit, I am quite tempted by BX1600MI which would cost me
about £183. The equivalent spec in the Pro range is more than twice
this price.

Although the battery is not strictly user-replaceable, I watched
some videos on the task and it seems pretty easily doable.

Something for me to think on.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after

[hw]

On Fri, 2024-01-26 at 15:56 +, Michael Kjörling wrote:
> On 26 Jan 2024 16:11 +0100, from h...@adminart.net (hw):
> > I rather spend the money on new batteries (EUR 40 last time after 5
> > years) every couple years [...]

To comment myself, I think was 3 years, not 5, sorry.

> > The hardware is usually extremely difficult --- and may be impossible
> > --- to replace.
> 
> And let's not forget that you can _plan_ to perform the battery
> replacement for whenever that is convenient.

How do you know in advance when the battery will have failed?

> Which is quite the contrast to a lightning strike blowing out even
> _just_ the PSU and it needing replacement before you can even use
> the computer again (and you _hope_ that nothing more took a hit,
> which it probably did even if the computer _seems_ to be working
> fine).

It would also hit the display(s), the switches and through that
everything that's connected to the network, the server(s) ...  That
adds up to a lot of money.

> [...]
> It's also worth talking to your local electrician about installing an
> incoming-mains overvoltage protection for lightning protection. I
> won't quote prices because I had mine installed a good while ago and
> also did it together with some other electrical work, but I was
> surprised at how low the cost for that was, and I _know_ that it has
> saved me on at least one occasion.

Hm I thought it's expensive.  I'll ask when I get a chance.

> [...]
> > You can always tell with a good hardware RAID because it
> > will indicate on the trays which disk has failed and the controller
> > tells you.
> 
> Or you can label the physical disks. Whenever I replace a disk, I
> print a label with the WWN of the new disk and place it so that it is
> readable without removing any disks or cabling;

That doesn't exactly help when the failed disk has disappeared
altogether, as if it had been removed ;)

But then, you can go by the numbers of the disks you can still see.

And beware of SSDs; when they fail, they're usually entirely
inaccessible whereas you may be still able to resuce (some) data from
a spinning disk after it failed.

It's probably really bad with mainbaords that use M2 storage since
apparently, they seem to support only one (of the some type at least)
rather than two.  So you can't use those at all.  What's the point of
that?  ZFS cache maybe?

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[hw]

On Fri, 2024-01-26 at 15:17 +, Andy Smith wrote:
> Hi,
> 
> On Fri, Jan 26, 2024 at 04:11:39PM +0100, hw wrote:
> > I've never had issues with any UPS due to self tests.  The batteries
> > need to be replaced when they are worn out.  How often that is
> > required depends on the UPS and the conditions it is working in,
> > usually every 3--5 years.
> 
> Out of interest what brand of UPS do you recommend for home use that
> has easily-replaceable batteries every 3–5 years? For a load of
> about 300W.

Generally I recommend APC because they work well (which is something
to be expected and shouldn't need to be pointed out), they can easily
be monitored with apcupsd and, very importantly, their batteries are
usually easily available so you can replace them without difficulty.

However, stay away from their cheap models as seen on this[1] picture
(Back UPS).  They work and you can replace the batteries yourself even
though you're not supposed to.  It's a minimum basic device.  It may
be on ok option if you're on a budget.  Their batteries last about 3
years.

I like the better models way better, like as on that[2] picture (Back
UPS pro).  I bought one a bit over 10 years ago (it even came with a
120k or so warranty for when a device protected by it would get
damaged) and replaced the batteries twice so far.  It's been working
without any issues ever since, and it'll probably work as long as new
batteries remain available.  So that's about 3 years battery life as
well.

Then it depends on a lot of things, primarily on the availability of
replacement batteries, then on how much you're willing to spend ---
since you can buy used ones because the only thing that goes bad is
the batteries, and you can find new old stock --- how much power you
need, if you want one that features a network card and if you want a
19" rack version or a standalone version.

Of course, their models change over time.  The 900VA smart UPS pro
delivers up to 540W, IIRC, and when it's overloaded it very annoyingly
beeps, but it continues to provide power.  I guess it shuts down when
it's overloaded and the main power fails, but I've never had that
happen yet.

For only 300W you go for this one:
https://www.apc.com/us/en/product/BR700G/apc-backups-pro-700va-420w-tower-120v-6x-nema-515r-outlets-avr-lcd-user-replaceable-battery/

Just keep in mind that you usually end up needing a UPS with higher
capacity than you planned for.  So it makes sense to check what the
batterie(s) cost and what the price difference between models with
lower and higher capacity is.  Some models take two or more batteries
while the version with lower capacity may take the same battery but
only one, making it overall so much cheaper that the model with more
capacity that requires two (or more) batteries may get too expensive.
But there may be a model with slightly more capacity that still takes
only one battery and you may be glad later that you spent a little
more money for more capacity.

Definitely stay away from UPSs from HP.  If you can reach someone from
HP at all, they will charge you before they would tell you what the
price of the batteries might be :(

Eaton probably makes good ones, too, but they're not common here, same
as another manufacturer the name of which I can't remember.  So I have
no experience with them.

Of course, you don't want to buy one from an unknown manufacturer with
no reputation, especially when it's a chinese one.  The batteries are
pretty generic, but for all you know, the manufacturer may have not
understood that pretty high currents can flow in an UPS and probably
has skimped on the wiring and/or other components to keep it cheap,
and it'll set your house on fire.  APC has understood that even in
their basic models (at least for the wiring; I can't tell for the
other components since I don't have enough knowledge about those).

After having said all the above, it's pretty simple because it comes
down to that, unless anything APC is difficult to come by where you
life, you can't go wrong with APC.


[1]: https://cdn-reichelt.de/bilder/web/xxl_ws/E910/APC_BX1400U_01.png
[2]:
https://oaziscomputer.hu/images/products/6934_apc-back-ups-pro-900-br900g-gr_1527776643.jpg

Re: rsync --delete vs rsync --delete-after

[hw]

On Fri, 2024-01-26 at 16:27 +, Michael Kjörling wrote:
> On 26 Jan 2024 16:39 +0100, from h...@adminart.net (hw):
> [...]
> > Having multiple generations of backups already increases the needed
> > storage space by a bit more than half.  That makes it already arguable
> > if it's better to make (multiple generations of) backups on a single
> > RAID or on N single disks.  Any of the disks can fail at any time.  If
> > you go with N == 2, a RAID (with multiple generations of backups on
> > it) can be better because when a disk fails, the RAID will very likely
> > survive and the non-RAID may not.
> 
> I'm not sure how you figure that.

It's simple: when using RAID1 with 2 disks, you double the physical
storage capacity needed.  When using 2 independent disks, you also
double the capacity.  In either case, when a disk fails, the RAID has
a chance to survive the failure while the single disks don't (ignoring
that you may be able to recover data from a failed disk).  So either
you don't loose a backup or you do loose one backup.

If you're lucky, you loose the outdated backup rather than the most
recent one.  If you made the backup on RAID, you don't loose the most
recent backup.  I like it better not to loose the most recent backup.

That's assuming that you have storage capacity for a single backup,
i. e. one backup on RAID vs. one most recent backup on one disk and on
older backup on the other disk.

Of course, when you have multiple generations of backups on each set
of disks, things get more complicated.  It also gets more complicated
when the volume you're making backups of doesn't fit on a single disk
...

> [...]
> > Trying to make things appear easier by pointing out that failed disks
> > can be replaced is not helpful.
> 
> It's a _backup_. _By definition_, a backup is only critical once the
> primary copy becomes inaccessible for some reason. Hence:

I have to disagree here.  The backup is always critical before you
have eliminated the possibility that the data can get lost.  Only when
you have done that, then you don't need a backup at all.

Re: rsync --delete vs rsync --delete-after

[Ralph Aichinger]

On Fri, 2024-01-26 at 16:11 +0100, hw wrote:
> I've never had issues with any UPS due to self tests.  The batteries
> need to be replaced when they are worn out.  How often that is
> required depends on the UPS and the conditions it is working in,
> usually every 3--5 years.

It was with some small to mid APC model, I think. We had about 1 to 2kW
worth of servers on it, so it was not that small, definitely no
consumer type. When I took over maintenance somebody had configured
some sort of weekly or biweekly self-test, that switched over to 
battery, was supposed to run the battery down to 25% or similar, and
then return to mains power/charging.

Except once what the UPS considered 25% charge seemingly was not, and
everything shut down instantly.

> I rather spend the money on new batteries (EUR 40 last time after 5
> years) every couple years rather than spending thousands on replacing
> the hardware when a power surge damages it which could have been
> prevented by the UPS, and it's better to have the machines shut down
> properly rather taking risks with potential data loss, regardless of
> file systems and RAID setups in use.

I think having hardware for "thousands" and having a UPS with that
cheap batteries is not that common. In above company we certainly had 
hardware for thousands, but changing batteries cost hundreds of Euros,
even with off-brand aftermarket parts. It also was complicated to order
the right parts etc.

> RAID isn't as complicated as you think.  Hardware RAID is most
> simple,
> followed by btrfs, followed by mdadm.

I have to disagree with that too. Some hardware RAIDs might be simple,
but others are not. Tracking down the rebrandings of Adaptec,
aquisitions and mergers, is a science by itself. As is finding and
installing their Firmware and utilites. Are they still calles Avago, 
or something new again?

Or all that BBU stuff: Tracking the state of battery backup units
on the controller, and ordering and replacing the correct battery
is also not really easy. Clearly enterprise IT type of stuff, keeping
even knowledgeable people busy for hours, if you don't do it at scale 
and regularily.

Also often Linux support is problematic. Yes, it will work, but
sometimes certain utilities are not available or work as good as
with Windows.

On the other hand mdadm software RAID is well documented and painless.

> 
> With hardware RAID I can instruct someone who has no idea what
> they're
> doing to replace a failed disk remotely.  Same goes for btrfs and
> mdadm, though it better be someone who isn't entirely clueless

In fact this was my job for some time: Administering hardware RAID 
equipped servers, and instructing "remote hands" or customers to 
swap harddisks. It was not always easy, not always were the correct
disks pulled, even though it was correctly labelled. Sometimes 
clueless people tried swapping by themselves, mixing stuff up. We
also had one server with wrong labelling, for whatever reason. That 
was no fun ;) 

Now I won't dispute that RAID has its place in data centers and many
other applications. I just doubt that it is the correct choice for many
home users.

> More importantly, the hassle involved in trying to recover from a
> failed disk is ridiculously enormous without RAID and can get
> expensive when hours of work were lost.  With RAID, you don't even
> notice unless you keep an eye on it, and when a disk has failed, you
> simply order a replacement and plug it in.

Yes, that can happen. But more often than not the scenario is like it 
is with most notebooks today. You send your notebook in for repair, and
have to reinstall anyway. Happened to me. I backed up my Debian system,
sent the device in for hardware repair, got it back with Windows 10 ;)
And no, it was not the disk that was broken, but the touchpad.

> 
> It's not like you could go to a hardware store around the corner and
> get a new disk same or next day.  Even if you have a store around,
> they will need to order the disk, and that can, these days, take
> weeks
> or months or longer if it's a small store. 

For consumer hard disks? I just go to my favourite shop if I need
a replacement, and they've got maybe 20 or 30 types of hard disk 
in stock, to be bought right away. Even more with SSDs. And I am
in a smallish city, pop. 250.000.


> That is simply wrong.  RAID doesn't protect you from malware, and
> nothing protects you from user error.  If you have data losses from
> malware and/or user error more often than from failed disks, you're
> doing something majorly wrong.

In my experience user error is the main source of data loss. By far.

> This shows that you have no experience with RAID and is not an
> argument.

I've got years of experience with RAID, both in my personal use and
with employers doing stuff on RAID for customers and internal services.
In my experience RAID is a nice solution for data center type setups.
RAID often is problematic for home users or even small offices.

> Making backups 

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Roger Price]

On Fri, 26 Jan 2024, David Wright wrote:

On Fri 26 Jan 2024 at 19:03:33 (+0100), Roger Price wrote:

I currently have two Eaton Ellipse ECO 1600's. ... The four screws are deeply 
recessed and difficult to see.  They have different heads: some are Torx 10, 
others are a star.



20/20 hindsight might suggest that you were only intended
to remove the star, if by that you mean Philips/Pozidrive.


What I called "star" is probably a Quadrex.

Roger

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[David Wright]

On Fri 26 Jan 2024 at 19:03:33 (+0100), Roger Price wrote:

> I currently have two Eaton Ellipse ECO 1600's.  I change the batteries
> every 4-5 years, but this is not as easy as it should be.  It is not
> evident that only one of the four back panel screws needs to be
> removed.  I took me a while to learn this.  The four screws are deeply
> recessed and difficult to see.  They have different heads: some are
> Torx 10, others are a star.  Keep trying different screwdrivers until
> you feel something turning.

20/20 hindsight might suggest that you were only intended
to remove the star, if by that you mean Philips/Pozidrive.

Cheers,
David.

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Roger Price]

On Fri, 26 Jan 2024, Andy Smith wrote:


Out of interest what brand of UPS do you recommend for home use that
has easily-replaceable batteries every 3–5 years? For a load of
about 300W.


I currently have two Eaton Ellipse ECO 1600's.  I change the batteries every 4-5 
years, but this is not as easy as it should be.  It is not evident that only one 
of the four back panel screws needs to be removed.  I took me a while to learn 
this.  The four screws are deeply recessed and difficult to see.  They have 
different heads: some are Torx 10, others are a star.  Keep trying different 
screwdrivers until you feel something turning.


The battery compartment is too tight. I took me 4 attempts to get the batteries 
back in, with the cables still connected and positioned such that the rear panel 
can be put back.


If you re-assemble and the UPS doesn't respond to pressing the ON/OFF button, 
then the battery leads have detached.  Start all over again.  Good luck!


It could have been a lot easier.  Roger

Re: rsync --delete vs rsync --delete-after

[Michael Kjörling]

On 26 Jan 2024 16:39 +0100, from h...@adminart.net (hw):
>> RAID is for uptime.
> 
> It's also for saving you from the hassle involved with loosing data
> when a disk fails.

Which translates to more quickly fully recovering from the loss of a
storage device.

When used for redundancy and staying within the redundancy threshold
of your setup, _done right_, RAID reduces unplanned downtime to zero
in case of loss of a storage device. Depending on your setup, the
replacement can either be made online or planned for, and once
complete, (hopefully) no data whatsoever has been lost and everything
has happened at minimal time cost. I'm assuming that _the physical
act_ of replacing the storage device is similar regardless of how it
is configured in software: the same number of screws, clamps or
similar need removing and reinstalling; the same amount of time is
required to physically move the old and the new storage device; etc.


>> If a week-long outage (to get replacement hardware and restore the
>> most recent backup) and a day's worth of data loss is largely
>> inconsequential, as quite frankly it likely is for most home users
>> save for the cost of replacement hardware, that's a very different
>> scenario from if that same outage costs $$€€¥¥ and could destroy
>> your livelihood; and consequently the choices made _should_ likely
>> be different.
> 
> That's assuming your time isn't worth anything and ignores whatever
> the loss of data may cost you.  If that isn't relevant to you, you
> don't backups, either.

Sure, you can tune the RPO (recovery point objective) for your backup
solution as needed. If you need a RPO of five minutes after
catastrophic storage failure (meaning that you lose no more than five
minutes of data regardless of when failure happens), then you're going
to make different choices than if a 24-48 hour RPO is fine. But I'm
willing to say that _most_ home users can recover from a loss of a
day's worth of changes to their data without that being a major blow
to whatever they are doing; and if the user does something important,
nothing prevents running an extra backup to capture those changes.
(I've done that myself on occasion.)

Similarly, you are going to make different choices based on your RTO
(recovery time objective). RAID gives you essentially zero RTO as long
as you retain sufficient redundancy, but _not_ past that. Once your
storage drops below whatever level of redundancy you have, you're
looking at rebuilding from backups, at which point backup frequency
and backup restoration time are the minimums which will dictate your
RPO and RTO respectively. So even if you have RAID, you need to know
what your target RPO and RTO are, respectively, because again those
values are going to be a major factor in the design of your backup
regimen.


>> _Mirrored backups_ makes very little sense to me. [...]
> 
> Having multiple generations of backups already increases the needed
> storage space by a bit more than half.  That makes it already arguable
> if it's better to make (multiple generations of) backups on a single
> RAID or on N single disks.  Any of the disks can fail at any time.  If
> you go with N == 2, a RAID (with multiple generations of backups on
> it) can be better because when a disk fails, the RAID will very likely
> survive and the non-RAID may not.

I'm not sure how you figure that. To survive the loss of N > 0 storage
devices within a set, a storage solution needs to have a raw capacity
greater than the usable capacity. An illustrative case would be a
three-way mirror: it can survive the loss of two out of three storage
devices, but only has the usable storage capacity of a single
(typically the smallest) of the devices. It's not really any different
from that a commodity HDD or SSD probably won't survive the failure of
one of its platters or flash chips respectively, even ignoring cascade
effects of either the failure or the cause of failure.

How much extra storage space you need to keep multiple generations of
backups is going to depend a lot on the rate of churn of your dataset.
Again as an illustrative example only, suppose you have 1 TB of data
and modify 1 MB per day, and make backups daily; with two devices each
capable of storing 2 TB you can have two full copies plus 1M days'
worth of history on each. This is irrespective of how those storage
devices themselves are furnished.

> Trying to make things appear easier by pointing out that failed disks
> can be replaced is not helpful.

It's a _backup_. _By definition_, a backup is only critical once the
primary copy becomes inaccessible for some reason. Hence:

>> The only time when something
>> like mirrored backups will help you is when you have only one backup
>> set, the backup itself works fine, but a backup drive fails, _and_ the
>> source fails before you've been able to make a new backup.

For a primary copy, _of course_ the calculus is different.

-- 
Michael Kjörling 🔗 https://michael.kjorli

Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after

[Michael Kjörling]

On 26 Jan 2024 16:11 +0100, from h...@adminart.net (hw):
> I rather spend the money on new batteries (EUR 40 last time after 5
> years) every couple years [...]
> 
> The hardware is usually extremely difficult --- and may be impossible
> --- to replace.

And let's not forget that you can _plan_ to perform the battery
replacement for whenever that is convenient. Which is quite the
contrast to a lightning strike blowing out even _just_ the PSU and it
needing replacement before you can even use the computer again (and
you _hope_ that nothing more took a hit, which it probably did even if
the computer _seems_ to be working fine).


>> I've had no external power outage in the last 5 or 10 years, but a UPS
>> often needs at least one battery replacement during that time.
> 
> Outages are (still) rare here, but it suffices to trigger a fuse or
> the main switch when some device shorts out, or someone working on the
> solar power systems some of the neighbours have, causing crazy voltage
> fluctuations, or a lightning strike somewhere in the vinicity or
> whatever reason for an UPS to be required.

It's also worth talking to your local electrician about installing an
incoming-mains overvoltage protection for lightning protection. I
won't quote prices because I had mine installed a good while ago and
also did it together with some other electrical work, but I was
surprised at how low the cost for that was, and I _know_ that it has
saved me on at least one occasion. It won't do power conditioning or
power loss protection of course, but it _does_ greatly increase the
odds that your home wiring survives a lightning-related voltage surge.
(Nothing will realistically protect you against a _direct_ lightning
strike; in that case the very best you can hope for is damage
containment.)


> More importantly, the hassle involved in trying to recover from a
> failed disk is ridiculously enormous without RAID and can get
> expensive when hours of work were lost.  With RAID, you don't even
> notice unless you keep an eye on it, and when a disk has failed, you
> simply order a replacement and plug it in.

Indeed; the point of RAID is uptime.


> You can always tell with a good hardware RAID because it
> will indicate on the trays which disk has failed and the controller
> tells you.

Or you can label the physical disks. Whenever I replace a disk, I
print a label with the WWN of the new disk and place it so that it is
readable without removing any disks or cabling; then I use the WWN to
identify the disk in software; in both cases because the WWN is a
stable identifier that I can fully expect will never change throughout
the disk's lifetime. So when the system tells me that
wwn-0x123456789abcdef0 is having issues, I can quickly and accurately
identify the exact physical device that needs replacement once I have
a replacement on hand. And if the kernel logs are telling me that,
say, sdg is having issues, I can map that back to whatever WWN happens
to map to that identifier at that particular time. (In practice, I'm
more likely to get useful error details through ZFS status monitoring
tools, where I already use the WWN, so I likely won't need to go that
somewhat circuitous route.)


> Yes, my setup is far from ideal when it comes to backups in that I
> should make backups more frequently.  That doesn't mean I shouldn't
> have good backups and that UPSs and RAID were not required.

Or that, again, they solve different problems.

-- 
Michael Kjörling 🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: rsync --delete vs rsync --delete-after

[hw]

On Thu, 2024-01-18 at 13:09 +, Michael Kjörling wrote:
> On 18 Jan 2024 13:26 +0100, from r...@h5.or.at (Ralph Aichinger):
> > As a home/SOHO user, I'd rather have a working backup every few hours
> > or every day than some RAID10 wonder
> 
> Definitely agree that a solid backup regimen (including regular
> automated backups; at least one off-site copy _at least_ of critical,
> hot data; and planning for the contingency that you need to restore
> that backup onto a brand new system without access to anything on your
> current system -- think "home burns down at night" or "burglar"
> scenario) is the _first_ step, and one that a great deal of people
> still fail at.
> 
> RAID is for uptime.

It's also for saving you from the hassle involved with loosing data
when a disk fails.

> If a week-long outage (to get replacement hardware and restore the
> most recent backup) and a day's worth of data loss is largely
> inconsequential, as quite frankly it likely is for most home users
> save for the cost of replacement hardware, that's a very different
> scenario from if that same outage costs $$€€¥¥ and could destroy
> your livelihood; and consequently the choices made _should_ likely
> be different.

That's assuming your time isn't worth anything and ignores whatever
the loss of data may cost you.  If that isn't relevant to you, you
don't backups, either.

> _Mirrored backups_ makes very little sense to me. If a storage device
> used for storage of backups fails prematurely, just toss it and get a
> new one and make a new backup. If the backup software goes haywire and
> starts overwriting everything with random garbage, having the garbage
> mirrored isn't going to help you. It's much better to have two
> independent backup targets and switching between them, and figuring
> the switching interval into your RPO. The only time when something
> like mirrored backups will help you is when you have only one backup
> set, the backup itself works fine, but a backup drive fails, _and_ the
> source fails before you've been able to make a new backup. That's a
> _very_ narrow scenario and easily solved by having two backup sets.

Having multiple generations of backups already increases the needed
storage space by a bit more than half.  That makes it already arguable
if it's better to make (multiple generations of) backups on a single
RAID or on N single disks.  Any of the disks can fail at any time.  If
you go with N == 2, a RAID (with multiple generations of backups on
it) can be better because when a disk fails, the RAID will very likely
survive and the non-RAID may not.

So for daily use, I'd ideally make multiple generations on RAID and
copy the latest generation to somewhere off-site, using either single
disks, or another RAID.

Having no hassle and no downtime at the production system and instead
having hassle and downtime off-site may be much more desirable than
having it the other way round.

Trying to make things appear easier by pointing out that failed disks
can be replaced is not helpful.  Replacing a disk in a RAID isn't any
more difficult (and can be much easier) than replacing a disk that
isn't in a RAID.

Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[fxkl47BF]

On Fri, 26 Jan 2024, Andy Smith wrote:

> Hi,
>
> On Fri, Jan 26, 2024 at 04:11:39PM +0100, hw wrote:
>> I've never had issues with any UPS due to self tests.  The batteries
>> need to be replaced when they are worn out.  How often that is
>> required depends on the UPS and the conditions it is working in,
>> usually every 3--5 years.
>
> Out of interest what brand of UPS do you recommend for home use that
> has easily-replaceable batteries every 3–5 years? For a load of
> about 300W.


just my experience
i have 2 apc 740's that date from the mid 90's
when the internal battery gave out i added a 50 amp anderson connector
i connect to a 35ah sla
you can change battteries while it's running

Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)

[Andy Smith]

Hi,

On Fri, Jan 26, 2024 at 04:11:39PM +0100, hw wrote:
> I've never had issues with any UPS due to self tests.  The batteries
> need to be replaced when they are worn out.  How often that is
> required depends on the UPS and the conditions it is working in,
> usually every 3--5 years.

Out of interest what brand of UPS do you recommend for home use that
has easily-replaceable batteries every 3–5 years? For a load of
about 300W.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: rsync --delete vs rsync --delete-after

[hw]

On Thu, 2024-01-18 at 13:26 +0100, Ralph Aichinger wrote:
> Hello fellow Debian users,
> 
> On Thu, 2024-01-18 at 12:18 +0100, hw wrote:
> 
> > Always use an UPS.
> 
> 
> Here I have a somewhat contrarian view, I hope not to offend too much:

It's not offending, you merely have a different opinion.

> For countries with stable electricity supplies (like Austria where I
> live) having a small UPS might actually lead to more problems instead
> of less, unless you are putting a lot of effort into it. Very often
> have I had problems with UPSes, e.g. batteries dying, the UPS going
> into some self test mode and inadvertedly shutting down, etc.

I've never had issues with any UPS due to self tests.  The batteries
need to be replaced when they are worn out.  How often that is
required depends on the UPS and the conditions it is working in,
usually every 3--5 years.

I rather spend the money on new batteries (EUR 40 last time after 5
years) every couple years rather than spending thousands on replacing
the hardware when a power surge damages it which could have been
prevented by the UPS, and it's better to have the machines shut down
properly rather taking risks with potential data loss, regardless of
file systems and RAID setups in use.

The hardware is usually extremely difficult --- and may be impossible
--- to replace.  On top of that, noone pays for the time and effort
lost, and even someone did, I'd rather keep my hardware instead of
going to all the lengths required to replace it.  Loosing hours of
work due to power outages can also be an issue, and nobody pays for
that, either.

> I've had no external power outage in the last 5 or 10 years, but a UPS
> often needs at least one battery replacement during that time.

Outages are (still) rare here, but it suffices to trigger a fuse or
the main switch when some device shorts out, or someone working on the
solar power systems some of the neighbours have, causing crazy voltage
fluctuations, or a lightning strike somewhere in the vinicity or
whatever reason for an UPS to be required.

Way too many times than I could count my UPSs saved me and the
hardware from power supply problems, so I can't recommend to go
without one.  It doesn't matter how stable you think the power grid in
your country is.

> [...]
> Here I also doubt if this is a wise suggestion for the typical home
> or small office user. RAID leads to lots and lots of complexity, that
> is often not needed in a home setup.

RAID isn't as complicated as you think.  Hardware RAID is most simple,
followed by btrfs, followed by mdadm.

With hardware RAID I can instruct someone who has no idea what they're
doing to replace a failed disk remotely.  Same goes for btrfs and
mdadm, though it better be someone who isn't entirely clueless.

More importantly, the hassle involved in trying to recover from a
failed disk is ridiculously enormous without RAID and can get
expensive when hours of work were lost.  With RAID, you don't even
notice unless you keep an eye on it, and when a disk has failed, you
simply order a replacement and plug it in.

It's not like you could go to a hardware store around the corner and
get a new disk same or next day.  Even if you have a store around,
they will need to order the disk, and that can, these days, take weeks
or months or longer if it's a small store.  The only way to get a
replacement is ordering online, which requires that your computer
still works and that you have access to your passwords.

> I'd rather have a working backup setup with many independent copies
> before I even start thinking about RAID. Yes, disks can fail, but
> data loss often is due to user error and malware.

That is simply wrong.  RAID doesn't protect you from malware, and
nothing protects you from user error.  If you have data losses from
malware and/or user error more often than from failed disks, you're
doing something majorly wrong.

> RAID helps very little with the latter two causes of data loss. And
> all too often have I seen people mess up their complicated RAID
> setups, because they pulled the wrong disk when another one broke,
> or because they misinterpreted complicated error messages, creating
> unnecessary data loss out of user error by themselves.

This shows that you have no experience with RAID and is not an
argument.

Making backups is way more complicated than RAID.  You can way more
easily overwrite the wrong backup or misinterpret error messages of
your backup solution than you can pull the wrong disk from a RAID or
misinterpret error messages from your RAID.

How exactly would you pull the wrong disk from a RAID and thus cause
data loss?  Before you pull one, you make a backup.  When the disk has
been pulled, its contents remain unchanged and when you put it back
in, your data is still there --- plus you have the backup.  Sure it
can sometimes be difficult to tell which disk you need to replace, and
it's not an issue because you can always figure out which one you need
to replace. 

Re: rsync --delete vs rsync --delete-after

[Michel Verdier]

On 2024-01-18, Andy Smith wrote:

> Could check the man page then like I said.
>
> Some options require rsync to know the full file list, so these
> options disable the incremental recursion mode. These include:
> --delete-before, --delete-after, --prune-empty-dirs, and
> --delay-updates.

You are right, thanks !

Re: rsync --delete vs rsync --delete-after

[Default User]

On Thu, 2024-01-18 at 21:44 +, Andy Smith wrote:
> Hello,
> 
> On Thu, Jan 18, 2024 at 10:01:46PM +0100, Michel Verdier wrote:
> > On 2024-01-18, Andy Smith wrote:
> > > If you use --delete-after (and some other options) then rsync has
> > > to
> > > check every file before it can do any work, whereas normally it
> > > will
> > > find a few files to work on and start work, meanwhile
> > > incrementally
> > > scanning for more.
> > 
> > Not sure of that.
> 
> Could check the man page then like I said.
> 
>     Some options require rsync to know the full file list, so these
>     options disable the incremental recursion mode. These include:
>     --delete-before, --delete-after, --prune-empty-dirs, and
>     --delay-updates.
> 
> Thanks,
> Andy
> 



Hi, OP here.

Just an update.
Last night I did the usual copy from the primary backup drive to the
secondary backup drive, except I did it using rsync --delete instead of
rsync --delete-after. This time, it took 73 minutes.  The previous time
it took 131 minutes.  So it did go significantly quicker.  And "seemed"
to go okay.  

Thank you to all who have weighed in on this topic!

Re: rsync --delete vs rsync --delete-after

[Andy Smith]

Hello,

On Thu, Jan 18, 2024 at 10:01:46PM +0100, Michel Verdier wrote:
> On 2024-01-18, Andy Smith wrote:
> > If you use --delete-after (and some other options) then rsync has to
> > check every file before it can do any work, whereas normally it will
> > find a few files to work on and start work, meanwhile incrementally
> > scanning for more.
> 
> Not sure of that.

Could check the man page then like I said.

Some options require rsync to know the full file list, so these
options disable the incremental recursion mode. These include:
--delete-before, --delete-after, --prune-empty-dirs, and
--delay-updates.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: rsync --delete vs rsync --delete-after

[Michel Verdier]

On 2024-01-18, Andy Smith wrote:

> If you use --delete-after (and some other options) then rsync has to
> check every file before it can do any work, whereas normally it will
> find a few files to work on and start work, meanwhile incrementally
> scanning for more.

Not sure of that. rsync always gets a list of files on both sides and
compares them before doing anything. But --delete-after forces a second
pass on the target for deletion instead of doing it during copying. So a
bit more IO.

Re: rsync --delete vs rsync --delete-after

[Stefan Monnier]

>> > However, I have read that using rsync --delete instead of rsync --
>> > delete-after is faster and uses less memory, and so is more efficient. 
>> I'd be surprised if it makes a significant difference.
> If you use --delete-after (and some other options) then rsync has to
> check every file before it can do any work,

Oh, right, I was thinking of `--delete-delay`.
[ Tho, as you mention, the performance impact of `--delete-after` is
  also negligible in many cases.  ]


Stefan

Re: rsync --delete vs rsync --delete-after

[Andy Smith]

Hello,

On Wed, Jan 17, 2024 at 11:09:35PM -0500, Stefan Monnier wrote:
> > However, I have read that using rsync --delete instead of rsync --
> > delete-after is faster and uses less memory, and so is more efficient. 
> 
> I'd be surprised if it makes a significant difference.

If you use --delete-after (and some other options) then rsync has to
check every file before it can do any work, whereas normally it will
find a few files to work on and start work, meanwhile incrementally
scanning for more. Each file that it's working on uses 100 bytes of
memory, so that's about 100MB for each million files.

Not a huge amount of memory either way so probably won't cause an
issue. It would be the thing about not starting any work until
having fully scanned the file tree.

It's all covered in the man page where it talks about those options.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: rsync --delete vs rsync --delete-after

[Ralph Aichinger]

On Thu, 2024-01-18 at 13:09 +, Michael Kjörling wrote:
> 
> Definitely agree that a solid backup regimen (including regular
> automated backups; at least one off-site copy _at least_ of critical,
> hot data; and planning for the contingency that you need to restore
> that backup onto a brand new system without access to anything on
> your
> current system -- think "home burns down at night" or "burglar"
> scenario) is the _first_ step, and one that a great deal of people
> still fail at.

Absolutely. I use a Raspberry Pi with an external
USB drive for my off-site backups with Resitic. Seems to
work fine for now, draws very little power, and the 4TB of a small
2.5" disk is plenty for my personal backups, when deduplicated. Still
this setup probably is too complicated for many home users, where a 
cloud backup or similar makes more sense.

> RAID is for uptime. If a week-long outage (to get replacement
> hardware
> and restore the most recent backup) and a day's worth of data loss is
> largely inconsequential, as quite frankly it likely is for most home
> users save for the cost of replacement hardware,

For me the calculation is more or less "next workday to go to the local
shop for a replacement hard drive" and a few hours to restore backups.
Yes, if you depend on mail order, one week might be more realistic.
Then I probably would keep a spare drive around even as a home user.

>  that's a very
> different scenario from if that same outage costs $$€€¥¥ and could
> destroy your livelihood; and consequently the choices made _should_
> likely be different.

Of course. As soon as you have to pay several people's salaries 
needlessly while they sit around for access to their data, RAID
makes more sense quickly. Still, it makes sense to think about what
you can do yourself vs. what needs external work done, also because
somebody external to repair a RAID might not show up all that quickly
unless you've got some pre-negotiated contract.

> _Mirrored backups_ makes very little sense to me. If a storage device
> used for storage of backups fails prematurely, just toss it and get a
> new one and make a new backup.

Absolutely! Just make more backups, or more backups with different,
independent strategies. As much as possible I try to do two independent
systems (e.g. Restic doing time-based offsite backups, and a cron job
doing a simple tar.gz file into some local drive or storage).

/ralph

Re: rsync --delete vs rsync --delete-after

[Michael Kjörling]

On 18 Jan 2024 13:26 +0100, from r...@h5.or.at (Ralph Aichinger):
> As a home/SOHO user, I'd rather have a working backup every few hours
> or every day than some RAID10 wonder

Definitely agree that a solid backup regimen (including regular
automated backups; at least one off-site copy _at least_ of critical,
hot data; and planning for the contingency that you need to restore
that backup onto a brand new system without access to anything on your
current system -- think "home burns down at night" or "burglar"
scenario) is the _first_ step, and one that a great deal of people
still fail at.

RAID is for uptime. If a week-long outage (to get replacement hardware
and restore the most recent backup) and a day's worth of data loss is
largely inconsequential, as quite frankly it likely is for most home
users save for the cost of replacement hardware, that's a very
different scenario from if that same outage costs $$€€¥¥ and could
destroy your livelihood; and consequently the choices made _should_
likely be different.

_Mirrored backups_ makes very little sense to me. If a storage device
used for storage of backups fails prematurely, just toss it and get a
new one and make a new backup. If the backup software goes haywire and
starts overwriting everything with random garbage, having the garbage
mirrored isn't going to help you. It's much better to have two
independent backup targets and switching between them, and figuring
the switching interval into your RPO. The only time when something
like mirrored backups will help you is when you have only one backup
set, the backup itself works fine, but a backup drive fails, _and_ the
source fails before you've been able to make a new backup. That's a
_very_ narrow scenario and easily solved by having two backup sets.

-- 
Michael Kjörling 🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: rsync --delete vs rsync --delete-after

[Michel Verdier]

On 2024-01-17, Default User wrote:

> BTW(2), I do use rsnapshot with cron jobs to back up the internal SSD
> to the primary backup drive daily (and weekly, monthly, yearly).  But I
> am not sure if I could also use it to do copies of the primary backup
> drive to the secondary backup drive (maybe using an additional
> configuration file)? 

You can safely rsync entire rsnapshot backups if you use --hard-links
option to preserve links set by rsnapshot. You have to rsync "entire"
rsnapshot directory else it can't set correct hard links.

Re: rsync --delete vs rsync --delete-after

[Ralph Aichinger]

Hello fellow Debian users,

On Thu, 2024-01-18 at 12:18 +0100, hw wrote:

> Always use an UPS.


Here I have a somewhat contrarian view, I hope not to offend too much:

For countries with stable electricity supplies (like Austria where I
live) having a small UPS might actually lead to more problems instead
of less, unless you are putting a lot of effort into it. Very often
have I had problems with UPSes, e.g. batteries dying, the UPS going
into some self test mode and inadvertedly shutting down, etc.

I've had no external power outage in the last 5 or 10 years, but a UPS
often needs at least one battery replacement during that time.

Unless you have some sort of professional server rack and redundant 2
phase supply, in my opinion UPS make very little sense to the home or 
small office user. Also modern Linux systems with journalling
filesystems will survive the occasional hard shutdown. Yes, I have
pulled the plug out of running Linux boxes occasionally because I was
too lazy to shut it down correctly and never had one break beyond the
usual fsck on boot.

> Always use redundancy to store data for a running system, like some
> form of RAID.  It won't hurt to use RAID for backups as well, though
> I don't think that's required when you use it for the data you're
> backing up.

Here I also doubt if this is a wise suggestion for the typical home
or small office user. RAID leads to lots and lots of complexity, that
is often not needed in a home setup. I'd rather have a working backup
setup with many independent copies before I even start thinking about
RAID. Yes, disks can fail, but data loss often is due to user
error and malware. RAID helps very little with the latter two causes
of data loss. And all too often have I seen people mess up their
complicated RAID setups, because they pulled the wrong disk when
another one broke, or because they misinterpreted complicated error
messages, creating unnecessary data loss out of user error by
themselves.

As a home/SOHO user, I'd rather have a working backup every few hours
or every day than some RAID10 wonder that makes me lose more time on
reading RAID documentation, and ordering spare drives (you've got
one of those spares for each array, do you?) than is actually lost by
not being able to restore to the exact last minute before a hard disk
died.

/ralph -- no UPS at home, using RAID1 md mirroring though

Re: rsync --delete vs rsync --delete-after

[Michael Kjörling]

On 18 Jan 2024 12:15 +0100, from to...@tuxteam.de:
>> **That "primary backup drive" is not a backup at all.**
> 
> It is: against the situation you fat-finger something and react
> before the next backup happens (this is a threat worth being taken
> into account). For that case, a backup with more than one "back"
> version would be more useful (see --link-dest in rsync for that).
> 
> It is not against the system running amok and thrashing all attached
> disks (be it by accident -- improbable, or by malice -- more probable.
> cf. crypto-malware). Disconnected back ups are better here.
> 
> It is not for against your shed going up in flames. Off site for this.

OP has specified (17 Jan 19:52 UTC) that the threat model includes,
among many other things, "mechanical failure" and "lightning".

A single copy offers zero protection against that.

-- 
Michael Kjörling 🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: rsync --delete vs rsync --delete-after

[hw]

On Wed, 2024-01-17 at 14:52 -0500, Default User wrote:
> On Wed, 2024-01-17 at 10:29 -0800, Kushal Kumaran wrote:
> > On Wed, Jan 17 2024 at 11:19:39 AM, Default User
> >  wrote:
> > > Hello!
> > > 
> > > Opinions, please.
> > > 
> > > I use rsync to copy my primary backup drive to a secondary backup
> > > drive
> > > , so that the secondary backup drive is theoretically always an
> > > exact
> > > copy of the primary backup drive.  
> > > 
> > > Here is the rsync command I use:
> > > 
> > > time sudo rsync -aAXHxvv --delete-after --numeric-ids --
> > > info=progress2,stats2,name2 --
> > > exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/m
> > > edia
> > > /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/
> > > 
> > > Question: 
> > > I use rsync --delete-after because it might seem to be "safer", so
> > > in
> > > case of a "glitch" of any kind, no file ever disappears from both
> > > the 
> > > source drive and the destination drive.  
> > > 
> > 
> > What do you mean by "glitch"?  Irrespective of whether you use --
> > delete
> > or --delete-after, deleted files on the source are deleted on the
> > destination once your rsync is complete (which is what I'd assume you
> > want when you want an exact copy).  I'd presume if you're ok with
> > that,
> > you are also fine with the deletion happening earlier in the rsync
> > process?
> > 
> > If you're concerned about accidental deletions, you should just not
> > use
> > any of the `--delete*` options (and give up on the exact copy
> > requirement).  You can look at alternatives to bare rsync that keep
> > track of multiple backed-up images (rsnapshot is a very simple
> > wrapper
> > over rsync that can do this, for example).
> > 
> > > However, I have read that using rsync --delete instead of rsync --
> > > delete-after is faster and uses less memory, and so is more
> > > efficient. 
> > > 
> > > Note: The current copy process time varies, but takes a long time -
> > > last night 131 minutes.
> > > :(
> > 
> > You can try using --delete for a couple of runs and see if it
> > actually
> > affects performance in your situation.
> > 
> > > 
> > > Disk space used is not currently an issue.
> > > 
> > > But, is rsync --delete AS SAFE as rsync --delete-after?
> > 
> > You'll need to define what safety means for you.
> > 
> 
> 
> 
> Hi, Kushal! 
> Thanks for replying. 
> 
> By "glitch", I mean anything that could interfere with the rsync copy
> process.  Possible causes: 
> - electrical outages, voltage spikes, voltage drops, "brownouts"
> - mechanical failure
> - earthquake
> - lightning
> - cat walking on keyboard
> - out of memory errors
> - out of disk space errors
> - PEBKAC errors
> - etc.
> 
> By safe, may I try to explain using a story? 
> 
> I once read that centuries ago, a king wanted his crown to be safe.  So
> four guards watched his crown at all times.  The guards were not
> allowed to take their eves off of the crown, even for a second, until
> the the their replacements, the next group of guards, all said "I see
> the crown". 
> 
> I am sorry if I can not come up with a better, technical explanation. 
> I use this story because it has always been meaningful to me, and seems
> to point to the essence of what I am getting at. 
> 
> I am writing as someone who has lost data more than once over time, for
> various reasons.  The loss has ranged from slightly annoying, to soul-
> rending catastrophe. It is NEVER appreciated. 
> 
> I do intend to try doing rsync --delete instead of rsync --delete
> after, to see if it "seems to work".  
> 
> But I just wanted to ask first ask, as it would seem better to hear
> someone say "How stupid are you? I can't believe you were going to do
> that!", than to have them say "How stupid are you? I can't believe you
> did that!" 
> 
> 

On Wed, 2024-01-17 at 14:52 -0500, Default User wrote:
> [...]
> By "glitch", I mean anything that could interfere with the rsync copy
> process.  Possible causes: 
> - electrical outages, voltage spikes, voltage drops, "brownouts"
> - mechanical failure
> - earthquake
> - lightning
> - cat walkin

Re: rsync --delete vs rsync --delete-after

[tomas]

On Thu, Jan 18, 2024 at 11:05:01AM +, Michael Kjörling wrote:
> On 17 Jan 2024 20:23 -0500, from hunguponcont...@gmail.com (Default User):

[...]

> Hold on. Let's pause right here.
> 
> **That "primary backup drive" is not a backup at all.**

It is: against the situation you fat-finger something and react
before the next backup happens (this is a threat worth being taken
into account). For that case, a backup with more than one "back"
version would be more useful (see --link-dest in rsync for that).

It is not against the system running amok and thrashing all attached
disks (be it by accident -- improbable, or by malice -- more probable.
cf. crypto-malware). Disconnected back ups are better here.

It is not for against your shed going up in flames. Off site for this.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: rsync --delete vs rsync --delete-after

[Michael Kjörling]

On 17 Jan 2024 20:23 -0500, from hunguponcont...@gmail.com (Default User):
> BTW, the two backup drives are external 4 Gb USB HDDs.  The secondary
> backup drive is always kept away from the computer, in a locked steel
> box, except when it is attached to the computer to have the primary
> backup drive copied to it. 
> 
> The primary backup drive is almost always attached to the computer, so
> that I can access files archived there, that are not on the computer.
> Probably not good practice, but that's why I have the secondary backup
> drive.

Hold on. Let's pause right here.

**That "primary backup drive" is not a backup at all.**

If at any point it can legitimately contain the only copy of a file
that you want to keep, then conceptually it is not a backup.

If it is continuously accessible from the system that you're trying to
protect, then anything bad which happens to that system is liable to
also at least potentially affect your "primary backup drive" as well.

There is nothing wrong with using external storage media to expand the
storage capacity of your computer, but you really shouldn't treat or
consider it as being a backup, because a backup is a _second_ copy to
be used if the primary copy somehow becomes unusable, corrupt,
inaccessible or whatever else might be the case.

Solve the right problem.


> I guess in the back of my mind I was thinking of a scenario where a
> file on the primary backup drive might be corrupted or deleted before
> being copied to the secondary backup drive.  Then if it is not present
> on the primary backup drive, rsync dutifully deletes it from the
> secondary backup drive. If the file is no longer on the computer's
> internal SSD, I am then SOL.

That is why a backup scheme should include some form of retention;
ideally immutable. A single botched backup run should not risk the
integrity of an only backup.


> BTW(2), I do use rsnapshot with cron jobs to back up the internal SSD
> to the primary backup drive daily (and weekly, monthly, yearly).  But I
> am not sure if I could also use it to do copies of the primary backup
> drive to the secondary backup drive (maybe using an additional
> configuration file)? 

You can use rsnapshot with any number of different configuration
files. Just pass `-c` to it to name the configuration file other than
the default. (For a long time I had two, and a script selecting either
one based on the backup target drive in use for that particular
backup. Worked great.)


> As for ZFS . . .   I wish!  But I think the resource requirements would
> be too high for my setup, so it probably would be impractical - or
> impossible.  And  then there's the complexity.  And the learning curve.

ZFS works fine on low-spec'd systems. I use it on a VPS with 2 GB RAM
without any problems. You really want 64-bit, but that's about it.

And while ZFS _can_ be complex, and supports rather complex usage
scenarios (because it is, at its core, an enterprise solution), at the
basic level the biggest difference is that you write "zpool create"
instead of "mkfs.ext4".

-- 
Michael Kjörling 🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: rsync --delete vs rsync --delete-after

[David Christensen]

On 1/17/24 17:23, Default User wrote:

On Wed, 2024-01-17 at 09:19 -0800, David Christensen wrote:

On 1/17/24 08:19, Default User wrote:

Opinions, please.

...
Hi guys, thanks for the replies.



YW.  :-)



BTW, the two backup drives are external 4 Gb USB HDDs.  The secondary
backup drive is always kept away from the computer, in a locked steel
box, except when it is attached to the computer to have the primary
backup drive copied to it.



That means the live data and all the backups are exposed to the same 
computer if and when it is compromised.  It would be safer to do the 
copy on another computer.  If you only have one computer, then boot live 
media to do the copy.




The primary backup drive is almost always attached to the computer, so
that I can access files archived there, that are not on the computer.



Accidental file modification and deletion are probably the most common 
failure modes.



One of the features of ZFS is snapshots.  Snapshots can be accessed via 
the hidden ".zfs/snapshot/" directory in the root of each file system. 
Users can browse the snapshots and copy out whatever files and 
directories they need, subject to the file system permissions at the 
time the snapshot was taken.  So, recovery from the above failure is 
"self serve" -- no sysadmin required, no backup/ restore software required.




Probably not good practice, but that's why I have the secondary backup
drive.

I guess in the back of my mind I was thinking of a scenario where a
file on the primary backup drive might be corrupted or deleted before
being copied to the secondary backup drive.  Then if it is not present
on the primary backup drive, rsync dutifully deletes it from the
secondary backup drive. If the file is no longer on the computer's
internal SSD, I am then SOL.



ZFS snapshots are read-only.  Individual files and directories within a 
snapshot cannot be created, updated, or deleted.



Another feature of ZFS is that both data and metadata are checksummed. 
So, if you can read a file, the metadata and data are good.  "bitrot" is 
extremely unlikely.




I have also thought of trying to use partclone to copy the data from
the primary backup drive to the secondary backup drive. Why not try,



Transferring 4 TB at 150 MB/s would require:

4 TB / 150 MB/s = 26,667 s

Or, about 7.4 hours.



since rsync takes an hour and a half, every day!



How much data is rsync(1) transferring?


Another feature of ZFS snapshots is that they can be replicated from one 
pool to another.  The first replication is full.  Subsequent 
replications are incremental.  Here are some recent transfers from my 
backup server to a removable HDD:


frequency  bytes   real timebandwidth
=  ==  ===  =
weekly 11,125,755,324   25m08.848s  7.37 MB/s
monthly75,218,102,216  140m53.710s  8.90 MB/s



As for ZFS . . .   I wish!  But I think the resource requirements would
be too high for my setup, so it probably would be impractical - or
impossible. 



ZFS will use as much or as little resources as you provide.


(ZFS is known to consume most of available memory OOTB; this can be 
tuned.  The simple answer is to put it on a dedicated file server/ NAS.)



Please tell us about your setup -- hardware, stored data, and I/O workload.



And  then there's the complexity.



I find ZFS has more conceptual integrity than cobbling together 
fdisk(8), mdadm(8), cryptsetup(8), lvm(8), mkfs(8), etc..




And the learning curve.



The Lucas books got me up the learning curve:

https://mwl.io/nonfiction/os#fmzfs

https://mwl.io/nonfiction/os#fmaz


While both have "FreeBSD" in the title, most of the content applies 
equally to OpenZFS on Debian.




Finally I really should have a third backup drive in the mix.  Yes, I
am familiar with the 1-2-3 backup theory.  But the third backup drive
could not be off-site, for various reasons.  And I do have other things
to do rather than spend all day, every day, managing backups.



Disaster preparedness is an open-ended problem.  Only you can decide how 
much solution to give it.



David

Re: rsync --delete vs rsync --delete-after

[Stefan Monnier]

> However, I have read that using rsync --delete instead of rsync --
> delete-after is faster and uses less memory, and so is more efficient. 

I'd be surprised if it makes a significant difference.


Stefan

Re: rsync --delete vs rsync --delete-after

[Michel Verdier]

On 2024-01-17, Default User wrote:

> By "glitch", I mean anything that could interfere with the rsync copy
> process.  Possible causes: 

Whatever the cause you just have to get return code and restart rsync
until it complete succesfully. Then you are sure to have an exact copy.
To cope with errors in primary backup you must have multiple copies as
suggest Michael. Personnally I use rsnapshot (a tool using rsync itself)
for that.

Re: rsync --delete vs rsync --delete-after

[Default User]

On Wed, 2024-01-17 at 09:19 -0800, David Christensen wrote:
> On 1/17/24 08:19, Default User wrote:
> > Hello!
> > 
> > Opinions, please.
> > 
> > I use rsync to copy my primary backup drive to a secondary backup
> > drive
> > , so that the secondary backup drive is theoretically always an
> > exact
> > copy of the primary backup drive.
> > 
> > Here is the rsync command I use:
> > 
> > time sudo rsync -aAXHxvv --delete-after --numeric-ids --
> > info=progress2,stats2,name2 --
> > exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/m
> > edia
> > /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/
> > 
> > Question:
> > I use rsync --delete-after because it might seem to be "safer", so
> > in
> > case of a "glitch" of any kind, no file ever disappears from both
> > the
> > source drive and the destination drive.
> > 
> > However, I have read that using rsync --delete instead of rsync --
> > delete-after is faster and uses less memory, and so is more
> > efficient.
> > 
> > Note: The current copy process time varies, but takes a long time -
> > last night 131 minutes.
> > :(
> > 
> > Disk space used is not currently an issue.
> > 
> > But, is rsync --delete AS SAFE as rsync --delete-after?
> 
> 
> In the past, I used the --backup and --backup-dir options to retain 
> files on the destination.
> 
> 
> Then I moved my primary backup to ZFS, implemented snapshots, and 
> implemented replication to the secondard backup devices.
> 
> 
> David
> 



Hi guys, thanks for the replies.

BTW, the two backup drives are external 4 Gb USB HDDs.  The secondary
backup drive is always kept away from the computer, in a locked steel
box, except when it is attached to the computer to have the primary
backup drive copied to it. 

The primary backup drive is almost always attached to the computer, so
that I can access files archived there, that are not on the computer.
Probably not good practice, but that's why I have the secondary backup
drive.

I guess in the back of my mind I was thinking of a scenario where a
file on the primary backup drive might be corrupted or deleted before
being copied to the secondary backup drive.  Then if it is not present
on the primary backup drive, rsync dutifully deletes it from the
secondary backup drive. If the file is no longer on the computer's
internal SSD, I am then SOL.

BTW(2), I do use rsnapshot with cron jobs to back up the internal SSD
to the primary backup drive daily (and weekly, monthly, yearly).  But I
am not sure if I could also use it to do copies of the primary backup
drive to the secondary backup drive (maybe using an additional
configuration file)? 

I have also thought of trying to use partclone to copy the data from
the primary backup drive to the secondary backup drive. Why not try,
since rsync takes an hour and a half, every day! 

As for ZFS . . .   I wish!  But I think the resource requirements would
be too high for my setup, so it probably would be impractical - or
impossible.  And  then there's the complexity.  And the learning curve.

Finally I really should have a third backup drive in the mix.  Yes, I
am familiar with the 1-2-3 backup theory.  But the third backup drive
could not be off-site, for various reasons.  And I do have other things
to do rather than spend all day, every day, managing backups. 



 

Re: rsync --delete vs rsync --delete-after

[Keith Bainbridgge]

On 18/1/24 04:19, David Christensen wrote:
> I use rsync to copy my primary backup drive to a secondary backup drive


Good morning

I wonder why both processes don't copy from the original data; so that 
you don't copy a potential glitch in the first backup?


on a separate matter
Glitch?  Power goes off inconveniently?
--
All the best

Keith Bainbridge

keithr...@gmail.com
keith.bainbridge.3...@gmail.com
+61 (0)447 667 468

UTC + 10:00

Re: rsync --delete vs rsync --delete-after

[Andy Smith]

Hi,

On Wed, Jan 17, 2024 at 02:52:49PM -0500, Default User wrote:
> By "glitch", I mean anything that could interfere with the rsync copy
> process.  Possible causes: 
> - electrical outages, voltage spikes, voltage drops, "brownouts"
> - mechanical failure
> - earthquake
> - lightning
> - cat walking on keyboard
> - out of memory errors
> - out of disk space errors
> - PEBKAC errors
> - etc.

But, both --delete and --delete-after only delete things from the
destination that ALREADY are missing on the source, so in which of
the above situations would something that still exists somewhere be
accidentally deleted with either option?

In my view the only ones that apply would be human error / cat
standard behaviour but even then you'd have to notice it had
happened and abort the transfer before rsync has chance to do the
delete. Do you typically sit and watch these transfers? It doesn't
sound like any kind of backup to me if it doesn't have history,
i.e. the state of your data *before* the cat walked on 'r' 'm' ' '
'-' 'f' 'r' ' ' '.' ''.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: rsync --delete vs rsync --delete-after

[Michael Kjörling]

On 17 Jan 2024 14:52 -0500, from hunguponcont...@gmail.com (Default User):
> I am writing as someone who has lost data more than once over time, for
> various reasons.  The loss has ranged from slightly annoying, to soul-
> rending catastrophe. It is NEVER appreciated. 

I think this gets closer to the root of what you're trying to achieve:
it sounds to me as though no matter what happens, you want to have a
restorable backup which you can trust to (reasonably well) match the
state of the source at the time when that backup was taken. That is a
commendable goal.

Twiddling with options to rsync won't offer you that in light of
several of the scenarios you listed, though; not least a lightning
strike. A lightning strike will blow out the drive just as much no
matter what options you're using to invoke rsync.

What you need is really rather multiple copies.

I suggest to get a second drive to use for backup purposes along with
the one you currently have. Only ever connect one of them at the same
time to data or power anywhere. Ideally, always keep at least one of
them in a separate location, powered off and disconnected.

That way, if you mess up one, or if one of them fails (for any
reason), or whatever else might happen, you have the other. It might
not be quite as up to date, but it will be in a known-good state.

For less drastic failures, really, look at rsnapshot. It's a wrapper
around rsync which makes maintaining multiple revisions of a backup
much easier. (It essentially passes to rsync with --link-dest, and
manages the respective root directories.)

-- 
Michael Kjörling 🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: rsync --delete vs rsync --delete-after

[Default User]

On Wed, 2024-01-17 at 10:29 -0800, Kushal Kumaran wrote:
> On Wed, Jan 17 2024 at 11:19:39 AM, Default User
>  wrote:
> > Hello!
> > 
> > Opinions, please.
> > 
> > I use rsync to copy my primary backup drive to a secondary backup
> > drive
> > , so that the secondary backup drive is theoretically always an
> > exact
> > copy of the primary backup drive.  
> > 
> > Here is the rsync command I use:
> > 
> > time sudo rsync -aAXHxvv --delete-after --numeric-ids --
> > info=progress2,stats2,name2 --
> > exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/m
> > edia
> > /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/
> > 
> > Question: 
> > I use rsync --delete-after because it might seem to be "safer", so
> > in
> > case of a "glitch" of any kind, no file ever disappears from both
> > the 
> > source drive and the destination drive.  
> > 
> 
> What do you mean by "glitch"?  Irrespective of whether you use --
> delete
> or --delete-after, deleted files on the source are deleted on the
> destination once your rsync is complete (which is what I'd assume you
> want when you want an exact copy).  I'd presume if you're ok with
> that,
> you are also fine with the deletion happening earlier in the rsync
> process?
> 
> If you're concerned about accidental deletions, you should just not
> use
> any of the `--delete*` options (and give up on the exact copy
> requirement).  You can look at alternatives to bare rsync that keep
> track of multiple backed-up images (rsnapshot is a very simple
> wrapper
> over rsync that can do this, for example).
> 
> > However, I have read that using rsync --delete instead of rsync --
> > delete-after is faster and uses less memory, and so is more
> > efficient. 
> > 
> > Note: The current copy process time varies, but takes a long time -
> > last night 131 minutes.
> > :(
> 
> You can try using --delete for a couple of runs and see if it
> actually
> affects performance in your situation.
> 
> > 
> > Disk space used is not currently an issue.
> > 
> > But, is rsync --delete AS SAFE as rsync --delete-after?
> 
> You'll need to define what safety means for you.
> 



Hi, Kushal! 
Thanks for replying. 

By "glitch", I mean anything that could interfere with the rsync copy
process.  Possible causes: 
- electrical outages, voltage spikes, voltage drops, "brownouts"
- mechanical failure
- earthquake
- lightning
- cat walking on keyboard
- out of memory errors
- out of disk space errors
- PEBKAC errors
- etc.

By safe, may I try to explain using a story? 

I once read that centuries ago, a king wanted his crown to be safe.  So
four guards watched his crown at all times.  The guards were not
allowed to take their eves off of the crown, even for a second, until
the the their replacements, the next group of guards, all said "I see
the crown". 

I am sorry if I can not come up with a better, technical explanation. 
I use this story because it has always been meaningful to me, and seems
to point to the essence of what I am getting at. 

I am writing as someone who has lost data more than once over time, for
various reasons.  The loss has ranged from slightly annoying, to soul-
rending catastrophe. It is NEVER appreciated. 

I do intend to try doing rsync --delete instead of rsync --delete
after, to see if it "seems to work".  

But I just wanted to ask first ask, as it would seem better to hear
someone say "How stupid are you? I can't believe you were going to do
that!", than to have them say "How stupid are you? I can't believe you
did that!" 

Re: rsync --delete vs rsync --delete-after

[Kushal Kumaran]

On Wed, Jan 17 2024 at 11:19:39 AM, Default User  
wrote:
> Hello!
>
> Opinions, please.
>
> I use rsync to copy my primary backup drive to a secondary backup drive
> , so that the secondary backup drive is theoretically always an exact
> copy of the primary backup drive.  
>
> Here is the rsync command I use:
>
> time sudo rsync -aAXHxvv --delete-after --numeric-ids --
> info=progress2,stats2,name2 --
> exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media
> /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/
>
> Question: 
> I use rsync --delete-after because it might seem to be "safer", so in
> case of a "glitch" of any kind, no file ever disappears from both the 
> source drive and the destination drive.  
>

What do you mean by "glitch"?  Irrespective of whether you use --delete
or --delete-after, deleted files on the source are deleted on the
destination once your rsync is complete (which is what I'd assume you
want when you want an exact copy).  I'd presume if you're ok with that,
you are also fine with the deletion happening earlier in the rsync
process?

If you're concerned about accidental deletions, you should just not use
any of the `--delete*` options (and give up on the exact copy
requirement).  You can look at alternatives to bare rsync that keep
track of multiple backed-up images (rsnapshot is a very simple wrapper
over rsync that can do this, for example).

> However, I have read that using rsync --delete instead of rsync --
> delete-after is faster and uses less memory, and so is more efficient. 
>
> Note: The current copy process time varies, but takes a long time -
> last night 131 minutes.
> :(

You can try using --delete for a couple of runs and see if it actually
affects performance in your situation.

>
> Disk space used is not currently an issue.
>
> But, is rsync --delete AS SAFE as rsync --delete-after?

You'll need to define what safety means for you.

-- 
regards,
kushal

Re: rsync --delete vs rsync --delete-after

[David Christensen]

On 1/17/24 08:19, Default User wrote:

Hello!

Opinions, please.

I use rsync to copy my primary backup drive to a secondary backup drive
, so that the secondary backup drive is theoretically always an exact
copy of the primary backup drive.

Here is the rsync command I use:

time sudo rsync -aAXHxvv --delete-after --numeric-ids --
info=progress2,stats2,name2 --
exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media
/*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/

Question:
I use rsync --delete-after because it might seem to be "safer", so in
case of a "glitch" of any kind, no file ever disappears from both the
source drive and the destination drive.

However, I have read that using rsync --delete instead of rsync --
delete-after is faster and uses less memory, and so is more efficient.

Note: The current copy process time varies, but takes a long time -
last night 131 minutes.
:(

Disk space used is not currently an issue.

But, is rsync --delete AS SAFE as rsync --delete-after?



In the past, I used the --backup and --backup-dir options to retain 
files on the destination.



Then I moved my primary backup to ZFS, implemented snapshots, and 
implemented replication to the secondard backup devices.



David

rsync --delete vs rsync --delete-after

[Default User]

Hello!

Opinions, please.

I use rsync to copy my primary backup drive to a secondary backup drive
, so that the secondary backup drive is theoretically always an exact
copy of the primary backup drive.  

Here is the rsync command I use:

time sudo rsync -aAXHxvv --delete-after --numeric-ids --
info=progress2,stats2,name2 --
exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media
/*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/

Question: 
I use rsync --delete-after because it might seem to be "safer", so in
case of a "glitch" of any kind, no file ever disappears from both the 
source drive and the destination drive.  

However, I have read that using rsync --delete instead of rsync --
delete-after is faster and uses less memory, and so is more efficient. 

Note: The current copy process time varies, but takes a long time -
last night 131 minutes.
:(

Disk space used is not currently an issue.

But, is rsync --delete AS SAFE as rsync --delete-after?

Re: Debian archive rsync service changed

[Byung-Hee HWANG]

On Fri, Nov 24, 2023 at 08:13:03AM -0500, Dan Ritter wrote:
> In case people didn't see the announcment:
> --
> 
> The worldwide Debian mirrors network has served archive.debian.org via both 
> HTTP and rsync. As part of improving the reliability of the service for 
> users, the Debian mirrors team is separating the access methods to different 
> host names:
> 
> http://archive.debian.org/ will remain the entry point for HTTP clients such 
> as APT
> 
> rsync://rsync.archive.debian.org/debian-archive/ is now available for those 
> who wish to mirror all or parts of the archives.
> 
> rsync service on archive.debian.org has stopped, and we encourage anyone 
> using the service to migrate to the new host name as soon as possible.
> --
> 
> 
> I am not responsible for any part of this, I'm just relaying the
> news.
> 

Thanks!

Sincerely, Byung-Hee

Debian archive rsync service changed

[Dan Ritter]

In case people didn't see the announcment:
--

The worldwide Debian mirrors network has served archive.debian.org via both 
HTTP and rsync. As part of improving the reliability of the service for users, 
the Debian mirrors team is separating the access methods to different host 
names:

http://archive.debian.org/ will remain the entry point for HTTP clients such as 
APT

rsync://rsync.archive.debian.org/debian-archive/ is now available for those who 
wish to mirror all or parts of the archives.

rsync service on archive.debian.org has stopped, and we encourage anyone using 
the service to migrate to the new host name as soon as possible.
--


I am not responsible for any part of this, I'm just relaying the
news.

-dsr-

Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

[Manphiz]

Jason  writes:

> Hi
>
> I was a user of OpenMediaVault for several years. I even donated money to the
> developer.
>
> But very provocatively OpenMediaVault is bloatware, way too big. The only 
> thing
> I need is a reliable backup.
>
> I had pure Debian (minimal installation, very few packages) installed with
> borgbackup and rsync.
>
> I am now very satisfied, much more streamlined than OpenMediaVault.
>
> Am I wrong, are my statements completely off?
>
> Or how does your backup look like?
>
> cheers
> Jason
>
>

Another OMV user for several years and have been happy about it.  It's a
NAS system based on Debian (latest version based on Buster), on top of
which it provides its NAS management layer and you can control what to
install or not.  There is no ads, no telemetry, and it manages my ZFS
raidz1 cluster without any issue for several years, and the system
itself takes less than 5GB space, so I'm not sure why you'd call it
bloatware.

-- 
Manphiz

Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

[jeremy ardley]

On 1/9/23 12:44, Jason wrote:


Or how does your backup look like?



I had a QNAP NAS but it became so incredibly slow I replaced it with 
Debian using Samba and SSH.


The backups are managed by the clients, but periodically I save part of 
the NAS to Amazon S3.


I also have a remote Nextcloud server which my clients use to share a 
small working set of documents and files. It's on Akami (nee Linode) and 
is very fast in my region and pretty cheap. I used to do this on AWS but 
Linode is cheaper and simpler for my applications


I'm also in the process of converting some of my storage from Dropbox to 
Nextcloud

I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsync

[Jason]

Hi

I was a user of OpenMediaVault for several years. I even donated money 
to the developer.


But very provocatively OpenMediaVault is bloatware, way too big. The 
only thing I need is a reliable backup.


I had pure Debian (minimal installation, very few packages) installed 
with borgbackup and rsync.


I am now very satisfied, much more streamlined than OpenMediaVault.

Am I wrong, are my statements completely off?

Or how does your backup look like?

cheers
Jason


OpenPGP_0x0D0C34B5DF58FE9D.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: syncthing, rsync for git; was: git setup

[Max Nikulin]

On 23/08/2023 02:27, Michael Kjörling wrote:

I know of rsync's shortcomings in the bidirectional-sync use case
because I looked for a good while for a way to get it to do that
safely, before coming across unison which being designed for that
solved that problem with for all intents and purposes no fuss at all.


Almost every version control system (besides legacy ones, e.g. RCS) is 
designed to combine changes from multiple working copies evolving in 
parallel.


Of course, git facilities allows to synchronize multiple repositories 
with minimal risk to loose changes committed to any copy. The idea has 
been posted in this thread already, see

Sven Joachim. Tue, 22 Aug 2023 17:28:40 +0200.
https://lists.debian.org/msgid-search/87h6orf653@turtle.gmx.de
setup an "upstream" *bare* repository that may reside on the same 
machine or accessed through ssh and add it using "git remote" to 
existing working copies (new ones created by "git clone" will have it).


You may need to set tracking for existing branches. Generally you may 
follow guides for git hosting services, some open source projects have 
their own introductions for git novices. You may e.g. choose a branch 
per working copy to be able to combine commits from any of them. Balance 
of "git merge" and "git rebase" is mater of taste in respect to linear 
commit history.


A policy for a *backup* repository may be different from an "upstream" 
one. Backup repositories should allow to recover after accidental force 
push to some old commit. Warning: "git push --mirror" does force push at 
least by default, use it with care. You may either fetch or push to 
backup repository depending on current working directory. When used 
properly, it is safer than general purpose synchronizing tools.


A single branch may be pushed to an "upstream" or a "backup" repository 
without permanently adding it as a remote by specifying path to that 
repository


git push user@host:/path/to/repository.git master

I would limit usage of rsync and similar tools to rather specific cases 
when git repositories are involved:

- copy data when you are going to decommission a disk
- working copy contains untracked files having some value or temporary 
branches undesired in upstream and backup repositories.

Re: syncthing, rsync for git; was: git setup

[Celejar]

On Tue, 22 Aug 2023 19:27:57 +
Michael Kjörling <2695bd53d...@ewoof.net> wrote:

> On 22 Aug 2023 14:33 -0400, from cele...@gmail.com (Celejar):
> >> Git tends to be very rsync-friendly.
> > 
> > I do something similar - I use syncthing to automatically keep the git
> > repositories on two of my machines in sync. rsync may be better, but
> > syncthing has more or less worked for me.
> 
> I'm not really familiar with syncthing, but it looks like it and rsync
> solve somewhat different problems; rsync being primarily intended to
> update one location (the destination) to match another (the source),
> whereas syncthing is primarily intended to update both locations such
> that they match (but can be run in one-way mode if desired).
> 
> Therefore syncthing would seem to be more analogous to unison than to
> rsync.

Correct. My use case is two systems both used for development
(I work sometimes on a laptop and sometimes on a desktop). I understand
that this is not the OP's case and the subject of the thread, and I
apologize for the confusion.

> I know of rsync's shortcomings in the bidirectional-sync use case
> because I looked for a good while for a way to get it to do that
> safely, before coming across unison which being designed for that
> solved that problem with for all intents and purposes no fuss at all.

-- 
Celejar

Re: syncthing, rsync for git; was: git setup

[Michael Kjörling]

On 22 Aug 2023 14:33 -0400, from cele...@gmail.com (Celejar):
>> Git tends to be very rsync-friendly.
> 
> I do something similar - I use syncthing to automatically keep the git
> repositories on two of my machines in sync. rsync may be better, but
> syncthing has more or less worked for me.

I'm not really familiar with syncthing, but it looks like it and rsync
solve somewhat different problems; rsync being primarily intended to
update one location (the destination) to match another (the source),
whereas syncthing is primarily intended to update both locations such
that they match (but can be run in one-way mode if desired).

Therefore syncthing would seem to be more analogous to unison than to
rsync.

I know of rsync's shortcomings in the bidirectional-sync use case
because I looked for a good while for a way to get it to do that
safely, before coming across unison which being designed for that
solved that problem with for all intents and purposes no fuss at all.

-- 
Michael Kjörling 🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: "(Server) Protocol versions: remote=31, negotiated=31". How to you rsync just certain file extensions? . . .

[Albretch Mueller]

On 3/11/23, Greg Wooledge  wrote:
...
> Hmm, that didn't work either.  Now it's time to read the documentation.
> (read, read, read)
> OK, here we go:
>
> unicorn:/tmp/src$ rsync -a --include="*/" --include="*.foo"
> --include="*.bar" --exclude="*" . /tmp/dest/
> unicorn:/tmp/src$ find /tmp/dest
> /tmp/dest
> /tmp/dest/a
> /tmp/dest/a/b2
> /tmp/dest/a/b2/good.bar
> /tmp/dest/a/b
> /tmp/dest/a/b/c
> /tmp/dest/a/b/c/good.foo
>
> There.  That's the desired result, if my interpretation of the Subject:
> header is correct.
> In order to understand it, you'll have to read the section titled
> "INCLUDE/EXCLUDE PATTERN RULES" in the man page.  I won't repeat it
> all here.
> If there are additional questions, please try to ask them clearly.

 that did it for me. Thank you.
 I had all that script that I had been using just fine, but at some
point it started to "misbehave"
 I use all that "obfuscation" because I like to keep logs of all I do

 lbrtchx

Re: "(Server) Protocol versions: remote=31, negotiated=31". How to you rsync just certain file extensions? . . .

[Greg Wooledge]

OK.  Let me just discard every single piece of quoted content from this
thread, because none of it makes ANY sense.

Here is what I see in the Subject: header:

1) Some piece of the verbose output of rsync, for unknown reason.
2) A question about how to do a specific task with rsync.

At some point the body of the thread mentioned an "error message" but
did not show one.  It also gave a code snippet with incredibly large
amounts of unnecessary obfuscation.  I'm going to ignore that.

As far as I can tell, the question is:

  "Given a directory hierarchy, I would like to rsync this hierarchy
  to a new location, but ignore all files except the ones ending with
  .foo and .bar"

Now let's see how to do that.  "man rsync" shows us the following options:

   --exclude=PATTERNexclude files matching PATTERN
   --exclude-from=FILE  read exclude patterns from FILE
   --include=PATTERNdon't exclude files matching PATTERN
   --include-from=FILE  read include patterns from FILE

Most likely we'll want one or two of those.  Now let's set up a test.

unicorn:~$ mkdir /tmp/src && cd "$_"
unicorn:/tmp/src$ mkdir -p a/b/c a/b2
unicorn:/tmp/src$ touch a/b/c/good.foo a/b/c/bad.bad a/b2/good.bar a/b2/notgood
unicorn:/tmp/src$ find .
.
./a
./a/b2
./a/b2/good.bar
./a/b2/notgood
./a/b
./a/b/c
./a/b/c/good.foo
./a/b/c/bad.bad

And a destination directory:

unicorn:/tmp/src$ mkdir /tmp/dest

Now let's try the most obvious thing I can think of:

unicorn:/tmp/src$ rsync -a --include="*.foo" --include="*.bar" . /tmp/dest/
unicorn:/tmp/src$ find /tmp/dest
/tmp/dest
/tmp/dest/a
/tmp/dest/a/b2
/tmp/dest/a/b2/good.bar
/tmp/dest/a/b2/notgood
/tmp/dest/a/b
/tmp/dest/a/b/c
/tmp/dest/a/b/c/good.foo
/tmp/dest/a/b/c/bad.bad

All right, that didn't work.  Now let's try the second most obvious thing:

unicorn:/tmp/src$ rm -rf /tmp/dest
unicorn:/tmp/src$ mkdir /tmp/dest
unicorn:/tmp/src$ rsync -a --exclude="*" --include="*.foo" --include="*.bar" . 
/tmp/dest/
unicorn:/tmp/src$ find /tmp/dest
/tmp/dest

Hmm, that didn't work either.  Now it's time to read the documentation.

(read, read, read)

OK, here we go:

unicorn:/tmp/src$ rsync -a --include="*/" --include="*.foo" --include="*.bar" 
--exclude="*" . /tmp/dest/
unicorn:/tmp/src$ find /tmp/dest
/tmp/dest
/tmp/dest/a
/tmp/dest/a/b2
/tmp/dest/a/b2/good.bar
/tmp/dest/a/b
/tmp/dest/a/b/c
/tmp/dest/a/b/c/good.foo

There.  That's the desired result, if my interpretation of the Subject:
header is correct.

In order to understand it, you'll have to read the section titled
"INCLUDE/EXCLUDE PATTERN RULES" in the man page.  I won't repeat it
all here.

If there are additional questions, please try to ask them clearly.

Re: "(Server) Protocol versions: remote=31, negotiated=31". How to you rsync just certain file extensions? . . .

[Albretch Mueller]

On 3/11/23, Dan Ritter  wrote:
> Albretch Mueller wrote:
>>  The one liner reporting that error is:
>>
>>   time( sudo rsync --rsync-path="/usr/local/bin/rsync" --debug=ALL
>> --archive --verbose --compress --recursive  --checksum --include="*/"
>> --include=".${_X}" --exclude="*" --prune-empty-dirs  "${_SRC}"
>> "${_DST}"  1> "${_LOG_FL}" 2> "${_ERRS_LOG}" ) >> "${_TM_LOG}" 2>&1
>>
>>  I found some useful info here:
>>
>> https://stackoverflow.com/questions/1562/rsync-copy-over-only-certain-types-of-files-using-include-option
>>
>>  but not much of explaning about that particular error.
>>
>>  You could always go monkey and find those files in the source
>> directory to then copy them to the destination (after some easy
>> cooking with the paths), but I think rsync should be able to do that
>
> Are you setting that rsync-path on purpose? Why?
>
> If _DST is on some other machine, can you ssh to it?
>
> -dsr-

 I couldn't make any sense of it, so I started following all kinds of
"esoteric" suggestions and guesses. I read off some post trying to
help someone out of that kind of error message. The same error message
happened without that --rsync-path="/usr/local/bin/rsync" option.

_DST is just an external drive, which I definitely know to be there
connected just fine physically and logically.

$ which rsync
/usr/bin/rsync
user@debian:~$ rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
symtimes, prealloc, stop-at, no crtimes
Optimizations:
SIMD, asm, openssl-crypto
Checksum list:
xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
$

Re: "(Server) Protocol versions: remote=31, negotiated=31". How to you rsync just certain file extensions? . . .

[Dan Ritter]

Albretch Mueller wrote: 
>  The one liner reporting that error is:
> 
>   time( sudo rsync --rsync-path="/usr/local/bin/rsync" --debug=ALL
> --archive --verbose --compress --recursive  --checksum --include="*/"
> --include=".${_X}" --exclude="*" --prune-empty-dirs  "${_SRC}"
> "${_DST}"  1> "${_LOG_FL}" 2> "${_ERRS_LOG}" ) >> "${_TM_LOG}" 2>&1
> 
>  I found some useful info here:
>  
> https://stackoverflow.com/questions/1562/rsync-copy-over-only-certain-types-of-files-using-include-option
> 
>  but not much of explaning about that particular error.
> 
>  You could always go monkey and find those files in the source
> directory to then copy them to the destination (after some easy
> cooking with the paths), but I think rsync should be able to do that

Are you setting that rsync-path on purpose? Why?

If _DST is on some other machine, can you ssh to it? 

-dsr-

"(Server) Protocol versions: remote=31, negotiated=31". How to you rsync just certain file extensions? . . .

[Albretch Mueller]

 The one liner reporting that error is:

  time( sudo rsync --rsync-path="/usr/local/bin/rsync" --debug=ALL
--archive --verbose --compress --recursive  --checksum --include="*/"
--include=".${_X}" --exclude="*" --prune-empty-dirs  "${_SRC}"
"${_DST}"  1> "${_LOG_FL}" 2> "${_ERRS_LOG}" ) >> "${_TM_LOG}" 2>&1

 I found some useful info here:
 
https://stackoverflow.com/questions/1562/rsync-copy-over-only-certain-types-of-files-using-include-option

 but not much of explaning about that particular error.

 You could always go monkey and find those files in the source
directory to then copy them to the destination (after some easy
cooking with the paths), but I think rsync should be able to do that

 lbrtchx

Re: nocache nice ionice -c3 ... chrt? for rsync.

[Dan Ritter]

Samuel Wales wrote: 
> i seem to get sluggish interactive x pointer etc. on rare occasions
> with rsync.  i use nocache nice ionice -c3.  i am wondering why chrt
> exists also, and what teh best set of options is for this kind of
> purpose.


Tell us about your hardware and network.

lscpu

free

 ip -s -s l show (narrow this one down to your primary
interfaces)

What disk technology is being used?

How much data are you moving through rsync? How long does it
usually take? How far is it going?

-dsr-

nocache nice ionice -c3 ... chrt? for rsync.

[Samuel Wales]

i seem to get sluggish interactive x pointer etc. on rare occasions
with rsync.  i use nocache nice ionice -c3.  i am wondering why chrt
exists also, and what teh best set of options is for this kind of
purpose.

-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: rsync trying to load rdiff-backup

[AC]

On 2022-05-08 18:27, AC wrote:

On 2022-05-08 18:03, David wrote:

On Mon, 9 May 2022 at 10:53, AC  wrote:

On 2022-05-08 16:45, David wrote:

On Mon, 9 May 2022 at 07:24, AC  wrote:



Now here's a new discovery on the machines having problems: as a normal
user I can rsync just fine to a remote machine.  But if I'm using rsync
as root to send a backup to a remote machine, I get the rdiff-backup
problem.  The other machines have no issues whether I'm a regular user
or root.


Ok. In that case, I would run again the check commands I gave, but this
time as root, just to see if there is any difference.

I don't know the cause of your problem, but these commands are useful
investigation to reveal what "rsync" means when the particular user runs
that command.


All of my tests were done as root already.  They all matched.




well I feel stupid but the problem ended up on the server side of this 
mess.  I had actually written command restrictions into the 
authorized_keys file on the server so that the unattended backups would 
use SSH keys that couldn't be used for anything else.  I did this ages 
ago and completely forgot about it.  The two machines that worked are 
brand new so they didn't have the keys set up and I was manually 
entering passwords when prompted just for testing.

Re: rsync trying to load rdiff-backup

[AC]

On 2022-05-08 18:03, David wrote:

On Mon, 9 May 2022 at 10:53, AC  wrote:

On 2022-05-08 16:45, David wrote:

On Mon, 9 May 2022 at 07:24, AC  wrote:



Now here's a new discovery on the machines having problems: as a normal
user I can rsync just fine to a remote machine.  But if I'm using rsync
as root to send a backup to a remote machine, I get the rdiff-backup
problem.  The other machines have no issues whether I'm a regular user
or root.


Ok. In that case, I would run again the check commands I gave, but this
time as root, just to see if there is any difference.

I don't know the cause of your problem, but these commands are useful
investigation to reveal what "rsync" means when the particular user runs
that command.


All of my tests were done as root already.  They all matched.

Re: rsync trying to load rdiff-backup

[David]

On Mon, 9 May 2022 at 10:53, AC  wrote:
> On 2022-05-08 16:45, David wrote:
> > On Mon, 9 May 2022 at 07:24, AC  wrote:

> Now here's a new discovery on the machines having problems: as a normal
> user I can rsync just fine to a remote machine.  But if I'm using rsync
> as root to send a backup to a remote machine, I get the rdiff-backup
> problem.  The other machines have no issues whether I'm a regular user
> or root.

Ok. In that case, I would run again the check commands I gave, but this
time as root, just to see if there is any difference.

I don't know the cause of your problem, but these commands are useful
investigation to reveal what "rsync" means when the particular user runs
that command.

Re: rsync trying to load rdiff-backup

[AC]

On 2022-05-08 16:45, David wrote:

On Mon, 9 May 2022 at 07:24, AC  wrote:


Due to some version mismatch issues I'm trying to switch over to rsync
from rdiff-backup for backing up some files on my systems.

On most of them this has worked with no issue but I have one system that
appears to be trying to load rdiff-backup whenever I run rsync:

root:~# /usr/bin/rsync testfile u...@remote.host:testfile
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 20, in 
  import rdiff_backup.Main
ModuleNotFoundError: No module named 'rdiff_backup'
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235)
[sender=3.1.3]

I don't understand why it's trying to load rdiff-backup, the traceback
is a giveaway that it's running Python but I explicitly called out the
path to rsync and I know it's not a python script:

root:~# less /usr/bin/rsync
"/usr/bin/rsync" may be a binary file.  See it anyway?
^?ELF^A^A^A^@^@^

This is all happening on Buster but I have three other Buster systems
that ran with rdiff-backup and I can run rsync fine on them without
rdiff-backup running.  What's the deal here?


Hi, answering this might require knowledge of rdiff-backup, which I
don't have. But the first thing I would check is to compare the
/usr/bin/rsync file on the bad system with a good system.

Do you get the same output as my four commands below?

$ cat /etc/debian_version
11.3
$ rsync --version
rsync  version 3.2.3  protocol version 31
[...extra output elided...]
$ type rsync
rsync is hashed (/usr/bin/rsync)
$ md5sum /usr/bin/rsync
b09b45b5eb0ff5275805aadff1d0ed86  /usr/bin/rsync
$




Comparing the broken system with a good system:
debian_version matches: 10.12
rsync --version matches: rsync  version 3.1.3  protocol version 31
type matches: rsync is hashed (/usr/bin/rsync)
md5sum matches: 400d9c7e740e1ba06ebf779a7a10fc00  /usr/bin/rsync

As far as I can tell everything is identical but it seems that starting 
the rsync command somehow there is something intercepting and 
redirecting to execute rdiff-backup.



I went and compared all of my systems which had rdiff-backup installed. 
 All of them are Buster varying between 10.11 and 10.12 and a mix of 
them are having the same problem.  Three of the machines are 10.11, two 
work fine and the other is trying to look for rdiff-backup when running 
rsync.  The others are 10.12 with a similar issue, all of them are 
working except for one.  All the machines share similar configurations, 
usually a vanilla installation (no desktop environment) followed by 
whatever services each one needs (e.g. mail, ssh, etc.) and for 
something like backup they would all share the same configuration 
because they all run their backups to the same server.



Now here's a new discovery on the machines having problems: as a normal 
user I can rsync just fine to a remote machine.  But if I'm using rsync 
as root to send a backup to a remote machine, I get the rdiff-backup 
problem.  The other machines have no issues whether I'm a regular user 
or root.  What would change the behavior of a root user versus a regular 
user?

Re: rsync trying to load rdiff-backup

[David]

On Mon, 9 May 2022 at 07:24, AC  wrote:
>
> Due to some version mismatch issues I'm trying to switch over to rsync
> from rdiff-backup for backing up some files on my systems.
>
> On most of them this has worked with no issue but I have one system that
> appears to be trying to load rdiff-backup whenever I run rsync:
>
> root:~# /usr/bin/rsync testfile u...@remote.host:testfile
> Traceback (most recent call last):
>File "/usr/bin/rdiff-backup", line 20, in 
>  import rdiff_backup.Main
> ModuleNotFoundError: No module named 'rdiff_backup'
> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
> rsync error: error in rsync protocol data stream (code 12) at io.c(235)
> [sender=3.1.3]
>
> I don't understand why it's trying to load rdiff-backup, the traceback
> is a giveaway that it's running Python but I explicitly called out the
> path to rsync and I know it's not a python script:
>
> root:~# less /usr/bin/rsync
> "/usr/bin/rsync" may be a binary file.  See it anyway?
> ^?ELF^A^A^A^@^@^
>
> This is all happening on Buster but I have three other Buster systems
> that ran with rdiff-backup and I can run rsync fine on them without
> rdiff-backup running.  What's the deal here?

Hi, answering this might require knowledge of rdiff-backup, which I
don't have. But the first thing I would check is to compare the
/usr/bin/rsync file on the bad system with a good system.

Do you get the same output as my four commands below?

$ cat /etc/debian_version
11.3
$ rsync --version
rsync  version 3.2.3  protocol version 31
[...extra output elided...]
$ type rsync
rsync is hashed (/usr/bin/rsync)
$ md5sum /usr/bin/rsync
b09b45b5eb0ff5275805aadff1d0ed86  /usr/bin/rsync
$

rsync trying to load rdiff-backup

[AC]

Due to some version mismatch issues I'm trying to switch over to rsync 
from rdiff-backup for backing up some files on my systems.


On most of them this has worked with no issue but I have one system that 
appears to be trying to load rdiff-backup whenever I run rsync:


root:~# /usr/bin/rsync testfile u...@remote.host:testfile
Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 20, in 
import rdiff_backup.Main
ModuleNotFoundError: No module named 'rdiff_backup'
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) 
[sender=3.1.3]



I don't understand why it's trying to load rdiff-backup, the traceback 
is a giveaway that it's running Python but I explicitly called out the 
path to rsync and I know it's not a python script:


root:~# less /usr/bin/rsync
"/usr/bin/rsync" may be a binary file.  See it anyway?
^?ELF^A^A^A^@^@^

This is all happening on Buster but I have three other Buster systems 
that ran with rdiff-backup and I can run rsync fine on them without 
rdiff-backup running.  What's the deal here?

Re: rsync copyed systemd-nspawn container won't start

[Alexandre Rossi]

Hi,

[...]
> Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Failed to create
> /init.scope control group: Operation not permitted
[...]
> Then I try it with
> 
> debootstrap --variant=minbase --include
> systemd,vim,libterm-readline-gnu-perl,iproute2,dialog,dbus stretch
> /var/lib/machines/foo
> 
> it work as expected.
> 
> Any ideas?

This may be related to systemd<236 in the container.

I would try one of the workarounds mentionned in 
https://github.com/systemd/systemd/issues/9563 :
- passing systemd.legacy_systemd_cgroup_controller=yes to systemd via
  Parameters=
- try with systemd-nspawn ... --private-users=0 --private-users-chown

Alex

rsync copyed systemd-nspawn container won't start

[basti]

Hello,

i have copy a running systemd-nspawn container via rsync.
The container won't start in the destination:

Jan 11 15:22:55 dhanna systemd[1]: Started Container foo.
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: systemd 232 running in
system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP
+LIBCRYPTSETUP +GC
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Detected virtualization
systemd-nspawn.
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Detected architecture x86-64.
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: [1B blob data]
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Welcome to Debian
GNU/Linux 9 (stretch)!
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: [1B blob data]
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Set hostname to .
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Failed to install release
agent, ignoring: No such file or directory
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Failed to create
/init.scope control group: Operation not permitted
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Failed to allocate manager
object: Operation not permitted
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: [!!] Failed to
allocate manager object, freezing.
Jan 11 15:22:55 dhanna systemd-nspawn[21268]: Freezing execution.


Then I try it with

debootstrap --variant=minbase --include
systemd,vim,libterm-readline-gnu-perl,iproute2,dialog,dbus stretch
/var/lib/machines/foo

it work as expected.

Any ideas?

Best regards

Re: rsync failing to back up to synology server

[mick crane]

On 2021-11-13 10:42, Dan Ritter wrote:

Sharon Kimble wrote:


After enduring a home move, I'm now trying to restart my rsync backups
to my synology server, but none are working!

I have this command for backing up my emacs -

- --8<---cut here---start->8---
/usr/bin/rsync -avhz --update --delete --partial 
--password-file="$PASS" ~/.emacs.d/ 
boudiccas@192.168.1.106::home/back-emacs/

- --8<---cut here---end--->8---

but every time it shows this error message -

- --8<---cut here---start->8---
@ERROR: auth failed on module home
rsync error: error starting client-server protocol (code 5) at 
main.c(1817) [sender=3.2.3]

- --8<---cut here---end--->8---

It seems that its not getting the correct password to enable the 
backup
to happen, but what password? Is it my password on my source machine, 
or
the log-in password for my synology server, or what? I've tried 
various

passwords, but I still can't get it to backup, so help please?


rsync either uses ssh or its own protocol to connect across
networks. Almost everyone uses ssh.

Using double colons
boudiccas@192.168.1.106::home/back-emacs
indicates that you want the rsync protocol, which required rsync
to be running in daemon mode on the other side.

Use one colon there, which indicates SSH, and use boudicca's
login password on the 192.168.1.106 box.

If
ssh boudiccas@192.168.1.106
doesn't work, debug that first.


-dsr-


when sorting mine out I found it handy to enable telnet while setting up 
ssh so I could go back and fix it when I invariably broke it.


mick
--
Key ID4BFEBB31

Re: rsync failing to back up to synology server

[Dan Ritter]

Sharon Kimble wrote: 
> 
> After enduring a home move, I'm now trying to restart my rsync backups
> to my synology server, but none are working!
> 
> I have this command for backing up my emacs -
> 
> - --8<---cut here---start----->8---
> /usr/bin/rsync -avhz --update --delete --partial --password-file="$PASS" 
> ~/.emacs.d/ boudiccas@192.168.1.106::home/back-emacs/
> - --8<---cut here---end--->8---
> 
> but every time it shows this error message -
> 
> - --8<---cut here---start->8---
> @ERROR: auth failed on module home
> rsync error: error starting client-server protocol (code 5) at 
> main.c(1817) [sender=3.2.3]
> - --8<---cut here---end--->8---
> 
> It seems that its not getting the correct password to enable the backup
> to happen, but what password? Is it my password on my source machine, or
> the log-in password for my synology server, or what? I've tried various
> passwords, but I still can't get it to backup, so help please?

rsync either uses ssh or its own protocol to connect across
networks. Almost everyone uses ssh.

Using double colons 
boudiccas@192.168.1.106::home/back-emacs
indicates that you want the rsync protocol, which required rsync
to be running in daemon mode on the other side.

Use one colon there, which indicates SSH, and use boudicca's
login password on the 192.168.1.106 box. 

If 
ssh boudiccas@192.168.1.106
doesn't work, debug that first.


-dsr-

rsync failing to back up to synology server

[Sharon Kimble]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512


After enduring a home move, I'm now trying to restart my rsync backups
to my synology server, but none are working!

I have this command for backing up my emacs -

- --8<---cut here---start->8---
/usr/bin/rsync -avhz --update --delete --partial --password-file="$PASS" 
~/.emacs.d/ boudiccas@192.168.1.106::home/back-emacs/
- --8<---cut here---end--->8---

but every time it shows this error message -

- --8<---cut here---start->8---
@ERROR: auth failed on module home
rsync error: error starting client-server protocol (code 5) at main.c(1817) 
[sender=3.2.3]
- --8<---cut here---end--->8---

It seems that its not getting the correct password to enable the backup
to happen, but what password? Is it my password on my source machine, or
the log-in password for my synology server, or what? I've tried various
passwords, but I still can't get it to backup, so help please?

Thanks
Sharon.
- -- 
Debian 11, fluxbox 1.3.7, emacs 28.0.50, org 9.5
-BEGIN PGP SIGNATURE-

iQJPBAEBCgA5FiEELSc/6QwVBIYugJDbNoGAGQr4g1sFAmGPfeYbHGJvdWRpY2Nh
c0Bza2ltYmxlLnBsdXMuY29tAAoJEDaBgBkK+INbg1MP/jVg/9GmwS2i4Ev6R4ss
hbwPDFkQM8g16se4Qih82OzhBN6NhAZzSegx7M0+NIo31QoF0FMYt0zleMJHlT5+
wMKQFcVBSa9u6HBl48I7aMTNv7QJ0YgOT/pis2nu5EBduU8OppWwZsZGSRn7//mP
plVjV34X1xNjDCA2minuQE+/EsC6qpiVDss5q0+6ZjZvhE4PvIpC6aCZWos3GyB8
thXD2Qn8BgpcfrjXE2sh9fw8NQeJUcV3kHJo17Re4EKlWgd7N/Y/ybPPoTMMxZ9y
Mrs4VLP0Q3uSi8VWT23VvBpxVNkS0zuDNNi2Vi1tToHLW2S5Idx8iBSLgO7aoMJC
ii+1tmDhXj2khqXHvjGeNtbA/z1gTja3jRIc/inNxJr4OZBtemZz5/lW3tmiSfkI
ivpaOnSUijuDpVfj/qy6siav1EceSCs+waqpSH4dwqG4xtzjy8h+tJPEAOdz/+Nx
nAuME+OeV2ut7Zc2RGviYsf2x7ioPPrq3ra5Z1bFHk7++psfIOCOw5Z9Cn1Ep7U3
a33PKGFe9R7vC7JkVfrOL9j6OEW1mmE+Mmi+X8y3uUzwbeMZJF7TRMZFExuKEZrb
LaYTaV+HMLYAuoTRMxnlQH1xe/XQx8hEv/e7YGrt/wPUKzBh86FVUYEBLhceYNeQ
JSCifcWlieVJh3TEPDmIlGKP
=aG/R
-END PGP SIGNATURE-

Re: rsync to NAS for backup

[Mirko Parthey]

On Mon, Feb 15, 2021 at 12:39:29PM +, mick crane wrote:
> Appears that to retain permissions need root at both ends of rsync.

Not necessarily. If the server filesystem supports xattrs, you can use
the --fake-super option with the rsync server, running as a non-root user
that can write to the storage.

To set the option on the client side for rsync over ssh, use:

--rsync-path "rsync --fake-super"

Regards,
Mirko

Re: rsync to NAS for backup

[tomas]

On Thu, Feb 18, 2021 at 05:06:33PM -0700, Charles Curley wrote:
> On Thu, 18 Feb 2021 18:24:43 +0100
>  wrote:
> 
> > Care to name some of those limitations?
> > 
> > (of course, rsync /is not/ a backup program in itself, but lets you
> > build one with 10-30 lines around it).
> 
> Why build one (which I have done) when you can get a perfectly good one
> from the Debian repos? E.g.: rsnapshot. Others have been mentioned on
> this thread.

My approach is: if reading the doc takes me significantly longer than
building, I do the latter.

In this case, this is... uh... the case :-)

YMMV, TANSTAAFL and all that.

For me, this is the end of the thread (I feel we've been everywhere,
everyone's happy with her solution and that).

Cheers

> -- 
> Does anybody read signatures any more?

I do, yes.

 - t



signature.asc
Description: Digital signature

Re: rsync to NAS for backup

[Charles Curley]

On Thu, 18 Feb 2021 18:24:43 +0100
 wrote:

> Care to name some of those limitations?
> 
> (of course, rsync /is not/ a backup program in itself, but lets you
> build one with 10-30 lines around it).

Why build one (which I have done) when you can get a perfectly good one
from the Debian repos? E.g.: rsnapshot. Others have been mentioned on
this thread.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/


pgpnL4mGr3L7u.pgp
Description: OpenPGP digital signature

Re: rsync to NAS for backup

[Gary Dale]

On 2021-02-18 12:22, to...@tuxteam.de wrote:

On Thu, Feb 18, 2021 at 06:59:03PM +0200, Teemu Likonen wrote:

* 2021-02-18 11:13:25-0500, Gary Dale wrote:


rsync is a quick & dirty backup tactic but it's got limitations.

1) files may stay around forever in the backup even if you've deleted
them from your main computer because you don't need them.

2) you only have one copy of a file and that only lasts until the next
rsync. This limits your ability to restore from a backup before it is
overwritten.
rsync is not a good substitute for backups.

No, it's not. It is a fantastic tool for backups :-)


Rsync is great backup program with "--link-dest" option. Here is the
idea in simplified code:

[...]

Absolutely. Time travel!

Actually, I've implemented this at a customer's place. They were
delighted.

Where rsync shows some weaknesses is on big, fat files (think
videos, one or several GB).

Really huge directories (tens to hundreds of TB) were once a
challenge, too, but I hear that they refined the scanning
part in the meantime. No direct experience, though.

And, oh, Gary: if you want to delete files which disappeared
in the source, check out the --delete option.

But this time-staggered backup with --link-dest is really great.

Cheers


While you can twist any tool to fit a task, real backup programs don't 
need to be twisted and do a better job. For example backup retention 
policy is intuitive and easy to set. Some backup programs even factor 
out common blocks for de-duplication, which can save a lot of space. 
Hard-links only do that if the file name is the same.


And when you need to restore a file, backup programs usually let you see 
when the files changed then let you choose which version to restore.


As for the delete option, it makes the rsync script even more 
complicated. A backup program will simply expire the file at the end of 
the retention period.

Re: rsync to NAS for backup

[Gary Dale]

On 2021-02-18 10:57, mick crane wrote:

On 2021-02-15 12:39, mick crane wrote:

On 2021-02-13 19:20, David Christensen wrote:

On 2021-02-13 01:27, mick crane wrote:

I made a mistake and instead of getting a PC for backup I got a NAS.
I'm struggling to get to grips with it.
If rsync from PC to NAS NAS changes the owner/group of files to 
me/users which is probably no good for backing up.

There's that problem then another that it won't let me login as root.
I asked on Synology forum but not getting a lot of joy.
https://community.synology.com/enu/forum/1/post/141137
Anybody used these things can advise ?


What is the model of the Synology NAS?  What options -- CPU, memory,
disks, bays, interfaces, PSU, whatever?  Support page URL?


Reading the forum post, it sounds like you damaged the sudoers file.
The fix would appear to be doing a Mode 2 reset per Synology's
instructions:

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS 




Once the NAS has been reset, figure out how to meet your needs within
the framework provided by Synology.  Follow the User Guide. Follow
the Admin Guide.  Do not mess around "under the hood" with a terminal
and sudo.  Make Synology earn your money.


But if you want complete control, buy or build an x86_64/amd64 server,
install Debian, and have at it.




thanks for advices folks.
It was indeed user error with being in a rush and blurred eyesight
mistook "%" for "#"
We are making progress.
Appears that to retain permissions need root at both ends of rsync.
Have keys working with ssh for users to NAS ( not helped by default
permissions for .ssh files being wrong) and can su to root so now need
to get ssh working with keys with no passphrase for root and all
should be good.


further to this if it helps anybody can start sshd with -d switch and 
at same time do client with -vv switch then can see where is falling 
down. Having telnet available helps if break sshd_config can still 
telnet and mend it.

mick


rsync is a quick & dirty backup tactic but it's got limitations.

1) files may stay around forever in the backup even if you've deleted 
them from your main computer because you don't need them.


2) you only have one copy of a file and that only lasts until the next 
rsync. This limits your ability to restore from a backup before it is 
overwritten.



Using a real backup program, which can run on you main computer to 
backup to the NAS, lets you define a retention policy so files no longer 
needed can be purged while you have multiple backups of files you are 
currently working on.


rsync is not a good substitute for backups.

Re: rsync to NAS for backup

[tomas]

On Thu, Feb 18, 2021 at 07:21:01PM +0200, Andrei POPESCU wrote:

[...]

> In my opinion the point still stands, rsync (by itself) has significant 
> limitations as a backup program, which is probably also the reason why 
> several backup programs using rsync exist.

Care to name some of those limitations?

(of course, rsync /is not/ a backup program in itself, but lets you
build one with 10-30 lines around it).

Cheers
-- t


signature.asc
Description: Digital signature

Re: rsync to NAS for backup

[tomas]

On Thu, Feb 18, 2021 at 06:59:03PM +0200, Teemu Likonen wrote:
> * 2021-02-18 11:13:25-0500, Gary Dale wrote:
> 
> > rsync is a quick & dirty backup tactic but it's got limitations.
> >
> > 1) files may stay around forever in the backup even if you've deleted 
> > them from your main computer because you don't need them.
> >
> > 2) you only have one copy of a file and that only lasts until the next 
> > rsync. This limits your ability to restore from a backup before it is 
> > overwritten.
> 
> > rsync is not a good substitute for backups.

No, it's not. It is a fantastic tool for backups :-)

> Rsync is great backup program with "--link-dest" option. Here is the
> idea in simplified code:

[...]

Absolutely. Time travel!

Actually, I've implemented this at a customer's place. They were
delighted.

Where rsync shows some weaknesses is on big, fat files (think
videos, one or several GB).

Really huge directories (tens to hundreds of TB) were once a
challenge, too, but I hear that they refined the scanning
part in the meantime. No direct experience, though.

And, oh, Gary: if you want to delete files which disappeared
in the source, check out the --delete option.

But this time-staggered backup with --link-dest is really great.

Cheers
-- t


signature.asc
Description: Digital signature

Re: rsync to NAS for backup

[Andrei POPESCU]

On Jo, 18 feb 21, 18:59:03, Teemu Likonen wrote:
> * 2021-02-18 11:13:25-0500, Gary Dale wrote:
> 
> > rsync is a quick & dirty backup tactic but it's got limitations.
> >
> > 1) files may stay around forever in the backup even if you've deleted 
> > them from your main computer because you don't need them.
> >
> > 2) you only have one copy of a file and that only lasts until the next 
> > rsync. This limits your ability to restore from a backup before it is 
> > overwritten.
> 
> > rsync is not a good substitute for backups.
> 
> Rsync is great backup program with "--link-dest" option. Here is the
> idea in simplified code:
 
[...]
 
> With that sort of code every backup is a new complete directory tree in
> a time stamped directory. If files have not changed since the latest
> backup "--link-dest" creates hard links. Old backups can be deleted by
> removing old (or any) directory tree. The trees don't depend on each
> other.
> 
> What else do we need? OK, some people may need compression but usually
> hard disk space is cheap.

In my opinion the point still stands, rsync (by itself) has significant 
limitations as a backup program, which is probably also the reason why 
several backup programs using rsync exist.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: rsync to NAS for backup

[Teemu Likonen]

* 2021-02-18 11:13:25-0500, Gary Dale wrote:

> rsync is a quick & dirty backup tactic but it's got limitations.
>
> 1) files may stay around forever in the backup even if you've deleted 
> them from your main computer because you don't need them.
>
> 2) you only have one copy of a file and that only lasts until the next 
> rsync. This limits your ability to restore from a backup before it is 
> overwritten.

> rsync is not a good substitute for backups.

Rsync is great backup program with "--link-dest" option. Here is the
idea in simplified code:

cd "$destination"
new=$(date +%Y%m%dT%H%M%S%z)
if mkdir "$new" && \
rsync -aHAX --link-dest=latest_backup [...] \
"$source" "$new"
then
# Recreate symlink to point to the new backup directory.
ln -sfn "$new" latest_backup
# Delete old backups.
find . -mindepth 1 -maxdepth 1 -type d -mtime +365 \
-exec /bin/rm -fr -- {} +
fi

With that sort of code every backup is a new complete directory tree in
a time stamped directory. If files have not changed since the latest
backup "--link-dest" creates hard links. Old backups can be deleted by
removing old (or any) directory tree. The trees don't depend on each
other.

What else do we need? OK, some people may need compression but usually
hard disk space is cheap.

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450


signature.asc
Description: PGP signature

Re: rsync to NAS for backup

[mick crane]

On 2021-02-18 16:13, Gary Dale wrote:

On 2021-02-18 10:57, mick crane wrote:

On 2021-02-15 12:39, mick crane wrote:

On 2021-02-13 19:20, David Christensen wrote:

On 2021-02-13 01:27, mick crane wrote:
I made a mistake and instead of getting a PC for backup I got a 
NAS.

I'm struggling to get to grips with it.
If rsync from PC to NAS NAS changes the owner/group of files to 
me/users which is probably no good for backing up.
There's that problem then another that it won't let me login as 
root.

I asked on Synology forum but not getting a lot of joy.
https://community.synology.com/enu/forum/1/post/141137
Anybody used these things can advise ?


What is the model of the Synology NAS?  What options -- CPU, memory,
disks, bays, interfaces, PSU, whatever?  Support page URL?


Reading the forum post, it sounds like you damaged the sudoers file.
The fix would appear to be doing a Mode 2 reset per Synology's
instructions:

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS 
Once the NAS has been reset, figure out how to meet your needs 
within

the framework provided by Synology.  Follow the User Guide. Follow
the Admin Guide.  Do not mess around "under the hood" with a 
terminal

and sudo.  Make Synology earn your money.


But if you want complete control, buy or build an x86_64/amd64 
server,

install Debian, and have at it.




thanks for advices folks.
It was indeed user error with being in a rush and blurred eyesight
mistook "%" for "#"
We are making progress.
Appears that to retain permissions need root at both ends of rsync.
Have keys working with ssh for users to NAS ( not helped by default
permissions for .ssh files being wrong) and can su to root so now 
need

to get ssh working with keys with no passphrase for root and all
should be good.


further to this if it helps anybody can start sshd with -d switch and 
at same time do client with -vv switch then can see where is falling 
down. Having telnet available helps if break sshd_config can still 
telnet and mend it.

mick


rsync is a quick & dirty backup tactic but it's got limitations.

1) files may stay around forever in the backup even if you've deleted
them from your main computer because you don't need them.

2) you only have one copy of a file and that only lasts until the next
rsync. This limits your ability to restore from a backup before it is
overwritten.


Using a real backup program, which can run on you main computer to
backup to the NAS, lets you define a retention policy so files no
longer needed can be purged while you have multiple backups of files
you are currently working on.

rsync is not a good substitute for backups.


OK, I didn't do this before.
Am I right in thinking that BackupNinja keeps a local directory with 
list of files that need to be backed up and then rsyncs those files to 
remote directory ?

mick

--
Key ID4BFEBB31

Re: rsync to NAS for backup

[mick crane]

On 2021-02-15 12:39, mick crane wrote:

On 2021-02-13 19:20, David Christensen wrote:

On 2021-02-13 01:27, mick crane wrote:

I made a mistake and instead of getting a PC for backup I got a NAS.
I'm struggling to get to grips with it.
If rsync from PC to NAS NAS changes the owner/group of files to 
me/users which is probably no good for backing up.

There's that problem then another that it won't let me login as root.
I asked on Synology forum but not getting a lot of joy.
https://community.synology.com/enu/forum/1/post/141137
Anybody used these things can advise ?


What is the model of the Synology NAS?  What options -- CPU, memory,
disks, bays, interfaces, PSU, whatever?  Support page URL?


Reading the forum post, it sounds like you damaged the sudoers file.
The fix would appear to be doing a Mode 2 reset per Synology's
instructions:

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS


Once the NAS has been reset, figure out how to meet your needs within
the framework provided by Synology.  Follow the User Guide.  Follow
the Admin Guide.  Do not mess around "under the hood" with a terminal
and sudo.  Make Synology earn your money.


But if you want complete control, buy or build an x86_64/amd64 server,
install Debian, and have at it.




thanks for advices folks.
It was indeed user error with being in a rush and blurred eyesight
mistook "%" for "#"
We are making progress.
Appears that to retain permissions need root at both ends of rsync.
Have keys working with ssh for users to NAS ( not helped by default
permissions for .ssh files being wrong) and can su to root so now need
to get ssh working with keys with no passphrase for root and all
should be good.


further to this if it helps anybody can start sshd with -d switch and at 
same time do client with -vv switch then can see where is falling down. 
Having telnet available helps if break sshd_config can still telnet and 
mend it.

mick

--
Key ID4BFEBB31

Re: rsync to NAS for backup

[mick crane]

On 2021-02-13 19:20, David Christensen wrote:

On 2021-02-13 01:27, mick crane wrote:

I made a mistake and instead of getting a PC for backup I got a NAS.
I'm struggling to get to grips with it.
If rsync from PC to NAS NAS changes the owner/group of files to 
me/users which is probably no good for backing up.

There's that problem then another that it won't let me login as root.
I asked on Synology forum but not getting a lot of joy.
https://community.synology.com/enu/forum/1/post/141137
Anybody used these things can advise ?


What is the model of the Synology NAS?  What options -- CPU, memory,
disks, bays, interfaces, PSU, whatever?  Support page URL?


Reading the forum post, it sounds like you damaged the sudoers file.
The fix would appear to be doing a Mode 2 reset per Synology's
instructions:

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS


Once the NAS has been reset, figure out how to meet your needs within
the framework provided by Synology.  Follow the User Guide.  Follow
the Admin Guide.  Do not mess around "under the hood" with a terminal
and sudo.  Make Synology earn your money.


But if you want complete control, buy or build an x86_64/amd64 server,
install Debian, and have at it.




thanks for advices folks.
It was indeed user error with being in a rush and blurred eyesight 
mistook "%" for "#"

We are making progress.
Appears that to retain permissions need root at both ends of rsync.
Have keys working with ssh for users to NAS ( not helped by default 
permissions for .ssh files being wrong) and can su to root so now need 
to get ssh working with keys with no passphrase for root and all should 
be good.


mick


--
Key ID4BFEBB31

Re: rsync to NAS for backup

[David Christensen]

On 2021-02-13 01:27, mick crane wrote:

I made a mistake and instead of getting a PC for backup I got a NAS.
I'm struggling to get to grips with it.
If rsync from PC to NAS NAS changes the owner/group of files to me/users 
which is probably no good for backing up.

There's that problem then another that it won't let me login as root.
I asked on Synology forum but not getting a lot of joy.
https://community.synology.com/enu/forum/1/post/141137
Anybody used these things can advise ?


What is the model of the Synology NAS?  What options -- CPU, memory, 
disks, bays, interfaces, PSU, whatever?  Support page URL?



Reading the forum post, it sounds like you damaged the sudoers file. 
The fix would appear to be doing a Mode 2 reset per Synology's instructions:


https://www.synology.com/en-global/knowledgebase/DSM/tutorial/General_Setup/How_to_reset_my_Synology_NAS


Once the NAS has been reset, figure out how to meet your needs within 
the framework provided by Synology.  Follow the User Guide.  Follow the 
Admin Guide.  Do not mess around "under the hood" with a terminal and 
sudo.  Make Synology earn your money.



But if you want complete control, buy or build an x86_64/amd64 server, 
install Debian, and have at it.



David

Re: rsync to NAS for backup

[Charles Curley]

On Sat, 13 Feb 2021 09:27:54 +
mick crane  wrote:

> I made a mistake and instead of getting a PC for backup I got a NAS.
> I'm struggling to get to grips with it.
> If rsync from PC to NAS NAS changes the owner/group of files to
> me/users which is probably no good for backing up.

Can you set up a network file system such as NFS on the NAS, then tar
your data to the NAS?

Can you install a backup server such as amanda on the NAS?


-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: rsync to NAS for backup

[Toni Mas Soler]

Is there an alternative if you want an incremental backup?

Obviously you could use tar-ed archives with unprivileged permissions. If you 
did, you would get a huge network overhead.

thks


Toni Mas
GPG 3F42A21D84D7E950

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
En dissabte 13 de febrer de 2021 a les 13:50, didier gaumet 
 va escriure:

> Hello,
> 

> Disclaimer: I do not use and am not familiar with Sinology hardware and
> software and generally speaking, I am not knowledgeable in networking
> 

> I would say that:
> 

> -   the owner:group names of a file on the PC you backup and the
> owner:group names of the backup files on the synology files might be
> different, even if you try to maintain ownership and rights. What really
> counts here are owner:group identifiers (UID:GID). Bob_user:Bob_group on
> your PC might equate to Alice_user:John_group on your NAS. Upon
> restoration that would be reversed to Bob_user:Bob_group.
> That would be typical without something like a LDAP server.
> 

> -   SSH root login seems to be discouraged for security reasons. Sinology
> probably adhere to this principle and the appropriate way to do what you
> want would probably be to access a shell on the Synology software to
> issue a sudo or su -c command.
> 

> -   editing /etc/sudoers is generally done via the visudo command
> -   if that is of interest to you, there is a way to install Debian in
> chroot on your NAS
>



signature.asc
Description: OpenPGP digital signature

Re: rsync to NAS for backup

[didier gaumet]

Hello,

Disclaimer: I do not use and am not familiar with Sinology hardware and 
software and generally speaking, I am not knowledgeable in networking


I would say that:

- the owner:group names of a file on the PC you backup and the 
owner:group names of the backup files on the synology files might be 
different, even if you try to maintain ownership and rights. What really 
counts here are owner:group identifiers (UID:GID). Bob_user:Bob_group on 
your PC might equate to Alice_user:John_group on your NAS. Upon 
restoration that would be reversed to Bob_user:Bob_group.

 That would be typical without something like a LDAP server.

- SSH root login seems to be discouraged for security reasons. Sinology 
probably adhere to this principle and the appropriate way to do what you 
want would probably be to access a shell on the Synology software to 
issue a sudo or su -c command.


- editing /etc/sudoers is generally done via the visudo command

- if that is of interest to you, there is a way to install Debian in 
chroot on your NAS

rsync to NAS for backup

[mick crane]

I made a mistake and instead of getting a PC for backup I got a NAS.
I'm struggling to get to grips with it.
If rsync from PC to NAS NAS changes the owner/group of files to me/users 
which is probably no good for backing up.

There's that problem then another that it won't let me login as root.
I asked on Synology forum but not getting a lot of joy.
https://community.synology.com/enu/forum/1/post/141137
Anybody used these things can advise ?

mick

--
Key ID4BFEBB31

Re: Hijacking Threads [was: Re: rsync link corruption with -H and --link-dest]

[Brad Rogers]

On Mon, 23 Nov 2020 11:17:56 -0500
Celejar  wrote:

Hello Celejar,

>I made this same point in another email, but I just want to remind

Indeed you did.  And rightly so;

It's possible that my messages could be construed as saying CM is the
_only_ MUA capable of such configuration.

That was not my intention.  There's definitely Sylpheed.  I have no
first hand knowledge of others, but suspect they exist.

-- 
 Regards  _
 / )   "The blindingly obvious is
/ _)radnever immediately apparent"
It's only the children of the f** wealthy tend to be good looking
Ugly - The Stranglers


pgptTm2ZCiYt1.pgp
Description: OpenPGP digital signature

Re: Hijacking Threads [was: Re: rsync link corruption with -H and --link-dest]

[Jeremy Nicoll]

On Sun, 22 Nov 2020, at 16:55, Gareth Evans wrote:

> > Once one sets up an MUA correctly, one only has to click 'Compose' for
> > all the required fields, apart from Subject, to be filled in.
> 
> When does clicking "compose" have this effect?

In a good (probably standalone) MUA.

In Fastmail's webmail system there's not much of that flexibility, however
in a specific folder's "Advanced Preferences" you can set the identity 
which will be used when you compose a mail when that is the current folder.

So here, if I have my debian-users folder selected, then click on Compose
(which annoyingly needs me to scroll the folders list back to the top), the 
new window will open with "From:" set appropriately.

I have to set the "To:" address manually, but in contacts I defined all my 
mailing list to-addresses as "ML - list name" so typing just ML in the TO
field gives me a picking list of valid values.

-- 
Jeremy Nicoll - my opinions are my own.

Re: Hijacking Threads [was: Re: rsync link corruption with -H and --link-dest]

[Celejar]

On Sun, 22 Nov 2020 19:14:05 +
Brad Rogers  wrote:

> On Sun, 22 Nov 2020 11:39:20 -0700
> Charles Curley  wrote:
> 
> Hello Charles,
> 
> >Thank you. I just learned something useful about Claws-Mail.
> 
> You're welcome.
> 
> For those interested, Claws Mail has it's own mailing list.
> 
> Subscribe here;
> 
> or by emailing
> 

I made this same point in another email, but I just want to remind
people generally that Sylpheed, Claws's "parent," has many of the
features that people like Claws for. It also has its own mailing list ;)

https://sylpheed.sraoss.jp/en/ml.html

Celejar

Re: Hijacking Threads [was: Re: rsync link corruption with -H and --link-dest]

[Celejar]

On Sun, 22 Nov 2020 17:39:06 +
"Gareth Evans"  wrote:

> > On Sun, 22 Nov 2020, at 17:08, Brad Rogers wrote:

...

> > > With a suitable MUA (example; Claws Mail) when you create a folder for
> > > mails from a mailing list, it's possible to set up default addresses for
> > > To, From and Reply-To to be used for that folder.  With Claws, it's even
> > > possible to set up default Cc: and BCc headers.
> > > 
> > > Once all that's done, one simply selects the folder, and clicks
> > > 'Compose'.  All you need to do then is insert a suitable subject header,
> > > and compose your message.
> > > 
> > > As I said above it does, of course, require an MUA that's suitably
> > > capable.

...

> Thanks Brad, I may be a Claws convert!

Just FTR, this is equally possible with Sylpheed (the MUA that Claws
began as a fork of).

Celejar