Re: Does anyone want key-resistant tamper-evident archives?

[Dennis Eriksen]

On Wed, Sep 07, 2016 at 03:10:42PM -0700, Colin Percival wrote:
> Hi all,

Hi!

> Is anyone interested in having this functionality?  It seems like too
> obscure a use case to write code for if nobody wants it yet, but if there's
> a demand then it's definitely doable.

As far as I am concerned, any function that improves security and
privacy, even if the usecase scenario is improbable, is a good thing
(and for me these are perhaps the two main selling points of tarsnap).
So yes, that would be interesting. :-)

-- 
mvh
Dennis Eriksen

Re: Does anyone want key-resistant tamper-evident archives?

[Colin Percival]

On 09/07/16 15:19, Dennis Eriksen wrote:
> On Wed, Sep 07, 2016 at 03:10:42PM -0700, Colin Percival wrote:
>> Is anyone interested in having this functionality?  It seems like too
>> obscure a use case to write code for if nobody wants it yet, but if there's
>> a demand then it's definitely doable.
> 
> As far as I am concerned, any function that improves security and
> privacy,

This wouldn't have any (direct) effect on privacy; just on allowing you
to detect if a (validly signed) archive is different from the one which
you had stored.  So, integrity, not confidentiality.

> even if the usecase scenario is improbable, is a good thing
> (and for me these are perhaps the two main selling points of tarsnap).
> So yes, that would be interesting. :-)

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

Re: Does anyone want key-resistant tamper-evident archives?

[Sudhi Herle]

I would certainly welcome this feature.

In a different domain - I have a similar scenario. There, I solved 
it by tying the hash of public key to the content that is 
signed+encrypted. This way, whenever a blob is decrypted (or 
verified), the pubkey-hash in the blob is compared to the hash of 
the public key used for decryption.

--
Sudhi



Colin Percival wrote:

Hi all,

Tarsnap is designed to detect if your data is modified: Archives are
cryptographically signed, and the signatures are verified before any
data is extracted.  However, this depends on the integrity of the key:
If someone has your delete and write keys, they could delete an archive
and create a new one with the same name, and (since they have the keys)
it would cryptographically validate.

It occurs to me that we could have a stronger unforgeability property
via out-of-band (non-cryptographic) verification of the archive metadata
hash; even with the keys, it would be impossible to create a different
archive which has the same hash (unless you find a SHA256 collision).  In
addition to the "stolen keys" scenario, this could be useful if you need
to prove (e.g., for auditing or legal purposes) that *you* haven't changed
an archive since the time when you created it.

Is anyone interested in having this functionality?  It seems like too
obscure a use case to write code for if nobody wants it yet, but if there's
a demand then it's definitely doable.

Re: Does anyone want key-resistant tamper-evident archives?

[Raphael Bauduin]

On Thu, Sep 8, 2016 at 12:10 AM, Colin Percival 
wrote:

> Hi all,
>
> Tarsnap is designed to detect if your data is modified: Archives are
> cryptographically signed, and the signatures are verified before any
> data is extracted.  However, this depends on the integrity of the key:
> If someone has your delete and write keys, they could delete an archive
> and create a new one with the same name, and (since they have the keys)
> it would cryptographically validate.
>
> It occurs to me that we could have a stronger unforgeability property
> via out-of-band (non-cryptographic) verification of the archive metadata
> hash; even with the keys, it would be impossible to create a different
> archive which has the same hash (unless you find a SHA256 collision).  In
> addition to the "stolen keys" scenario, this could be useful if you need
> to prove (e.g., for auditing or legal purposes) that *you* haven't changed
> an archive since the time when you created it.
>
> Is anyone interested in having this functionality?  It seems like too
> obscure a use case to write code for if nobody wants it yet, but if there's
> a demand then it's definitely doable.
>
>
It is interesting, but I would prefer improvements to restore speed.
I currently need to keep a second backup copy just in case I need a fast
restore,
as the restore with tarsnap is unacceptably slow (last time I tested a
couple of months ago).

Cheers

Raphaël


> --
> Colin Percival
> Security Officer Emeritus, FreeBSD | The power to serve
> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
>



-- 
[image: Tessares SA]  Raphael Bauduin | Senior R&D
Engineer
raphael.baud...@tessares.net | +32 10 392 252

Tessares SA
www.tessares.net
6 Rue Louis de Geer, 1348 Louvain-la-Neuve, Belgium


-- 

--
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.

Re: Does anyone want key-resistant tamper-evident archives?

[Daniel Neades]

Hi,

> On 8 Sep 2016, at 06:53, Raphael Bauduin  wrote:
> It is interesting, but I would prefer improvements to restore speed. 
> I currently need to keep a second backup copy just in case I need a fast 
> restore, 
> as the restore with tarsnap is unacceptably slow (last time I tested a couple 
> of months ago).

+1

We also keep backups elsewhere, partially to spread risk, but also because 
Tarsnap restores are so slow.

The slow list and restore times also make testing Tarsnap backups *extremely* 
painful. For example, we backup dumps (4–5 GB in size) of one of our databases. 
We can restore a dump via SSH from a remote backup machine located on a 
different continent in a matter of minutes. Restoring the identical dump from 
Tarsnap takes hours.

I am not sure we’d have chosen Tarsnap had we realized how slow these essential 
and common operations would be. That said, it is good to have a secondary 
backup mechanism that is completely separate from our primary one, and we do 
very much like Tarsnap’s excellent de-duplication.

While I am requesting things, it would also be nice to have a convenient way to 
delete old archives built into the tarsnap command. At the moment, everyone 
seems to have to script their own wrappers for common operations like this. The 
expiration problem is compound by the fact that both listing and deleting 
archives is so painfully slow.

-- 
Daniel Neades
Director, Araxis Ltd
www.araxis.com

Re: Does anyone want key-resistant tamper-evident archives?

[Thomas Tempelmann]

As a developer (and currently not a user) I wonder why restoring would be
so slow.

Is that just a question of manpower or is there an algorithm problem that's
hard to solve? Or is it just the fact that the bandwidth to the server is
limited?

Thomas

Re: Does anyone want key-resistant tamper-evident archives?

[Arnt Gulbrandsen]

Thomas Tempelmann writes:
Is that just a question of manpower or is there an algorithm 
problem that's hard to solve? Or is it just the fact that the 
bandwidth to the server is limited?


Restoring is much too slow to be explained by bandwidth. My worst restore 
took several days and used less than 1% of available bandwidth.


Single-file restores are fast (at least fast enough not to irritate me). I 
think the problem is that that requests aren't pipelined. The client asks 
for a file, the server retrieves the file from the storage backend, the 
file is transferred, the client writes it to the file system, the client 
thinks about what file to ask for next, GOTO 1. If you're restoring half a 
million small files the network connection will be idle a lot of the time.


Life is much better if you restore a single big file (and even better if 
you don't have to restore).


Arnt

Re: Does anyone want key-resistant tamper-evident archives?

[Arthur Chance]

On 08/09/2016 11:12, Arnt Gulbrandsen wrote:
> Thomas Tempelmann writes:
>> Is that just a question of manpower or is there an algorithm problem
>> that's hard to solve? Or is it just the fact that the bandwidth to the
>> server is limited?
> 
> Restoring is much too slow to be explained by bandwidth. My worst
> restore took several days and used less than 1% of available bandwidth.
> 
> Single-file restores are fast (at least fast enough not to irritate me).
> I think the problem is that that requests aren't pipelined. The client
> asks for a file, the server retrieves the file from the storage backend,
> the file is transferred, the client writes it to the file system, the
> client thinks about what file to ask for next, GOTO 1. If you're
> restoring half a million small files the network connection will be idle
> a lot of the time.
> 
> Life is much better if you restore a single big file (and even better if
> you don't have to restore).

A question for Colin: would it be possible for tarsnap to have a faster
restore mode where it simply created an unencrypted tar file on the
client equivalent to a given tarsnap archive, to be restored from as
necessary, or does the architecture preclude it? That would at least
speed up full restores after a crash that lost everything.

-- 
Schrödinger's cat had 18 half lives.

Re: Does anyone want key-resistant tamper-evident archives?

[Daniel Neades]

> On 8 Sep 2016, at 11:12, Arnt Gulbrandsen  wrote:
> Life is much better if you restore a single big file (and even better if you 
> don't have to restore).

Single, big files are very slow for us, and we are not at all limited by 
bandwidth. We do have a lot of archives, however, with extensive 
de-duplication, so that may explain the speed.

-- 
Daniel

Re: Does anyone want key-resistant tamper-evident archives?

[Scott Wheeler]

> On Sep 8, 2016, at 11:43 AM, Daniel Neades  wrote:
> 
> The slow list and restore times also make testing Tarsnap backups *extremely* 
> painful. For example, we backup dumps (4–5 GB in size) of one of our 
> databases. We can restore a dump via SSH from a remote backup machine located 
> on a different continent in a matter of minutes. Restoring the identical dump 
> from Tarsnap takes hours.
> 
> I am not sure we’d have chosen Tarsnap had we realized how slow these 
> essential and common operations would be.

I realize this probably won't help if you're restoring single file database 
dumps, but for doing complete (rather than hand-picking single files) restores 
with a lot of files (about 70k in our case) using multiple tarsnap processes 
can speed things up dramatically.  I wrote a little Ruby tool to do this for us 
years ago:

https://github.com/directededge/redsnapper

Again though, if that can be done with a tiny Ruby wrapper, it should be done 
in the default client.  It's the only thing that makes doing complete restores 
for a catastrophic case of complete data loss almost tenable for us with 
Tarsnap.

-Scott

--
Scott Wheeler | Co-founder | Directed Edge | www.directededge.com | 
@directededge

Re: Does anyone want key-resistant tamper-evident archives?

[Daniel Neades]

Hi Scott,

> On 8 Sep 2016, at 14:40, Scott Wheeler  wrote:
> 
>> On Sep 8, 2016, at 11:43 AM, Daniel Neades  wrote:
>> 
>> The slow list and restore times also make testing Tarsnap backups 
>> *extremely* painful. For example, we backup dumps (4–5 GB in size) of one of 
>> our databases. We can restore a dump via SSH from a remote backup machine 
>> located on a different continent in a matter of minutes. Restoring the 
>> identical dump from Tarsnap takes hours.
>> 
>> I am not sure we’d have chosen Tarsnap had we realized how slow these 
>> essential and common operations would be.
> 
> I realize this probably won't help if you're restoring single file database 
> dumps, but for doing complete (rather than hand-picking single files) 
> restores with a lot of files (about 70k in our case) using multiple tarsnap 
> processes can speed things up dramatically.  I wrote a little Ruby tool to do 
> this for us years ago:
> 
> https://github.com/directededge/redsnapper
> 
> Again though, if that can be done with a tiny Ruby wrapper, it should be done 
> in the default client.  It's the only thing that makes doing complete 
> restores for a catastrophic case of complete data loss almost tenable for us 
> with Tarsnap.

That is helpful for people with lots of files (though not, as you surmized, for 
us); thank you for mentioning that here. It is a shame that people are having 
to do these sorts of work-arounds, though – being able to restore reasonably 
quickly from a backup ought to be a core capability of any backup solution.

-- 
Daniel Neades
Director, Araxis Ltd
www.araxis.com

Re: Does anyone want key-resistant tamper-evident archives?

[James Cass]

+1
Great idea!  I would definitely be in favor of such a feature.  Anything to
detect unauthorized access/tampering with my backups would be great.  It
would also be nice to be able to run a simple low bandwidth test to have
confidence that it works.

...James


On Thu, Sep 8, 2016 at 10:09 AM, Daniel Neades  wrote:

> Hi Scott,
>
> > On 8 Sep 2016, at 14:40, Scott Wheeler  wrote:
> >
> >> On Sep 8, 2016, at 11:43 AM, Daniel Neades  wrote:
> >>
> >> The slow list and restore times also make testing Tarsnap backups
> *extremely* painful. For example, we backup dumps (4–5 GB in size) of one
> of our databases. We can restore a dump via SSH from a remote backup
> machine located on a different continent in a matter of minutes. Restoring
> the identical dump from Tarsnap takes hours.
> >>
> >> I am not sure we’d have chosen Tarsnap had we realized how slow these
> essential and common operations would be.
> >
> > I realize this probably won't help if you're restoring single file
> database dumps, but for doing complete (rather than hand-picking single
> files) restores with a lot of files (about 70k in our case) using multiple
> tarsnap processes can speed things up dramatically.  I wrote a little Ruby
> tool to do this for us years ago:
> >
> > https://github.com/directededge/redsnapper
> >
> > Again though, if that can be done with a tiny Ruby wrapper, it should be
> done in the default client.  It's the only thing that makes doing complete
> restores for a catastrophic case of complete data loss almost tenable for
> us with Tarsnap.
>
> That is helpful for people with lots of files (though not, as you
> surmized, for us); thank you for mentioning that here. It is a shame that
> people are having to do these sorts of work-arounds, though – being able to
> restore reasonably quickly from a backup ought to be a core capability of
> any backup solution.
>
> --
> Daniel Neades
> Director, Araxis Ltd
> www.araxis.com
>

Re: Does anyone want key-resistant tamper-evident archives?

[Colin Percival]

On 09/08/16 03:27, Arthur Chance wrote:
> A question for Colin: would it be possible for tarsnap to have a faster
> restore mode where it simply created an unencrypted tar file on the
> client equivalent to a given tarsnap archive, to be restored from as
> necessary, or does the architecture preclude it? That would at least
> speed up full restores after a crash that lost everything.

There are several things which need to be addressed in order to get
better extract performance.  Speeding up --list-archives is the simplest
(although still far from simple); next is 'tarsnap -r' (which is what you're
describing); next is a partial fix for 'tarsnap -x' which speeds up large
file extraction but has little effect on small files; and the hardest is
a full fix for 'tarsnap -x'.  The difference between these is basically just
how hard it is to figure out which blocks we need.

So yes, the plan is to do what you describe.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

Re: Does anyone want key-resistant tamper-evident archives?

[Stephen]

I would be interested in this mainly for the "prove that files have
not been modified" aspect - I'm only not interested in the other
components because my delete + read keys live offline. Otherwise, I
could also see a use-case for that :)

On Wed, 07 Sep 2016, Colin Percival wrote:

> Hi all,
> 
> Tarsnap is designed to detect if your data is modified: Archives are
> cryptographically signed, and the signatures are verified before any
> data is extracted.  However, this depends on the integrity of the key:
> If someone has your delete and write keys, they could delete an archive
> and create a new one with the same name, and (since they have the keys)
> it would cryptographically validate.
> 
> It occurs to me that we could have a stronger unforgeability property
> via out-of-band (non-cryptographic) verification of the archive metadata
> hash; even with the keys, it would be impossible to create a different
> archive which has the same hash (unless you find a SHA256 collision).  In
> addition to the "stolen keys" scenario, this could be useful if you need
> to prove (e.g., for auditing or legal purposes) that *you* haven't changed
> an archive since the time when you created it.
> 
> Is anyone interested in having this functionality?  It seems like too
> obscure a use case to write code for if nobody wants it yet, but if there's
> a demand then it's definitely doable.
> 
> -- 
> Colin Percival
> Security Officer Emeritus, FreeBSD | The power to serve
> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid