Re: your mail

2017-03-28 Thread Seth Arnold
Hi Leo, great response,

On Tue, Mar 28, 2017 at 08:56:25PM -0600, Leo Arias wrote:
> It's stored in $SNAP_USER_COMMON, ~/snap/bitcoin/common.
> This was the only ugly part. There are some bugs in bitcoin reported
> [...]
> problem is not yet affecting anyone else. So, for now we are patching
> bitcoin's default data dir [2].

bitcoin may be the most extreme example I can think of quickly but surely
it's not alone in this.

> > - Have you run into problems moving BDB forward or backwards when
> >   upgrading or downgrading? Is BDB used in the storage of the blockchain
> >   or is it used for 'simpler' snap storage?
> 
> I'm not sure what do you mean with simpler snap storage. There's no
> storage specific to the snap. Where the bitcoin code uses the default
> data dir, that's unversioned. Where it uses HOME or XDG dirs directly,
> that data is versioned. I don't have high hopes for better data
> handling because the bugs I found were from a couple of years ago, and
> mentioned the split of the wallet code that was supposed to happen
> soon at that time, but never happened.

I was mostly curious if BDB was being used for the blockchain or user
preferences or other metadata; I had some impression many years ago that
BDB was touchy, so I wondered if you had run into trouble.

> > It feels like bitcoin's 100-ish gigabyte blob makes an interesting
> > constraint that many simpler tools may not need to address.
> 
> That was precisely my thinking. Would you like to help us maintaining
> the snap until we make the transfer upstream? It would be great to
> increase our bus factor, and more hands will make it easier to improve
> the packaging.

Thanks for the invitation but I'm already feeling vastly oversubscribed. :(

Thanks


signature.asc
Description: PGP signature
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft


Re: your mail

2017-03-28 Thread Leo Arias
Hello Seth,

In many things, this works just like the tar.gz that you download from
the bitcoin website.

On Tue, Mar 28, 2017 at 4:49 PM, Seth Arnold  wrote:
> - At what point do you download the blockchain?

It will download the blockchain when you open the qt client.

> - Where does the blockchain get stored?

It's stored in $SNAP_USER_COMMON, ~/snap/bitcoin/common.
This was the only ugly part. There are some bugs in bitcoin reported
to allow more control over the data files. But currently, it stores
everything in $HOME, which inside the snap is redirected to
$SNAP_USER_DATA. This path is versioned, but we don't want to copy the
entire blockchain every time the snap is updated. A few days ago I
proposed an option to have an unversioned home [1], but it seems this
problem is not yet affecting anyone else. So, for now we are patching
bitcoin's default data dir [2].

> - How are interrupted downloads handled?

Nothing specific to the snap here, bitcoin handles it. You close the
app or get disconnected and the download stops. You open the app or
reconnect and it resumes.

> - How are package upgrades handled? Is a copy of the blockchain made?

No, it's kept in the common dir, unversioned.

> - How are package downgrades handled? Is a copy of the blockchain made?

Nothing specific to the snap here yet. Currently, if the upgrade is
not backwards compatible, the same will happen as if you open an older
bitcoin version with a directory holding newer data.

We could do fancier things on upgrades and downgrades with the epoch
work, but that's not yet documented, I'm not even sure if it's
complete.

> - Have you run into problems moving BDB forward or backwards when
>   upgrading or downgrading? Is BDB used in the storage of the blockchain
>   or is it used for 'simpler' snap storage?

I'm not sure what do you mean with simpler snap storage. There's no
storage specific to the snap. Where the bitcoin code uses the default
data dir, that's unversioned. Where it uses HOME or XDG dirs directly,
that data is versioned. I don't have high hopes for better data
handling because the bugs I found were from a couple of years ago, and
mentioned the split of the wallet code that was supposed to happen
soon at that time, but never happened.

We haven't had problems with updates and downgrades, but well, this is
the first version pushed to the store, no much room for failure there.
What's cool with the snap is that we get a new version pushed to edge
each day, so we can catch any potential breakages right after they are
introduced. And as soon as upstream starts tagging candidates for the
0.14.1 release we can move them to the candidate channel and easily
test updates from 0.14 with snap refresh bitcoin --candidate.

However, these checks are not yet automated, so the success of the
snap depends on the community testing before we push to stable. That's
the reason for this call for testing.

During the experiments with IPFS [3] we already had 3 or 4 successful
stable updates, and the process is just lovely and it has worked
without any incidents. That's why I'm interested in this more complex
scenario with bitcoin.

> It feels like bitcoin's 100-ish gigabyte blob makes an interesting
> constraint that many simpler tools may not need to address.

That was precisely my thinking. Would you like to help us maintaining
the snap until we make the transfer upstream? It would be great to
increase our bus factor, and more hands will make it easier to improve
the packaging.

pura vida.

[1] https://lists.ubuntu.com/archives/snapcraft/2017-March/003494.html
[2] 
https://github.com/elopio/blockchain-snaps/blob/master/bitcoin/snap/patches/default_data_dir.patch
[3] http://elopio.net/blog/ipfs-crowdtesting/

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft


Re: your mail

2017-03-28 Thread Seth Arnold
On Tue, Mar 28, 2017 at 11:20:41AM -0600, Leo Arias wrote:
> Here are the sources for the snap and CI scripts, in case you want to
> verify them or build it yourself:
> https://github.com/elopio/blockchain-snaps/blob/master/bitcoin/snap/snapcraft.yaml

Hi Leo, it looks like you've documented this fairly well but I'm not
familiar enough with the details of snap packaging to know how it
actually works in practice. This might serve as a good example for the
snap community:

- At what point do you download the blockchain?
- Where does the blockchain get stored?
- How are interrupted downloads handled?
- How are package upgrades handled? Is a copy of the blockchain made?
- How are package downgrades handled? Is a copy of the blockchain made?
- Have you run into problems moving BDB forward or backwards when
  upgrading or downgrading? Is BDB used in the storage of the blockchain
  or is it used for 'simpler' snap storage?

It feels like bitcoin's 100-ish gigabyte blob makes an interesting
constraint that many simpler tools may not need to address.

Thanks


signature.asc
Description: PGP signature
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft