Re: [Bitcoin-development] Service bits for pruned nodes
On Sun, Apr 28, 2013 at 11:51 AM, Pieter Wuille wrote: > Hello all, > > I think it is time to move forward with pruning nodes, i.e. nodes that fully > validate and relay blocks and transactions, but which do not keep (all) > historic blocks around, and thus cannot be queried for these. > > The biggest roadblock is making sure new and old nodes that start up are > able to find nodes to synchronize from. To help them find peers, I would > like to propose adding two extra service bits to the P2P protocol: > * NODE_VALIDATE: relay and validate blocks and transactions, but is only > guaranteed to answer getdata requests for (recently) relayed blocks and > transactions, and mempool transactions. > * NODE_BLOCKS_2016: can be queried for the last 2016 blocks, but without > guarantee for relaying/validating new blocks and transactions. > * NODE_NETWORK (which existed before) will imply NODE_VALIDATE and guarantee > availability of all historic blocks. In general, I support this, as anybody on IRC knows. Though it does seem to open the question about snapshotting. Personally, it seems important to enable running a fully validating node, that may bootstrap from a UTXO snapshot + all blocks since that snapshot. NODE_BLOCKS_2016, in particular, is too short. For users, I've seen plenty of use cases in the field where you start a network sync after a 2-week period. Set a regular interval for creating a UTXO snapshot, say 3 months (6*2016 blocks), and serve all blocks after that snapshot. For older nodes, they would contact an archive node or torrent for >3 month blocks, and then download normally <= 3 month blocks (if the archive node didn't serve up to present day). Where are we on nailing down a stable, hash-able UTXO serialization? -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Tuesday 30 April 2013 21:11:47 Jeff Garzik wrote: > Hardly. The storage format is bitcoin protocol wire format, plus a > tiny header. It is supported in multiple applications already, and is > the most efficient storage format for bitcoin protocol blocks. "Most efficient" for what purpose? There is more that one might do than just duplicate bitcoind exactly. I can well imagine storing bitcoin blocks parsed and separated out into database fields. > > Wouldn't it be better to add support for more bitcoin-protocol-oriented > > HTTP requests? Then any client can supply the same interface, rather > > than being forced to create blk.dat on the fly? > > You don't have to create anything on the fly, if you store blocks in > their native P2P wire protocol format. If. What if I'm writing a client and don't want to store them the way bitcoind has? > This is a whole new client interface. It's fun to dream this up, but > it is far outside the scope of an efficient HTTP protocol that > downloads blocks. Except the alternative is no schema at all -- essentially it's just give access to a file on disk. Well, that hardly needs discussion at all, and it hardly needs the involvement of bitcoind, apache could do it right now. > Your proposal is closer to a full P2P rewrite over HTTP (or a proxy > thereof). I don't think it's a "rewrite". The wire protocol is only a small part of what bitcoind does. Adding another thread listening for HTTP requests at the same time as on 8333 for stadnard format. Anyway -- I've obviously misunderstood what the idea behind a HTTP protocol was, and it's not like I was volunteering to do any of the work ;-) Andy -- Dr Andy Parkins andypark...@gmail.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Wed, May 1, 2013 at 10:05 AM, Andy Parkins wrote: > On Tuesday 30 April 2013 21:11:47 Jeff Garzik wrote: > >> Hardly. The storage format is bitcoin protocol wire format, plus a >> tiny header. It is supported in multiple applications already, and is >> the most efficient storage format for bitcoin protocol blocks. > > "Most efficient" for what purpose? There is more that one might do than just > duplicate bitcoind exactly. I can well imagine storing bitcoin blocks parsed > and separated out into database fields. [...] >> You don't have to create anything on the fly, if you store blocks in >> their native P2P wire protocol format. > > If. What if I'm writing a client and don't want to store them the way > bitcoind has? That posits -expanding- blocks from their native form into a larger form, and then squashing them back down again upon request. A lot of extra work from the point of view of clients downloading blocks themselves. But sure, if you want to do it, yes, it is possible to design an interface like that. >> This is a whole new client interface. It's fun to dream this up, but >> it is far outside the scope of an efficient HTTP protocol that >> downloads blocks. > > Except the alternative is no schema at all -- essentially it's just give > access to a file on disk. Well, that hardly needs discussion at all, and it > hardly needs the involvement of bitcoind, apache could do it right now. Correct, Apache today could easily serve an HTTP-based layout of blk.dat, plus a tiny JSON metadata file. That's not "no schema", just a different layout. >> Your proposal is closer to a full P2P rewrite over HTTP (or a proxy >> thereof). > > I don't think it's a "rewrite". The wire protocol is only a small part of > what bitcoind does. Adding another thread listening for HTTP requests at the > same time as on 8333 for stadnard format. > > Anyway -- I've obviously misunderstood what the idea behind a HTTP protocol > was, and it's not like I was volunteering to do any of the work ;-) In the context of this thread: distributing and downloading blocks. All current users require the native binary block format. A generalized HTTP REST query protocol would be a nice addition... it is just off-topic for this thread. On IRC yesterday, we discussed an HTTP query interface like you suggested. It was agreed that it was a nice interface, and might be a nice addition to bitcoind. That is a separate topic for a separate email thread, though. As an example, see the pull request I wrote for an HTTP REST interface that downloads an encrypted wallet backup: https://github.com/bitcoin/bitcoin/pull/1982 -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Fwd: Service bits for pruned nodes
On Wednesday 01 May 2013 15:26:57 Jeff Garzik wrote: > A generalized HTTP REST query protocol would be a nice addition... it > is just off-topic for this thread. On IRC yesterday, we discussed an > HTTP query interface like you suggested. It was agreed that it was a > nice interface, and might be a nice addition to bitcoind. > > That is a separate topic for a separate email thread, though. > > As an example, see the pull request I wrote for an HTTP REST interface > that downloads an encrypted wallet backup: > https://github.com/bitcoin/bitcoin/pull/1982 Fair enough. I'm usually behind the state-of-the-art when I suggest things here :-) I should just trust you guys have already planned everything I might think of. Andy -- Dr Andy Parkins andypark...@gmail.com -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
