Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-04 Thread Matt Corallo via bitcoin-dev
On December 3, 2015 7:02:20 AM GMT+08:00, Peter Tschipper wrote: >On 02/12/2015 2:23 PM, Matt Corallo wrote: >> My issue is more that its additional complexity and attack surface, >> and for a very minor gain >What is a minor gain? 15 to 27% compression sounds good to me and the >larger the d

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-03 Thread Rusty Russell via bitcoin-dev
Gavin Andresen via bitcoin-dev writes: > On Wed, Dec 2, 2015 at 1:57 PM, Emin Gün Sirer < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> How to Do It >> >> If we want to compress Bitcoin, a programming challenge/contest would be >> one of the best ways to find the best possible, Bitcoin-spec

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-03 Thread Gavin Andresen via bitcoin-dev
On Wed, Dec 2, 2015 at 1:57 PM, Emin Gün Sirer < bitcoin-dev@lists.linuxfoundation.org> wrote: > How to Do It > > If we want to compress Bitcoin, a programming challenge/contest would be > one of the best ways to find the best possible, Bitcoin-specific > compressor. This is the kind of self-conta

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-03 Thread Dave Scotese via bitcoin-dev
Emin's email presents to me the idea of dictionaries that already contain the data we'd want to compress. With 8 bytes of indexing data, we can refer to a TxID or a Public Key or any existing part of the blockchain. There are also data sequences like scripts that contain a few variable chunks and

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Peter Tschipper via bitcoin-dev
On 30/11/2015 9:28 PM, Matt Corallo wrote: > I'm really not a fan of this at all. To start with, adding a compression > library that is directly accessible to the network on financial software is a > really, really scary idea. Why scary? LZO has no current security issues, and it will be confi

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Peter Tschipper via bitcoin-dev
On 02/12/2015 2:23 PM, Matt Corallo wrote: > My issue is more that its additional complexity and attack surface, > and for a very minor gain What is a minor gain? 15 to 27% compression sounds good to me and the larger the data the better the compression. And although there is a decent peformance

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Matt Corallo via bitcoin-dev
My issue is more that its additional complexity and attack surface, and for a very minor gain which should disappear with further optimization elsewhere and less that we absolutely shouldn't add compression because we're definitely gonna have issues. On 12/02/15 20:16, Peter Tschipper via bitc

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Peter Tschipper via bitcoin-dev
Building a compressor from scratch may yeild some better compression ratios, or not, but having trust and faith in whether it will stand up against attack vectors another matter. LZO has been around for 20 years with very few problems and no current issues. Maybe something better can be built, bu

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Emin Gün Sirer via bitcoin-dev
Thanks Peter for the careful, quantitative work. I want to bring one additional issue to everyone's consideration, related to the choice of the Lempel-Ziv family of compressors. While I'm not familiar with every single compression engine tested, the Lempel-Ziv family of compressors are generally

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Patrick Strateman via bitcoin-dev
If compression is to be used a custom compression algorithm should be written. Bitcoin data is largely incompressible outside of a tiny subset of fields. On 12/01/2015 11:33 PM, Simon Liu via bitcoin-dev wrote: > Hi Pavel, > > (my earlier email was moderated, so the list can only see it via your

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-02 Thread Simon Liu via bitcoin-dev
Hi Pavel, (my earlier email was moderated, so the list can only see it via your reply), Yes, an attacker could try and send malicious data to take advantage of a compression library vulnerability... but is it that much worse than existing attack vectors which might also result in denial of servi

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-01 Thread Pavel Janík via bitcoin-dev
> On 02 Dec 2015, at 00:44, Simon Liu wrote: > > Hi Matt/Pavel, > > Why is it scary/undesirable? Thanks. Select your preferable compression library and google for it with +CVE. E.g. in zlib: http://www.cvedetails.com/vulnerability-list/vendor_id-72/product_id-1820/GNU-Zlib.html …allows rem

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-12-01 Thread Pavel Janík via bitcoin-dev
> On 01 Dec 2015, at 06:28, Matt Corallo via bitcoin-dev > wrote: > > I'm really not a fan of this at all. To start with, adding a compression > library that is directly accessible to the network on financial software is a > really, really scary idea. I have the same opinion. On the other h

Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-11-30 Thread Matt Corallo via bitcoin-dev
I'm really not a fan of this at all. To start with, adding a compression library that is directly accessible to the network on financial software is a really, really scary idea. If there were a massive improvement, I'd find it acceptable, but the improvement you've shown really isn't all that mu

[bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions

2015-11-30 Thread Peter Tschipper via bitcoin-dev
@gmaxwell Bip Editor, and the Bitcoin Dev Community, After several weeks of experimenting and testing with various compression libraries I think there is enough evidence to show that compressing blocks and transactions is not only beneficial in reducing network bandwidth but is also provides a sm