Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-02-20 Thread Mike Hearn
I paid the new anti-spam deposit and updated the BIP 37 page to the latest version of the protocol, then marked it as accepted. High fives all round, but especially to Matt for doing the heavy lifting on this feature. On Wed, Feb 6, 2013 at 5:45 PM, Gregory Maxwell wrote: > On Wed, Feb 6, 2013

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-02-06 Thread Gregory Maxwell
On Wed, Feb 6, 2013 at 8:33 AM, Mike Hearn wrote: > Can somebody please unlock the BIP wiki page? I don't know why it was locked > but it's stale. I asked for permissions to unlock it but haven't heard back— will prod. -

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-02-06 Thread Mike Hearn
Can somebody please unlock the BIP wiki page? I don't know why it was locked but it's stale. On Wed, Jan 30, 2013 at 12:13 PM, Mike Hearn wrote: > Sorry, to clarify, these are test builds available here: > > > https://code.google.com/p/bitcoin-wallet/downloads/detail?name=bitcoin-wallet-2.39_bi

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-30 Thread Mike Hearn
Sorry, to clarify, these are test builds available here: https://code.google.com/p/bitcoin-wallet/downloads/detail?name=bitcoin-wallet-2.39_bitcoinj0.7.apk&can=2&q= It's not on the Play store yet. It probably makes sense to release after some more testing and after Bitcoin 0.8 comes out, as othe

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-30 Thread Mike Hearn
Andreas has uploaded Android builds that use the new bloom filtering and peer selection code (also, dependency analysis of transactions). The performance gain is very cool. The app feels dramatically faster to start up and sync. Because the app syncs on charge when I opened it around lunchtime it

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-19 Thread Andreas Schildbach
Matt, I saw your commit and immediately started using it for testing. Now I think the bitcoinj side needs some love because not one transaction is being confirmed (all just pending) when replaying the blockchain. On 01/18/2013 05:38 PM, Mike Hearn wrote: > I'm thinking we should actually make the

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-18 Thread Mike Hearn
I'm thinking we should actually make the change we talked about before and have the filtered block sent before the transaction data. For one, it's not intuitive (API wise) that you'd get a callback saying "new pending tx" immediately before another callback saying "tx was confirmed", but that's wh

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-16 Thread Matt Corallo
Actually, there is one more minor algorithmic change I would like to make to the way the hash function is computed really quick before it gets merged, I'll have that finished up by the end of today. Matt On Wed, 2013-01-16 at 11:43 +0100, Mike Hearn wrote: > Matts latest code has been tested by A

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-16 Thread Mike Hearn
Matts latest code has been tested by Andreas and seems to work correctly. He had to extend the client a bit to refresh the filter every 25k blocks because even with the extra flag, eventually the filter degrades into uselessness, but it did still improve the situation quite a bit. Because it's uni

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-11 Thread Mike Hearn
Oh, one last stat - syncing the entire chain with a wallet containing two keys and a 0.0001 FP rate (one or two FPs every 5 blocks or so) resulted in a download of about 46mb of data. On Fri, Jan 11, 2013 at 3:11 PM, Mike Hearn wrote: > I did some very rough initial performance tests. > > Syncing

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-11 Thread Mike Hearn
I did some very rough initial performance tests. Syncing from a local peer gives me about 50 blocks per second in the later parts of the chain (post SD), which is about a 10-20x speedup over what I could do before. This is on a MacBook Pro. But at those points it's clearly bottlenecked by bitcoind

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-10 Thread Jeff Garzik
On Thu, Jan 10, 2013 at 10:59 PM, Matt Corallo wrote: > Ive been missing lately, when is 0.8 targeted for freeze? 0.8rc1 will probably happen when the core ultraprune/leveldb stuff is stable. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-10 Thread Matt Corallo
On Thu, 2013-01-10 at 16:21 +0100, Mike Hearn wrote: > Here's a quick update on where we're up to. > > Thanks to Matts excellent work, I was able to test his bitcoinj and > bitcoin-qt work together today. There are a few minor tweaks needed, > but I feel like we're maybe a week away from having al

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2013-01-10 Thread Mike Hearn
Here's a quick update on where we're up to. Thanks to Matts excellent work, I was able to test his bitcoinj and bitcoin-qt work together today. There are a few minor tweaks needed, but I feel like we're maybe a week away from having all the code in a mergeable state. Here is the remaining work: -

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-11-27 Thread Pieter Wuille
On Wed, Nov 21, 2012 at 01:38:37PM -0500, Matt Corallo wrote: > On Wed, 2012-11-21 at 16:15 +0100, Pieter Wuille wrote: > > On Wed, Oct 24, 2012 at 05:56:07PM +0200, Mike Hearn wrote: > > > I've written a draft BIP describing the bloom filtering protocol > > > extension developed by myself and Matt

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-11-21 Thread Matt Corallo
On Wed, 2012-11-21 at 16:15 +0100, Pieter Wuille wrote: > On Wed, Oct 24, 2012 at 05:56:07PM +0200, Mike Hearn wrote: > > I've written a draft BIP describing the bloom filtering protocol > > extension developed by myself and Matt. > > > > https://en.bitcoin.it/wiki/BIP_0037 > > Two comments I mad

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-11-21 Thread Pieter Wuille
On Wed, Oct 24, 2012 at 05:56:07PM +0200, Mike Hearn wrote: > I've written a draft BIP describing the bloom filtering protocol > extension developed by myself and Matt. > > https://en.bitcoin.it/wiki/BIP_0037 Two comments I made on the pullreq page, which are probably better discussed here: * Th

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-11-06 Thread Pieter Wuille
On Fri, Oct 26, 2012 at 04:01:58PM +0200, Mike Hearn wrote: > I don't feel I understand the effort required to do some kind of > partial tree encoding. Having a kind of custom compression whereby > branches are represented as varint indexes into a dictionary, I can > feel how much work that involve

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-26 Thread Gregory Maxwell
On Fri, Oct 26, 2012 at 10:21 AM, Mike Hearn wrote: > Anyway, it's trivial to DoS the entire Bitcoin network today. It > hasn't ever happened. Maybe one day it will, but the only rationale > people can come up with for such an attack beyond random griefing is Which happens and is a concern. Altco

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-26 Thread Mike Hearn
> Because I can potentially waste bandwidth of all nodes forever (well as long > as users are still scanning blocks with my transactions in them) with O(1) > work. And this gets you what? Users who have active wallets will have their bandwidth wasted for as long as you keep up the attack. Once y

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-26 Thread Gregory Maxwell
On Fri, Oct 26, 2012 at 10:01 AM, Mike Hearn wrote: > If you just want to waste bandwidth of nodes you can connect to nodes > and repeatedly download blocks, or fill the network with fake nodes > that spam random generated transactions to whoever connects. I don't > see how to avoid that so it se

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-26 Thread Mike Hearn
> Presumably these components will just get implemented a few times in > some carefully constructed library code, so I don't see an > implementation complexity argument here— except the fact that it isn't > what Matt has implemented so far. Well, yes, that is basically the implementation complexit

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-25 Thread Gregory Maxwell
On Thu, Oct 25, 2012 at 12:56 PM, Gregory Maxwell wrote: > I still don't understand what purpose the apparently gratuitous > inefficiency of constantly resending common tree fragments. Sorry for the rapid additional comment, but I should also have mentioned that the in efficiency is at odds with

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-25 Thread Gregory Maxwell
On Wed, Oct 24, 2012 at 11:56 AM, Mike Hearn wrote: > I've written a draft BIP describing the bloom filtering protocol > extension developed by myself and Matt. > > https://en.bitcoin.it/wiki/BIP_0037 Thanks for taking the time to write this up. I still don't understand what purpose the apparent

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Jeff Garzik
On Wed, Oct 24, 2012 at 4:29 PM, Gavin Andresen wrote: > Oops, right. That memory usage is bounded right now by bounds on the > memory pool size, though, right? (I'm being lazy and not digging into > that code) Correct me if I'm wrong but... I do not think there is any bound on mempool size. My

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Mike Hearn
> What is the worst-case for an attacker interested in trying to get you > to saturate your upstream bandwidth or use lots of memory? Set a > bloom filter that matches everything, and then start requesting old > blocks in the chain? It would be slightly worse than shipping a full block but not se

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Gavin Andresen
On Wed, Oct 24, 2012 at 3:10 PM, Mike Hearn wrote: > Bitcoin already keeps track of which nodes have seen what to avoid redundant > inv announcements. Oops, right. That memory usage is bounded right now by bounds on the memory pool size, though, right? (I'm being lazy and not digging into that co

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Mike Hearn
Bitcoin already keeps track of which nodes have seen what to avoid redundant inv announcements. I think if you are approaching most transactions in a block matching the filter then you would just request full blocks and do all the filtering client side On Oct 24, 2012 8:54 PM, "Gavin Andresen" wr

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Matt Corallo
On Wed, 2012-10-24 at 14:54 -0400, Gavin Andresen wrote: > RE: sharing parts of the merkle branches when returning a 'merkleblock' : > > I think I agree that complicating the BIP for what should be a very > rare case (more than a handful of transactions in a block match the > transactions in your

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Gavin Andresen
RE: sharing parts of the merkle branches when returning a 'merkleblock' : I think I agree that complicating the BIP for what should be a very rare case (more than a handful of transactions in a block match the transactions in your wallet) is the right decision. I want to make sure I'm understandi

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Pieter Wuille
On Wed, Oct 24, 2012 at 06:35:08PM +0200, Mike Hearn wrote: > > * what does "each hash and key in the output script" mean exactly? what > about the output script in its entirety? > > It's an informal way to say data elements. If you insert a key then it > matches both single and multi sig outputs

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Pieter Wuille
On Wed, Oct 24, 2012 at 05:56:07PM +0200, Mike Hearn wrote: > I've written a draft BIP describing the bloom filtering protocol > extension developed by myself and Matt. > > https://en.bitcoin.it/wiki/BIP_0037 > > Please read it and let me know if there are any missing details or > things which so

Re: [Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Mike Hearn
> Some questions: > * why limit the number of matching transactions to 255? Copy/paste error in the does :( > * what does "each hash and key in the output script" mean exactly? what about the output script in its entirety? It's an informal way to say data elements. If you insert a key then it ma

[Bitcoin-development] Draft BIP for Bloom filtering

2012-10-24 Thread Mike Hearn
I've written a draft BIP describing the bloom filtering protocol extension developed by myself and Matt. https://en.bitcoin.it/wiki/BIP_0037 (yes I know there's some kind of process around getting allocated a number - it seems overkill for this). Please read it and let me know if there are any m