Re: [Bitcoin-development] Draft BIP for Bloom filtering
Here's a quick update on where we're up to. Thanks to Matts excellent work, I was able to test his bitcoinj and bitcoin-qt work together today. There are a few minor tweaks needed, but I feel like we're maybe a week away from having all the code in a mergeable state. Here is the remaining work: - There are a couple of bugfixes needed on the bitcoinj side: the fallback to downloading full blocks is problematic and needs to be deleted, there's an API change we want - Adjust the default FP rate requested by BCJ to be 0.0001, this is appropriate for the latest blocks in the chain and yields 0-5 false positives per block - Introduce a new part to the filter protocol that allows clients to control auto-expansion. This turned out to be very volatile, we saw jumps from 0-3 FPs per block to 500 in the space of 1 block, perhaps if a SatoshiDice transaction got into the filter. A simple yes/no flag can suffice for now, but a better solution would be for the client to submit templates for output scripts that would trigger auto-adding the matched outpoint - autoexpansion is only needed in the case where the input script doesn't contain any predictable data. For pay-to-address and P2SH it does, so expansion doesn't help. Matt said he'd hopefully try to look at this soon. With auto-expansion disabled, the FP rate adjusted and a bugfix on the bcj side I was able to sync a wallet using a bloom filtered chain. Although it's tight, I think this work should go into 0.8 - it'll be much more compelling to advertise it this way, we can say Upgrade to 0.8 and help network performance for everyone. And in the case that we discover a showstopper problem, we just don't deploy the code that uses the new messages into clients. -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnmore_122712 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] BitCoin and MinorFs/AppArmor
My efforts on MinorFs2 have been stalled for a large period of time (resulting from the fact that I was writing it in Python and Google pulling the plug on its codesearch server showed me that Python was a language that I hadn't sufficiently mastered. Recently, after almost a year of reluctance at starting from scratch, I finaly set myself to rewriting what I had done so far in C++ and to continue development. Yesterday I've also started at a library for making the usage of MinorFs by applications like Bitcoin more convenient. The idea is that LibMinorFs will become a simple to use library for making applications MinorFs2-aware. I would like to ask if the bitcoin developers would be open to using this library within bitcoin for making BitCoin MinorFs2 aware, allowing it to store things like the BitCoin purse in a private directory that is not available to other unrelated (potentially trojan) processes running under the same uid. The API is defined in the inc/minorfs/ directory and the usage shown in src/testmain.cpp https://github.com/pibara/LibMinorFs2 Both the library and the filesystems are still far from being done, but I would like to know if you guys would be open to a pull-request that would incorporate a copy of this library into Bitcoin, making BitCoin use private storage when available, implemented using the above API. Please let me know what you think. Rob On Fri, August 26, 2011 08:48, Rob Meijer wrote: A few years ago I wrote a least authority based set of filesystems named MinorFs that worked closely together with AppArmor (suse/ubuntu) to give ' pseudo persistent processes' their own private but decomposable and delegatable piece of filesystem storage: http://www.linuxjournal.com/magazine/minorfs http://www.capibara.com/blog/2011/05/25/taming-mutable-state-for-file-systems/ Currently there is only one perfect fit for MinorFs and that's the stack AppArmor/MinorFs/E-language-persistent-application. There are some close fits like running ssh without a passphrase ( http://minorfs.polacanthus.net/wiki/Ssh_private_keys_without_passphrase ) but these require lots of manual fiddling by the user to get working. The ssh trick would probably work with bitcoin, but as you can see from the link above, it would be rather cumbersome. I am trying to get specs together for rewriting MinorFs (in Python) in a way that would make it easy and natural for application developers that want their application to be able to protect user data (like bitcoin wallets) from mallware running under the same uid as that user. Currently minorfs granularity is hard fixed to that of the 'pseudo persistent process', and that granularity is determined as described in the following link: http://minorfs.polacanthus.net/wiki/Pseudo_persistent_process When using pseudo persistent processes, you basically end up with file-system storage that follows almost all of the modeling principles of the object capability model. This is great when designing a least authority program from scratch and writing it in the (object capability) e-language using its persistence facilities. Given however that I don't expect bitcoin, openssh, chrome, firefox, or any other application that would benefit from what MinorFs provides to be rewritten in E, it seems like the next version of MinorFs should give up on the purity of its least authority model, and take an approach that better suits common development languages and practices. With bitcoin being a project that could benefit most from what MinorFs has to offer, I would like to ask bitcoin developers to think about what attributes from the current granularity level (pseudo persistent process) should be kept, what attributes should be dropped, and what properties should be added to arrive at an 'id' that is the best fit for granularity of persistent private storage for bitcoin. I really want to accommodate bitcoin developer needs in this, so all input that helps me help you guys to get the next MinorFs version to accommodate your needs to a level that code to use MinorFs where available can be added to bitcoin, would be extremely welcome. Let me know what you think, Rob -- EMC VNX: the world's simplest storage, starting under $10K The only unified storage solution that offers unified management Up to 160% more powerful than alternatives and 25% more efficient. Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video
Re: [Bitcoin-development] Draft BIP for Bloom filtering
On Thu, 2013-01-10 at 16:21 +0100, Mike Hearn wrote: Here's a quick update on where we're up to. Thanks to Matts excellent work, I was able to test his bitcoinj and bitcoin-qt work together today. There are a few minor tweaks needed, but I feel like we're maybe a week away from having all the code in a mergeable state. Here is the remaining work: - There are a couple of bugfixes needed on the bitcoinj side: the fallback to downloading full blocks is problematic and needs to be deleted, there's an API change we want First of the two is done. - Adjust the default FP rate requested by BCJ to be 0.0001, this is appropriate for the latest blocks in the chain and yields 0-5 false positives per block Is a part of the larger API changes mentioned above. - Introduce a new part to the filter protocol that allows clients to control auto-expansion. This turned out to be very volatile, we saw jumps from 0-3 FPs per block to 500 in the space of 1 block, perhaps if a SatoshiDice transaction got into the filter. A simple yes/no flag can suffice for now, but a better solution would be for the client to submit templates for output scripts that would trigger auto-adding the matched outpoint - autoexpansion is only needed in the case where the input script doesn't contain any predictable data. For pay-to-address and P2SH it does, so expansion doesn't help. Matt said he'd hopefully try to look at this soon. The flags mentioned have been implemented, both to disable autoexpansion, enable it for all outputs, enable for only pay to pubkey outputs (the most likely use-case), or use a set of templates. The matched templates part isn't properly tested and I would like comments on that part (see the last few commits at https://github.com/bitcoin/bitcoin/pull/1795). With auto-expansion disabled, the FP rate adjusted and a bugfix on the bcj side I was able to sync a wallet using a bloom filtered chain. Although it's tight, I think this work should go into 0.8 - it'll be much more compelling to advertise it this way, we can say Upgrade to 0.8 and help network performance for everyone. And in the case that we discover a showstopper problem, we just don't deploy the code that uses the new messages into clients. Ive been missing lately, when is 0.8 targeted for freeze? Matt -- Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and much more. Get web development skills now with LearnDevNow - 350+ hours of step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122812 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Draft BIP for Bloom filtering
On Thu, Jan 10, 2013 at 10:59 PM, Matt Corallo bitcoin-l...@bluematt.me wrote: Ive been missing lately, when is 0.8 targeted for freeze? 0.8rc1 will probably happen when the core ultraprune/leveldb stuff is stable. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and much more. Get web development skills now with LearnDevNow - 350+ hours of step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122812 ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
