[bitcoin-dev] Fast bootstrapping with a pre-generated UTXO-set database
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi I’ve been thinking around a solution to reduce nodes bootstrap time (IBD) as well as a way to reduce the amount of bandwidth/network usage per node. Not sure if this idea was/is already discussed, haven’t found anything in a quick research. ==Title== Fast bootstrapping with a pre-generated UTXO-set database. ==Abstract== This documents describes a way how bitcoin nodes can bootstrap faster by loading a pre-generated UTXO-set datafile with moderate reduction of the security model. ==Specification== Bitcoin-core or any other full node client will need to provide a feature to "freeze" the UTXO-set at a specified height (will require a reindex). The frozen UTXO-set – at a specific height – will be deterministic linearized in a currently not specified data-serializing-format. Additionally, a serialized form of the current chain-index (chain containing all block-headers) up to the specified height will be appended to the pre-generated UTXO-set-datafile. The datafile will be hashed with a double SHA256. The corresponding hash will be produced/reproduced and signed (ECDSA) by a group of developers, ideally the same group of developers who are also signing deterministic builds (binary distribution). Full node client implementations that supports bootstrapping from a pre-generated UTXO-set, need to include... 1.) a set of pubkeys from trusted developers 2.) the hash (or hashes) of the pre-generated UTXO-set-datafile(s) 3.) n signatures of the hash(es) from 2) from a subset of developers defined in 1) To guarantee the integrity of developers pubkeys & signatures, methods like the current gitian build, used in bitcoin-core, must be used. New nodes could download a copy of the pre-generated UTXO-set, hash it, verify the hash against the allowed UTXO-sets, verify the ECDSA signatures from various developers, and continue bootstrapping from the specified height if the users accepts the amount of valid signatures . Sharing of the pre-generated UTXO-set can be done over CDNs, bit-torrent or any other file hosting solution. It would also be possible to extend the bitcoin p2p layer with features to distribute/share a such pre-generated UTXO-set, in chunks and with the according hashes to detect invalidity before downloading the whole content (but would probably end up in something very similar to bit-torrent). - -- -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJW1B1wAAoJECnUvLZBb1PsqzsP/iSdvyhUzy+BZVSZbKXNjk5P 2vrtirI6NvKQd8hHbrcFeLfyswzYc2JWRnX8sATlauIS0pYdr97JriwUGlvxvNrY iVTDdf8MIVu8zScLQtJbMatpMvsewtqQEidn/yxWIhiCg4G2T5DZmlBU6O4XIKR6 5aPHElGOKZ15EWGHBG7z4owj3MiOaxhD9q5erBbfLPpcm08o6XAv5miqmGnnn3zh gocg4Gxs6iDygh3b2dCJFwWIVPxF6UVJhyjv2kLZUmEHT2Y2QvdGcLIIewcWHDze kgoZYmOEowujCbmeJ+LBwgOI0c1N6L/ciomPBne7ILmK4LyUEzyMLJKNYf/sZ8vI sVlmwZwZZLfILC7mzMAM0pfj99IOW680WHch9v31lWFlxW/bLvLqAO7n3acQuD6s xCZN2nAhmWC8FnMFxqB3EUz0lX8giV3qRJZjbQMS+ZrngYkAmVv2bAsoLndqf6MO l9W8B+ICg1KZLGIOF2pUrInpkB6gUALDFnypV4CeIVdeqtk5l4LnCHK6c4++Hl5n Bv5HQ/wTgKKNFtHBEJpWyYWvAjfFtgUZUKblR+Bh+D7/Gte1ehiYd02KYD4ds9Y4 3gfO8YbAz/I14Yuh2bIlvVKPWnLQBwL5BBioBfvmhV/r6rEpzWvB7H6Fmi1c759l VlL0GiUV8ar2LlFhEmWk =lZSy -END PGP SIGNATURE- ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] BIP CPRKV: Check private key verify
This is actually very useful for LN too, see relevant discussion here http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011827.html 2016-02-12 11:05 GMT+01:00 Tier Nolan via bitcoin-dev : > On Fri, Feb 12, 2016 at 5:02 AM, wrote: >> >> Seems it could be done without any new opcode: > > > The assumption was that the altcoin would only accept standard output > scripts. Alice's payment in step 2 pays to a non-standard script. > > This is an improvement over the cut and choose, but it will only work for > coins which allow non-standard scripts (type 2 in the BIP). > > I guess I was to focused on maintaining standard scripts on the altcoin. > > ___ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Fast bootstrapping with a pre-generated UTXO-set database
One of the proposals was to build the UTXO set backwards. You start from the newest block and work backwards. The database contains UTXOs (unspent transaction outputs) and "UFTXI" (unfunded transaction inputs). The procedure would be For each transaction (last to first ordering) For each output - check if it is in the UFTXI set -- If so, validate the signatures -- If not, add it to the UTXO set For each input - Add to the UFTXI set When you receive a transaction, it checks all the inputs -- If all inputs are in the UTXO set, it says confirmed -- Otherwise, gets marked as "unknown inputs" There would also be a counter indicating how many blocks it has validated. A transaction with an unfunded input counts as validated back to the block it was included in. Transactions count as confirmed to their ancestor that has the newest validation time. Assume that the node had validated the last 1 blocks and you had a transaction with one input. Assume the input transaction was included 5000 blocks ago and its input was included 50,000 blocks ago. TX-A) input (TX-B:0) included in block 6 blocks ago TX-B) input (TX-C:0) included in block 5000 ago TX-C) input (TX-B:0) included in block 2 ago TX-C would not be known to the node since it has only gone back 1 blocks. TX-A would have confirms 6 / 5000. This means that its outputs have been confirmed by 6 blocks (confirms work as currently) and that its inputs have been confirmed by 5000 blocks. The reference client could mark transactions with 6+ output confirms and 1000+ input confirms as confirmed. Once it hits the genesis block, then all transactions would be 6/, so it could drop the second number. On Mon, Feb 29, 2016 at 10:29 AM, Jonas Schnelli via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Hi > > I’ve been thinking around a solution to reduce nodes bootstrap time > (IBD) as well as a way to reduce the amount of bandwidth/network usage > per node. > Not sure if this idea was/is already discussed, haven’t found anything > in a quick research. > > > ==Title== > Fast bootstrapping with a pre-generated UTXO-set database. > > ==Abstract== > This documents describes a way how bitcoin nodes can bootstrap faster > by loading a pre-generated UTXO-set datafile with moderate reduction > of the security model. > > ==Specification== > Bitcoin-core or any other full node client will need to provide a > feature to "freeze" the UTXO-set at a specified height (will require a > reindex). The frozen UTXO-set – at a specific height – will be > deterministic linearized in a currently not specified > data-serializing-format. > Additionally, a serialized form of the current chain-index (chain > containing all block-headers) up to the specified height will be > appended to the pre-generated UTXO-set-datafile. > The datafile will be hashed with a double SHA256. > > The corresponding hash will be produced/reproduced and signed (ECDSA) > by a group of developers, ideally the same group of developers who are > also signing deterministic builds (binary distribution). > > Full node client implementations that supports bootstrapping from a > pre-generated UTXO-set, need to include... > 1.) a set of pubkeys from trusted developers > 2.) the hash (or hashes) of the pre-generated UTXO-set-datafile(s) > 3.) n signatures of the hash(es) from 2) from a subset of developers > defined in 1) > > To guarantee the integrity of developers pubkeys & signatures, methods > like the current gitian build, used in bitcoin-core, must be used. > > New nodes could download a copy of the pre-generated UTXO-set, hash > it, verify the hash against the allowed UTXO-sets, verify the ECDSA > signatures from various developers, and continue bootstrapping from > the specified height if the users accepts the amount of valid signatures > . > > Sharing of the pre-generated UTXO-set can be done over CDNs, > bit-torrent or any other file hosting solution. It would also be > possible to extend the bitcoin p2p layer with features to > distribute/share a such pre-generated UTXO-set, in chunks and with the > according hashes to detect invalidity before downloading the whole > content (but would probably end up in something very similar to > bit-torrent). > > > - -- > > -BEGIN PGP SIGNATURE- > Version: GnuPG v2 > > iQIcBAEBCAAGBQJW1B1wAAoJECnUvLZBb1PsqzsP/iSdvyhUzy+BZVSZbKXNjk5P > 2vrtirI6NvKQd8hHbrcFeLfyswzYc2JWRnX8sATlauIS0pYdr97JriwUGlvxvNrY > iVTDdf8MIVu8zScLQtJbMatpMvsewtqQEidn/yxWIhiCg4G2T5DZmlBU6O4XIKR6 > 5aPHElGOKZ15EWGHBG7z4owj3MiOaxhD9q5erBbfLPpcm08o6XAv5miqmGnnn3zh > gocg4Gxs6iDygh3b2dCJFwWIVPxF6UVJhyjv2kLZUmEHT2Y2QvdGcLIIewcWHDze > kgoZYmOEowujCbmeJ+LBwgOI0c1N6L/ciomPBne7ILmK4LyUEzyMLJKNYf/sZ8vI > sVlmwZwZZLfILC7mzMAM0pfj99IOW680WHch9v31lWFlxW/bLvLqAO7n3acQuD6s > xCZN2nAhmWC8FnMFxqB3EUz0lX8giV3qRJZjbQMS+ZrngYkAmVv2bAsoLndqf6MO > l9W8B+ICg1KZLGIOF2pUrInpkB6
Re: [bitcoin-dev] BIP CPRKV: Check private key verify
On Mon, Feb 29, 2016 at 10:58 AM, Mats Jerratsch wrote: > This is actually very useful for LN too, see relevant discussion here > > > http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011827.html > Is there much demand for trying to code up a patch to the reference client? I did a basic one, but it would need tests etc. added. I think that segregated witness is going to be using up any potential soft-fork slot for the time being anyway. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Open Bitcoin Privacy Protect Privacy Questionnaire, Mid-Year 2015 report
Better late than never, I should correct things here. In the future it would probably be more productive to open an issue. Otherwise there is no mechanism for someone to take ownership of a response. On Sun, Aug 30, 2015 at 7:45 PM, Kristov Atlas via bitcoin-dev wrote: >> 1. Does your application take any steps to create ambiguity between >> transactions which unavoidably spend from multiple addresses at the same >> time and intentional mixing transactions? > No, Bitcoin-Qt does not try to make non-mixing transactions look like mixing > transactions. >> 2. What algorithms does your application use for ordering inputs and >> outputs in a transaction? In particular, how do you handle the change output >> and do you take into account common practices of other wallet applications >> when determining ordering? > > Not yet BIP 69. These notes suggest that outputs are randomized: > https://bitcoin.org/en/release/v0.8.1 The ordering used by Bitcoin-QT is cryptographically randomized. This provides the greatest privacy possible. The BIP 69 recommendation would currently be equally as private if universally used, but today would reduce privacy by making the software more distinguishable. It is unclear if BIP69 will be equal in privacy in the future, because external infrastructure may impose ordering requirements that are incompatible with it. >> 3. Does your application minimize the harmful effects of address >> reuse by spending every spendable input (“sweeping”) from an address when a >> transaction is created? > > Unknown >> 4. Does your application fully implement BIP 62? BIP 62 is withdrawn. The useful mechanisms in it for standardness rules are, of course, implemeted in Bitcoin Core-- were invented there, and have been there for years. >> Mixing >> >> 5. If your application supports mixing: It's unclear to me precisely what is meant here. I'll answer broadly. Bitcoin Core is compatible with and can be used with the joinmarket module to include coinjoins. The raw transaction functionality in Bitcoin Core was also created specifically to facilitate coinjoins. Beyond joinmarket there have been several other coinjoin modules created for Bitcoin Core though today JM is by far the most common, This functionality is not directly implemented for a number of reasons including the non-existence of decenteralized tools for this that don't harm the user's privacy in other ways. >> a. What is the average number of participants a user can expect to >> interact with on a typical join transaction? >> b. Does your application attempt to construct join transactions in a >> way that avoids distinguishing them from non-join transactions? >> c. Does your application perform any kind of reversibility analysis >> on join transactions prior to presenting them to the user for confirmation? >> d. Is the mixing technique employed secure against correlation >> attacks by the facilitator, such as a CoinJoin server or off-chain mixing >> service? >> e. Is the mixing technique employed secure against theft of funds by >> the facilitator or its participants? Skipped as these are specific to the implementation in use. >> Donations >> 6. If your application has a fee or donation to the developers >> feature: > No donation feature last time I checked. >> a. What steps do you take to make the donations indistinguishable >> from regular spend in terms of output sizes and destination addresses? As Kristov noted, Bitcoin Core does not implement anti-features like donations. >> Balance Queries and Tx Broadcasting >> >> 7. Please describe how your application obtains balance information >> in terms of how queries from the user’s device can reveal a connection >> between the addresses in their wallet. >> a. Does the application keep a complete copy of the blockchain >> locally (full node)? > Yes Optionally, but in all cases the user's privacy is indistinguishable from keeping all the data locally. >> b. Does the user’s device provide a filter which matches some >> fraction of the blockchain while providing a false positive rate (bloom or >> prefix filters)? > No, it just downloads the whole blockchain and performs local queries. It would be more correct to say that Bitcoin Core always has the highest possible FP rate. It uses the only currently available tool to avoid leaking private address information to indexing services. As several academic studies have shown, bloom filters are completely inadequate for protecting user privacy. >> i. If so, approximately what fraction of the blockchain does the >> filter match in a default configuration (0% - 100%)? > 100%, unless a user bootstraps downloading the blockchain. Bootstrapping > will potentially limit the user's anonymity set to other people who have > downloaded that bootstrap.dat file. I user that has downloaded a bootstrap.dat is indistinguishable from any other user on the network; their transa
Re: [bitcoin-dev] SIGHASH_NOINPUT in Segregated Witness
Joseph Poon via bitcoin-dev writes: > Ideally, a 3rd-party can be handed a transaction which can encompass all > prior states in a compact way. For currently-designed Segregated Witness > transactions, this requires storing all previous signatures, which can > become very costly if individuals to thousands of channel state updates > per day. AFAICT we need more than this. Or are you using something other than the deployable lightning commit tx style? If each HTLC output is a p2sh[1], you need the timeout and rhash for each one to build the script to redeem it. In practice, there's not much difference between sending a watcher a tx for every commit tx and sending it information for every new HTLC (roughly a factor of 2). So we also need to put more in the scriptPubKey for this to work; either the entire redeemscript, or possibly some kind of multiple-choice P2SH where any one of the hashes will redeem the payment. Cheers, Rusty. [1] eg. from https://github.com/ElementsProject/lightning/blob/master/doc/deployable-lightning.pdf OP_HASH160 OP_DUP # Replace top element with two copies of its hash OP_EQUAL # Test if they supplied the HTLC R value OP_SWAP OP_EQUAL OP_ADD # Or the commitment revocation hash OP_IF # If any hash matched. # Pay to B. OP_ELSE # Must be A, after HTLC has timed out. OP_CHECKLOCKTIMEVERIFY Ensure (absolute) time has passed. OP_CHECKSEQUENCEVERIFY # Delay gives B enough time to use revocation if it has it. OP_2DROP # Drop the delay and htlc-timeout from the stack. # Pay to A. OP_ENDIF OP_CHECKSIG # Verify A or B's signature is correct. Cheers, Rusty. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
