Re: [bitcoin-dev] [Lightning-dev] On the scalability issues of onboarding millions of LN mobile clients
On Fri, May 8, 2020 at 2:00 PM Keagan McClelland via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Perhaps I wasn't explicit in my previous note but what I mean is that > there seems to be a demand for something *in between* a peer interface, > and an owner interface. I have little opinion as to whether this belongs in > core or not, I think there are much more experienced folks who can weight > in on that, but without something like this, you cannot limit your exposure > for serving something like bip157 filters without removing your own ability > to make use of some of those same services. > Our FullyNoded2 multisig wallet on iOS & Mac, communicates with your own personal node over RPC, securing the connection using Tor over a hidden onion service and two-way client authentication using a v3 Tor Authentication key: https://github.com/BlockchainCommons/FullyNoded-2 It many ways the app (and its predecessor FullyNoded1) is an interface between a personal full node and a user. However, we do wish that the full RPC functionality was not exposed in bitcoin-core. I’d love to see a cryptographic capability mechanism such that the remote wallet could only m ask the node functions that it needs, and allow escalation for other rarer services it needs with addition authorization. This capability mechanism feature set should go both ways, to a minimum subset needed for being a watch-only transaction verification tool, all the way to things RPC can’t do like deleting a wallet and changing bitcoin.conf parameters and rebooting, without requiring full ssh access to the server running the node. If there are people interested in coordinating some proposals on how to defining different sets of wallet functionality, Blockchain Commons would be interested in hosting that collaboration. This could start as just being a transparent shim between bitcoin-core & remote RPC, but later could inform proposals for the future of the core wallet functionality as it gets refactored. — Christopher Allen ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] [Lightning-dev] On the scalability issues of onboarding millions of LN mobile clients
On 5/5/20 5:31 PM, Olaoluwa Osuntokun via bitcoin-dev wrote: > Hi Antoine, > >> Even with cheaper, more efficient protocols like BIP 157, you may have a >> huge discrepancy between what is asked and what is offered. Assuming 10M >> light clients [0] each of them consuming ~100MB/month for filters/headers, >> that means you're asking 1PB/month of traffic to the backbone network. If >> you assume 10K public nodes, like today, assuming _all_ of them opt-in to >> signal BIP 157, that's an increase of 100GB/month for each. Which is >> consequent with regards to the estimated cost of 350GB/month for running >> an actual public node > One really dope thing about BIP 157+158, is that the protocol makes serving > light clients now _stateless_, since the full node doesn't need to perform > any unique work for a given client. As a result, the entire protocol could > be served over something like HTTP, taking advantage of all the established > CDNs and anycast serving infrastructure, which can reduce syncing time > (less latency to > fetch data) and also more widely distributed the load of light clients using > the existing web infrastructure. Going further, with HTTP/2's server-push > capabilities, those serving this data can still push out notifications for > new headers, etc. The statelessness of compact block filters does look useful. Bloom filters for blocks can be inefficient, during a scan with a BIP37 wallet, it's necessary to discard already received merkle blocks as the filter has been updated and the previous results may have missed transactions. Both bitcoinj [1] and breadwallet-core [2] handle it using a similar method. The alternative of synchronizing and alternating between requesting blocks and filter updates leads to slow scan times. With compact block filters, a separate wallet process (from the full node) can make adjustments necessary to what it needs to filter without having to communicate with the full node. [1]: https://github.com/bitcoinj/bitcoinj/blob/806afa04419ebdc3c15d5adf065979aa7303e7f6/core/src/main/java/org/bitcoinj/core/Peer.java#L1076-L1079 [2]: https://github.com/breadwallet/breadwallet-core/blob/8eb05454df3e2d5cca248b4e24eeffa420c97e3a/bitcoin/BRPeer.c#L83-L85 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] [Lightning-dev] On the scalability issues of onboarding millions of LN mobile clients
> The RPC interface in Bitcoin Core, and others, is not great for this > because it exposes a lot of functionality that isn't necessary and > introduces risks. This is actually somewhat my point. If the RPC interface was good for this and *didn't* introduce risks, we could just use that and be done with it. But I'm finding there are many use cases that you want to have low cost ways to serve peer services to people whom you have given explicit permission, but they shouldn't have full ability to administrate the node. Perhaps I wasn't explicit in my previous note but what I mean is that there seems to be a demand for something *in between* a peer interface, and an owner interface. I have little opinion as to whether this belongs in core or not, I think there are much more experienced folks who can weight in on that, but without something like this, you cannot limit your exposure for serving something like bip157 filters without removing your own ability to make use of some of those same services. Keagan On Fri, May 8, 2020 at 1:51 PM Braydon Fuller wrote: > On 5/6/20 9:07 PM, Keagan McClelland wrote: > > > I think that one of the solutions here is to have light clients choose > > their full node tethers explicitly. Even if you think it is unrealistic > to > > have everyone run their own node (fwiw, I don’t), there is still a trust > > model where you can pick your trusted source. > > > > This way you could have many light clients working off of a family node, > > and the peer services could be limited to some sort of “authenticated” > > peers. Perhaps this is better accomplished over the RPC interface in > Core, > > but the idea is to have some sort of peer service model between “full > > public” and “owner only”. This limits the amount of costs that can be > > properly externalized, without exposing risk of consensus capture by > > economically weighty institutions. > > The RPC interface in Bitcoin Core, and others, is not great for this > because it exposes a lot of functionality that isn't necessary and > introduces risks. For example the `gettxoutsetinfo` can start a very > intensive CPU and disk I/O task. There are several others, for example: > `stop`, `addnode`, `clearbanned`, `setban`, and etc. Furthermore reading > full raw blocks isn't very efficient with JSON. Electrum servers (e.g > electrs) for example read blocks from disk instead and use the RPC > interface to sync headers. Though, Electrum servers also have a risk of > DoS with addresses that have many transactions, see the `--txid-limit` > option [2]. > > [1]: > > https://github.com/bitcoin/bitcoin/blob/5b24f6084ede92d0f493ff416b4726245140b2c1/src/rpc/blockchain.cpp#L954-L956 > [2]: > > https://github.com/romanz/electrs/blob/f0a7a325af495ecbc152c0866550dc300011779b/src/query.rs#L284-L289 > > > ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] [Lightning-dev] On the scalability issues of onboarding millions of LN mobile clients
On 5/6/20 9:07 PM, Keagan McClelland wrote: > I think that one of the solutions here is to have light clients choose > their full node tethers explicitly. Even if you think it is unrealistic to > have everyone run their own node (fwiw, I don’t), there is still a trust > model where you can pick your trusted source. > > This way you could have many light clients working off of a family node, > and the peer services could be limited to some sort of “authenticated” > peers. Perhaps this is better accomplished over the RPC interface in Core, > but the idea is to have some sort of peer service model between “full > public” and “owner only”. This limits the amount of costs that can be > properly externalized, without exposing risk of consensus capture by > economically weighty institutions. The RPC interface in Bitcoin Core, and others, is not great for this because it exposes a lot of functionality that isn't necessary and introduces risks. For example the `gettxoutsetinfo` can start a very intensive CPU and disk I/O task. There are several others, for example: `stop`, `addnode`, `clearbanned`, `setban`, and etc. Furthermore reading full raw blocks isn't very efficient with JSON. Electrum servers (e.g electrs) for example read blocks from disk instead and use the RPC interface to sync headers. Though, Electrum servers also have a risk of DoS with addresses that have many transactions, see the `--txid-limit` option [2]. [1]: https://github.com/bitcoin/bitcoin/blob/5b24f6084ede92d0f493ff416b4726245140b2c1/src/rpc/blockchain.cpp#L954-L956 [2]: https://github.com/romanz/electrs/blob/f0a7a325af495ecbc152c0866550dc300011779b/src/query.rs#L284-L289 ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] [Lightning-dev] On the scalability issues of onboarding millions of LN mobile clients
On 5/8/20 1:01 PM, Keagan McClelland wrote: >> The RPC interface in Bitcoin Core, and others, is not great for this >> because it exposes a lot of functionality that isn't necessary and >> introduces risks. > This is actually somewhat my point. If the RPC interface was good for this > and *didn't* introduce risks, we could just use that and be done with it. > But I'm finding there are many use cases that you want to have low cost > ways to serve peer services to people whom you have given explicit > permission, but they shouldn't have full ability to administrate the node. > > Perhaps I wasn't explicit in my previous note but what I mean is that there > seems to be a demand for something *in between* a peer interface, and an > owner interface. I have little opinion as to whether this belongs in core > or not, I think there are much more experienced folks who can weight in on > that, but without something like this, you cannot limit your exposure for > serving something like bip157 filters without removing your own ability to > make use of some of those same services. An idea I was thinking about was having three ports for a full node: 1) Consensus bitcoin protocol. This is the existing peer-to-peer protocol without additional services. 2) Wallet services protocol. Adds additional functionality for wallets. For example bloom filtering, compact block filters, and potentially output and address indexes for electrum-like support. It's nearly identical to the consensus peer-to-peer protocol, supporting the same wire format. As it's on another port, various middleware could be added to support various authentication and transports. 3) Control interface. This is the existing JSON-RPC interface, without all wallet related RPC methods. ___ bitcoin-dev mailing list bitcoin-dev@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Re: [bitcoin-dev] Revault: a multi-party vault architecture
The fee bumping construction I described in the previous post is potentially vulnerable to transaction pinning. We shared a SINGLE | ANYONECANPAY signature for the first (and only) input of revaulting transactions to allow any party to append an input and an output in order to bump the transaction fees. An user would either append an input signed with ALL, or replace their SINGLE | ANYONECANPAY signature with one using ALL before broadcasting the transaction. This allowed one party to decrease the transaction fees down to the minimum relay fees, and possibly pin the transaction by spending their added single-pubkey output. We now exchange ALL | ANYONECANPAY signatures for revaulting transactions to restrict the creation of a new output only spendable by one party. The fee bumping is now done in two stages (to avoid consuming an entire utxo) : Unvaulting transaction | vault prevout | unvault output |-- \ \ Revaulting transaction \ --- | unvault prevout | new vault output | --- | fee bump prevout | / Single-party wallet transaction / -/ | wallet prevout | fee bump output|-- - | wallet change output | This construction isn't perfect as a malicious party could still pin its fee bumping transaction and prevent the other stakeholders from **immediatly** replacing this input, because of the second rule of BIP125 : > The replacement transaction may only include an unconfirmed input if that > input was included > in one of the original transactions. However, I think it's preferable as : - Depending on the unvault CSV, the honest party might pay a high fee to have the fee-bumping transaction confirm in one of the next two blocks, and then use this now confirmed output as an additional input of the revaulting transaction. - If the amount is consequent, the honest party may sacrifice an entire confirmed utxo from its wallet (effectively skipping the fee bumping transaction). - It's realistic to expect, for such an application, users' wallets to have a pool of confirmed utxo that might be sacrificed if the amount is consequent AND the CSV is so small (which is anyway a bad idea in the first place) that you are not sure to have the fee bumping transaction to be confirmed before its maturity, ). Thanks, Antoine / Darosior ‐‐‐ Original Message ‐‐‐ Le vendredi, avril 24, 2020 5:00 PM, darosior a écrit : > Hi all, > > Kevin Loaec and I have been working on a new multiparty vault architecture > and I think it reached the point where we’d welcome some feedback. > > Intended usage and limitations > > === > > The aim is to secure the shared storage of coins without relying on a trusted > third party and by disincentivizing theft attempts, while not restricting the > usage of the funds for day-to-day operations. > > Revault uses N-of-N multisigs and thus does not protect against intentional > locking of funds (such as refusal to sign, or key erasure). Therefore it > assumes its users (likely companies with already on-going agreements between > shareholders) to be able to solve intentional blockage outside the Bitcoin > network (such as through legal contracts). > > The actual architecture > > > > We called it revault as it relies on pre-signed and revocable (revaultable) > transactions. > The users pre-sign a transaction chain as the only used way to spend from a > vault output. > They would have signed a set of transactions to either cancel a spend attempt > or lock the funds for some time beforehand. The funds are always better > locked for a long time than stolen. > > The transactions > > > > The system is composed of mainly 6 transaction types (with N the number of > stakeholders) : > > - The “vault” transaction which pays to a N-of-N, by which funds are > received. > - The “emergency” transaction, which spends the vault output and pays to a > [here goes a > high value]-days timelocked N-of-N (with N differents but statics keys, > assumed to be physically stored in hard(/long) to access locations). > > - The “unvault” transaction, which spends the vault output and pays to > [either the vault’s N-of-N, or after X blocks
[bitcoin-dev] Compressed block headers
Hello list, I would like to propose a compressed block header scheme for IBD and block announcements. This proposal is derivative of previous proposals found on this list (see links in spec below) with some modifications and clarifications. The below specification (also found at https://github.com/willcl-ark/compressed-block-headers/blob/v1.0/compressed-block-headers.adoc ) details the compression recommended along with the generated bandwidth savings in the best-case scenario. I look forward to any feedback anyone has to offer on the specification itself, as well as any additions or objections to the motivation. Cheers, Will = Compressed block headers Will Clark v1.0, May 2020: :toc: preamble :toclevels: 4 This work is a derivation of these mailing list posts: 1. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-August/014876.html[bitcoin-dev: "Compressed" headers stream - 2017] (with resurrection https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015385.html[here]) 2. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-March/015851.html[bitcoin-dev: Optimized Header Sync] ''' == Motivation Block headers as exchanged by nodes over the p2p network are currently 81 bytes each. For low bandwidth nodes who are doing a headers-only sync, reducing the size of the headers can provide a significant bandwidth saving. Also, nodes can support more header-only peers for IBD and protection against eclipse attacks if header bandwidth is reduced. === Background Currently headers are sent over the p2p network as a vector of `block_headers`, which are composed of the following sized fields: [cols="<,>"] |=== |Field |Size |Version |4 bytes |Previous block hash |32 bytes |Merkle root hash|32 bytes |Time|4 bytes |nBits |4 bytes |nonce |4 bytes |txn_count |1 byte |*Total* |81 bytes |=== Some fields can be removed completely, others can be compressed under certain conditions. == Proposed specification === block_header2 data type The following table illustrates the proposed `block_header2` data type specification. [cols="<,>,>"] |=== |Field |Size |Compressed |Bitfield|1 byte | 1 byte |Version |4 bytes |0 \| 4 bytes |Previous block hash |32 bytes |0 \| 32 bytes |Merkle root hash|32 bytes |32 bytes |Time|4 bytes |2 \| 4 bytes |nBits |4 bytes |0 \| 4 bytes |nonce |4 bytes |4 bytes |*Total* |81 bytes |range: 39 - 81 bytes |=== This compression results in a maximum reduction from an 81 byte header to best-case 39 byte header. With 629,474 blocks in the current blockchain, a continuous header sync from genesis (requiring a single full 81 byte header followed by only compressed `block_header2`) has been tested to have its required bandwidth reduced from 50.98MB down to 25.86MB, a saving of 49%. Bitfield To make parsing of header messages easier and further increase header compression, a single byte bitfield was suggested by gmaxwell footnote:[https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015397.html]. We propose the following amended bitfield meanings (bits re-ordered to match `headers2` field order): [cols="<,<"] |=== |Bit |Meaning + field size to read |0 + 1 + 2|version: same as the last *distinct* value 1st ... 7th (0 byte field) or a new 32bit distinct value (4 byte field). |3 |prev_block_hash: is omitted (0 byte field) or included (32 byte field) |4 |timestamp: as small offset (2 byte field) or full (4 byte field). |5 |nbits: same as last header (0 byte field) or new (4 byte field). |6 |possibly to signal "more headers follow" to make the encoding self-delimiting. |7 |currently undefined |=== This bitfield adds 1 byte for every block in the chain, for a current total increase of 629,474B. Version In most cases the Version field will be identical to one referenced in one of the previous 7 unique versions, as indicated by bits 0,1,2 of the Bitfield. To block 629,474 there were 616,137 blocks whose version was in the previous 7 distinct versions, and only 13,338 blocks whose version was not, this includes any version bit manipulation done via overt ASIC boost. [cols=">,>,>,>"] |=== |Genesis to block |Current (B) |Compressed (B) |Saving (%) |629,474 |2,517,896 |53,352 |98 |=== Previous block hash The previous block hash will always be the `SHA256(SHA256())` so is redundant, presuming you have the previous header in the chain. [cols=">,>,>,>"] |=== |Genesis to block |Current (B) |Compressed (B) |Saving (%) |629,474 |20,143,168 |0 |100 |=== Time The timestamp (in seconds) is consensus bound, based both on the time in the previous header: `MAX_FUTURE_BLOCK_TIME = 2 * 60 * 60 = 7200`, and being greater than the `MedianTimePas
