Re: [bitcoin-dev] Fee estimates and RBF

[ZmnSCPxj via bitcoin-dev]

Good morning Prayank,

>  But it will involve lot of exception handling.


Yes, that is precisely the problem here.

If you select a fixed feerate and then just broadcast-and-forget, you have no 
real exceptions you have to handle --- but that means not using RBF at all.


Testing the handling of reorgs in particular is important, as a reorg might use 
an older version of an RBFed transaction rather than a newer version.
This also implies that further follow-up transactions might need to be 
recreated in such a case.

As this is financial code, we need a lot of testing, and code that has a lot of 
branches due to having to handle a lot of possible exceptions and so forth is a 
headache to completely cover in testing.


C-lightning supposedly supports RBF, in the sense that every transaction it 
makes always signals RBF, but I am almost certain there are edge cases where it 
might mishandle a replaced transaction and lose track of onchain funds, and it 
is difficult to support both "we can spend unconfirmed change outputs" (a very 
common feature of nearly every onchain wallet) with "we can change the feerate 
of unconfirmed transactions" (which changes the txid and therefore the UTXO id 
of the change output spent by use of the previous feature).

Regards,
ZmnSCPxj
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Fee estimates and RBF

[Prayank via bitcoin-dev]

I have shared response by Jeremy and ZmnSCPxj in an answer to 
https://bitcoin.stackexchange.com/questions/105860/what-are-we-trying-to-predict-in-fee-estimation-and-why

Also find the recent CVE related to RBF by Antoine Riard and implementation of 
RBF in Bitcoin Core compared to btcd interesting. Even though I am not sure why 
inherited signalling is not implemented in Bitcoin Core.
RBF, CPFP and their combinations are something that is less explored IMO. For 
example: I had discussed one usecase of CPFP with Harding in IRC once in which 
a project uses maker-taker model. Maker broadcasts transaction with 1 sat/vByte 
and taker has to confirm this transaction by creating a child transaction with 
an effective fee rate according to mempool stats. Basically, the idea of 
receiver paying for the transaction instead of sender. But it will involve lot 
of exception handling.
-- 
Prayank___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Fee estimates and RBF

[Prayank via bitcoin-dev]

Good morning ZmnSCPxj,

Thanks for your response. I agree there are few exceptions: 

1.Unconfirmed output can be spent resulting in conflict with RBF
2.Race condition and mining pool may include old transaction with low fee
I am trying few things related to RBF and handling such exceptions, will share 
if I find anything interesting.
-- 
Prayank

May 3, 2021, 09:32 by zmnsc...@protonmail.com:

> Good morning Prayank,
>
> I believe a "true" full-RBF wallet should be what every onchain wallet 
> aspires to.
>
> However, I think a lot of the effort necessary here has to do with sheer 
> engineering issues.
>
> For example, if you think "RBF does not exist", you can do things like:
>
> * Spend an unconfirmed input from a third party.
>  * This is not actually safe since an unconfirmed tx might have a conflicting 
> transaction get confirmed, but a lot of onchain wallets support this for 
> non-RBF unconfirmed inputs because 99.9% of the time this never happens.
> * When you spend from a (confirmed or unconfirmed) input, delete it from your 
> db forever (because you do not have to worry about alternate transactions 
> spending the same input).
>  * This simplifies db design, you do not have to keep track of states like 
> "has been spent but tx is not confirmed yet", "has two different alternate 
> transactions spending it that have not confirmed", "is on a transaction that 
> is not confirmed and therefore this input might disappear completely" etc.
>
> In particular, if we want a "true" full-RBF wallet:
>
> * Suppose the user wants to spend some amount to address A.
>  * The user imposes a limit on up to how much to spend on fees to have this 
> spend happen.
> * The wallet optimistically creates a low-fee send transaction.
> * After some time, the wallet bumps up the fee by creating a new transaction.
>  * The wallet keeps bumping up, up to the designated limit, the longer the 
> transaction is not confirmed.
>
> Of note is that there is a *race condition* in the above case.
> When the wallet is bumping up and constructing a new transaction with higher 
> fee, a miner could find a new block that has the old transaction with lower 
> fee.
>
> Now consider the subsequent user story.
>
> * After some time, the user wants to spend another amount to address B.
>  * Again the user imposes a limit on how much to spend on fees to have this 
> spend happen.
> * The wallet RBFs the existing transaction to include the spend to address B.
>
> Again, a race condition can occur --- while the wallet is feebumping a new 
> transaction that includes the new output, a random miner can find a new block 
> that includes the old transaction.
>
> Thus, the wallet really needs to keep track of any "pending spends" and 
> correlate them with actual transactions.
>
> Further, of course it is convenient to be able to spend money even while it 
> is unconfirmed.
> But the sender of the unconfirmed input might be using the same software as 
> this wallet as well, meaning that the actual transaction output might change 
> as the original spender keeps fee-bumping it over time.
>
> I confess I have not been thinking of this as well as I should have.
>
> Regards,
> ZmnSCPxj
>___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Fee estimates and RBF

[ZmnSCPxj via bitcoin-dev]

Good morning Prayank,

I believe a "true" full-RBF wallet should be what every onchain wallet aspires 
to.

However, I think a lot of the effort necessary here has to do with sheer 
engineering issues.

For example, if you think "RBF does not exist", you can do things like:

* Spend an unconfirmed input from a third party.
  * This is not actually safe since an unconfirmed tx might have a conflicting 
transaction get confirmed, but a lot of onchain wallets support this for 
non-RBF unconfirmed inputs because 99.9% of the time this never happens.
* When you spend from a (confirmed or unconfirmed) input, delete it from your 
db forever (because you do not have to worry about alternate transactions 
spending the same input).
  * This simplifies db design, you do not have to keep track of states like 
"has been spent but tx is not confirmed yet", "has two different alternate 
transactions spending it that have not confirmed", "is on a transaction that is 
not confirmed and therefore this input might disappear completely" etc.

In particular, if we want a "true" full-RBF wallet:

* Suppose the user wants to spend some amount to address A.
  * The user imposes a limit on up to how much to spend on fees to have this 
spend happen.
* The wallet optimistically creates a low-fee send transaction.
* After some time, the wallet bumps up the fee by creating a new transaction.
  * The wallet keeps bumping up, up to the designated limit, the longer the 
transaction is not confirmed.

Of note is that there is a *race condition* in the above case.
When the wallet is bumping up and constructing a new transaction with higher 
fee, a miner could find a new block that has the old transaction with lower fee.

Now consider the subsequent user story.

* After some time, the user wants to spend another amount to address B.
  * Again the user imposes a limit on how much to spend on fees to have this 
spend happen.
* The wallet RBFs the existing transaction to include the spend to address B.

Again, a race condition can occur --- while the wallet is feebumping a new 
transaction that includes the new output, a random miner can find a new block 
that includes the old transaction.

Thus, the wallet really needs to keep track of any "pending spends" and 
correlate them with actual transactions.

Further, of course it is convenient to be able to spend money even while it is 
unconfirmed.
But the sender of the unconfirmed input might be using the same software as 
this wallet as well, meaning that the actual transaction output might change as 
the original spender keeps fee-bumping it over time.

I confess I have not been thinking of this as well as I should have.

Regards,
ZmnSCPxj
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Fee estimates and RBF

[Prayank via bitcoin-dev]

Thanks Jeremy for sharing this link: 
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-September/018168.html

Trying to understand everything mentioned and "fee-only" wallet sounds 
interesting.
-- 
Prayank

May 1, 2021, 05:41 by jlru...@mit.edu:

> Hi Prayank,
>
> Very glad to hear you are weathering the storm OK, wishing you and yours 
> safety in the weeks ahead.
>
> I think you'll be interested to see > 
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-September/018168.html>
>   especially with regards to background services.
>
> In the long term, this can be particularly useful since you can use a 
> separate fee-only wallet to arrange the bumps if your main wallet keys are 
> offline, and you do not disturb any of the on-chain dependants.
>
> It can also be much cheaper to use this mechanism than actual RBF because you 
> are not subject to the feerate improvement rule, which forces you to pay for 
> the bump on all tx data.
>
> I would be very interested to see a system whereby you can make a market 
> (maybe via LN) for someone to get your TX included (using oracle network OK) 
> by a particular date. Then you can abstract the whole system a lot better, so 
> that you can fee bump without having to create any direct on chain traffic, 
> and a sponsor vector can be offered across a number of txns by a service 
> provider.
>
> Cheers,
>
> Jeremy
> --
> @JeremyRubin  
> 
>
>
> On Fri, Apr 30, 2021 at 2:40 PM Prayank via bitcoin-dev <> 
> bitcoin-dev@lists.linuxfoundation.org> > wrote:
>
>> Hello World, 
>>
>> I hope everyone is doing okay. Things are not good in India and even I was 
>> tested covid positive few days back. Recovered and feeling better now. 
>> Hoping everything gets back to normal soon.
>>
>> There are different estimations used in wallets, explorers and other Bitcoin 
>> projects. For example: `estimatesmartfee` in Bitcoin Core (One of the 
>> implementation for Bitcoin which is used more but not official as there is 
>> nothing official in Bitcoin).  Are different estimations misleading and 
>> affect the way fees are used in Bitcoin transactions? Will it be better if 
>> we just share mempool stats and user can decide the fee rate accordingly?
>>
>> If I compare this with BTCUSD orderbook on any exchange, are we trying to 
>> estimate at what price buy order will get filled in certain time? Does that 
>> make sense?
>>
>> Mempool Stats: >> https://i.imgur.com/r4XKk2p.png
>> BTCUSD Orderbook: >> https://i.imgur.com/ylGVHJB.png
>>
>> I consider it misleading because lot of users think a transaction with fee 
>> rate 1-5 sat/vByte will be included in 1 week or maybe a transaction with X 
>> sat/VByte will be included in Y time which is not true. Users can decide the 
>> fee rate and can do bidding, transaction will be included based on demand, 
>> supply and miners.
>>
>> Will it be better if the wallets used this approach?
>> 1.Show mempool stats
>> 2.Leave the fee rate for user to decide
>> 3.RBF every transaction and follow different algorithms for automated bidding
>>
>> A basic algorithm for automated bidding can be: >> 
>> https://i.stack.imgur.com/1SlPv.png
>>
>> Such RBF algos can be helpful for users when Bitcoin wallets are open in 
>> background. Maybe it will work better for mobile wallets in which you can 
>> see a notification every time transaction is replaced with a new fee rate 
>> automatically.
>>
>> I wanted to know what others think about this approach of creating and using 
>> different RBF algos instead of predicting something that is difficult or 
>> doesn't make sense. Also if there was a way we could achieve this even if 
>> the user goes offline. For example: Alice broadcasts Tx1 with 1 sat/vByte, 
>> its replaced with Tx2 (2 sat/vByte) after 2 blocks because Tx1 was not 
>> confirmed. Alice decides to shut down her system or switch off mobile or 
>> mobile data. Tx2 is still not confirmed after another 2 blocks but it has 
>> some information as one OP_RETURN output which is used by Bitcoin nodes that 
>> see this transaction in the mempool. Bob's node use this information to 
>> replace the transaction with Tx3 and use fee rate 3 sat/vByte.
>>
>> --
>> Prayank
>>
>> ___
>>  bitcoin-dev mailing list
>>  >> bitcoin-dev@lists.linuxfoundation.org
>>  >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>

___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Fee estimates and RBF

[Jeremy via bitcoin-dev]

Hi Prayank,

Very glad to hear you are weathering the storm OK, wishing you and yours
safety in the weeks ahead.

I think you'll be interested to see
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-September/018168.html
especially with regards to background services.

In the long term, this can be particularly useful since you can use a
separate fee-only wallet to arrange the bumps if your main wallet keys are
offline, and you do not disturb any of the on-chain dependants.

It can also be much cheaper to use this mechanism than actual RBF because
you are not subject to the feerate improvement rule, which forces you to
pay for the bump on all tx data.

I would be very interested to see a system whereby you can make a market
(maybe via LN) for someone to get your TX included (using oracle network
OK) by a particular date. Then you can abstract the whole system a lot
better, so that you can fee bump without having to create any direct on
chain traffic, and a sponsor vector can be offered across a number of txns
by a service provider.

Cheers,

Jeremy
--
@JeremyRubin 



On Fri, Apr 30, 2021 at 2:40 PM Prayank via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hello World,
>
> I hope everyone is doing okay. Things are not good in India and even I was
> tested covid positive few days back. Recovered and feeling better now.
> Hoping everything gets back to normal soon.
>
> There are different estimations used in wallets, explorers and other
> Bitcoin projects. For example: `estimatesmartfee` in Bitcoin Core (One of
> the implementation for Bitcoin which is used more but not official as there
> is nothing official in Bitcoin).  Are different estimations misleading and
> affect the way fees are used in Bitcoin transactions? Will it be better if
> we just share mempool stats and user can decide the fee rate accordingly?
>
> If I compare this with BTCUSD orderbook on any exchange, are we trying to
> estimate at what price buy order will get filled in certain time? Does that
> make sense?
>
> Mempool Stats: https://i.imgur.com/r4XKk2p.png
> BTCUSD Orderbook: https://i.imgur.com/ylGVHJB.png
>
> I consider it misleading because lot of users think a transaction with fee
> rate 1-5 sat/vByte will be included in 1 week or maybe a transaction with X
> sat/VByte will be included in Y time which is not true. Users can decide
> the fee rate and can do bidding, transaction will be included based on
> demand, supply and miners.
>
> Will it be better if the wallets used this approach?
> 1.Show mempool stats
> 2.Leave the fee rate for user to decide
> 3.RBF every transaction and follow different algorithms for automated
> bidding
>
> A basic algorithm for automated bidding can be:
> https://i.stack.imgur.com/1SlPv.png
>
> Such RBF algos can be helpful for users when Bitcoin wallets are open in
> background. Maybe it will work better for mobile wallets in which you can
> see a notification every time transaction is replaced with a new fee rate
> automatically.
>
> I wanted to know what others think about this approach of creating and
> using different RBF algos instead of predicting something that is difficult
> or doesn't make sense. Also if there was a way we could achieve this even
> if the user goes offline. For example: Alice broadcasts Tx1 with 1
> sat/vByte, its replaced with Tx2 (2 sat/vByte) after 2 blocks because Tx1
> was not confirmed. Alice decides to shut down her system or switch off
> mobile or mobile data. Tx2 is still not confirmed after another 2 blocks
> but it has some information as one OP_RETURN output which is used by
> Bitcoin nodes that see this transaction in the mempool. Bob's node use this
> information to replace the transaction with Tx3 and use fee rate 3
> sat/vByte.
>
> --
> Prayank
>
> ___
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev