Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

Good afternoon Alan,


It is stated in the proposal that it is intended for blocks to be validated as 
the output of the priority method, to ensure that they conform. Unfortunately, 
the math necessary for this sort of statistical function is outside the scope 
of my formal education and I will need to rely on someone to develop what is 
necessary. If it does turn out that this is not ultimately possible then I 
suppose at that stage the proposal would need to be abandoned since I agree - 
validation must be necessary. Blocks created with cheating should be too 
unlikely.


>All block created with dynamic size should be verified to ensure
conformity to a probability distribution curve resulting from the
priority method. Since the input is a probability, the output should
conform to a probability distribution.


Regards,

Damian Williamson


From: Alan Evans <thealanev...@gmail.com>
Sent: Sunday, 21 January 2018 1:46:41 AM
To: Damian Williamson; Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

I don't see any modifications to the proposal that addresses the issue that 
miners will always be free to choose their own priority that a few people 
brought up before.

I understand you think it's in the miners best long-term interest to follow 
these rules, but even if a miner agrees with you, if that miner thinks the 
other miners are following the fee curve, they will know it makes no overall 
difference if they cheat (you can't prove how long a miner has had a 
transaction in their mempool).

The opportunity to cheat, the anonymity of mining, the low negative effect of a 
single cheating instance, all combined with a financial incentive to cheat 
means that cheating will be rife.


On Sat, Jan 20, 2018 at 8:04 AM, Damian Williamson via bitcoin-dev 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
 wrote:

Tried a different approach for the curves, would appreciate it if someone has 
the energy to work on this and help me to resolve it a bit more scientifically:


p(tx) = (fx - (fl - 0.0001)) / (fh - (fl - 0.0001))) * 100) + 1) ^ 
y) + (wx - 0.9) / ((86400 * n) - 0.9)) * 100) + 1) ^ y)

p is the calculated priority number for tx the specific valid transaction.
fx is the fee in BTC/KB for the specific transaction.
fl is the lowest valid fee in BTC/KB currently in the nodes mempool.
fh is the highest valid fee in BTC/KB currently in the nodes mempool.
wx is the current wait in seconds for tx the specific valid transaction.
n is the number of days maximum wait consensus value.
y can be 10 or, y can be a further developed to be a formula based on the 
number of required inclusions to vary the steepness of the curve as the mempool 
size varies.

In the next step, the random value must be:
if random(101^y) < p then transaction is included;

Regards,
Damian Williamson



From: Damian Williamson <willt...@live.com.au<mailto:willt...@live.com.au>>
Sent: Saturday, 20 January 2018 10:25:43 AM
To: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


An example curve:

The curve curently described here is ineffective at acheiving the requirements. 
It seems to be not nearly steep enough resulting in too many inclusions (as it 
happens, this may not metter - needs further evaluation) and, the lower end 
values seem problematically small but, results in a number between 100 for the 
highest fee BTC/KB and a small fraction of 1 for the lowest. This math needs to 
be improved.


pf(tx) = sin2((fx-(fl-0.0001))/(fh-(fl-0.0001))*1.570796326795)*100


pf is the calculated priority number for the fee for tx the specifc valid 
transaction.
fx is the fee in BTC/KB for the specific transaction.
fl is the lowest valid fee in BTC/KB currently in the nodes mempool.
fh is the highest valid fee in BTC/KB currently in the nodes mempool.


From: 
bitcoin-dev-boun...@lists.linuxfoundation.org<mailto:bitcoin-dev-boun...@lists.linuxfoundation.org>
 
<bitcoin-dev-boun...@lists.linuxfoundation.org<mailto:bitcoin-dev-boun...@lists.linuxfoundation.org>>
 on behalf of Damian Williamson via bitcoin-dev 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
Sent: Thursday, 4 January 2018 8:01:10 PM
To: Bitcoin Protocol Discussion
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


This proposal has a new update, mostly minor edits. Additionally, I had a logic 
flaw in the hard fork / soft fork declaration statement. The specific terms of 
the CC-BY-SA-4.0 licence the document is published under have now been updated 
to include 

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Alan Evans via bitcoin-dev]

I don't see any modifications to the proposal that addresses the issue that
miners will always be free to choose their own priority that a few people
brought up before.

I understand you think it's in the miners best long-term interest to follow
these rules, but even if a miner agrees with you, if that miner thinks the
other miners are following the fee curve, they will know it makes no
overall difference if they cheat (you can't prove how long a miner has had
a transaction in their mempool).

The opportunity to cheat, the anonymity of mining, the low negative effect
of a single cheating instance, all combined with a financial incentive to
cheat means that cheating will be rife.


On Sat, Jan 20, 2018 at 8:04 AM, Damian Williamson via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Tried a different approach for the curves, would appreciate it if someone
> has the energy to work on this and help me to resolve it a bit more
> scientifically:
>
>
> p(tx) = (fx - (fl - 0.0001)) / (fh - (fl - 0.0001))) * 100) +
> 1) ^ y) + (wx - 0.9) / ((86400 * n) - 0.9)) * 100) + 1) ^ y)
>
> p is the calculated priority number for tx the specific valid transaction.
> fx is the fee in BTC/KB for the specific transaction.
> fl is the lowest valid fee in BTC/KB currently in the nodes mempool.
> fh is the highest valid fee in BTC/KB currently in the nodes mempool.
> wx is the current wait in seconds for tx the specific valid transaction.
> n is the number of days maximum wait consensus value.
> y can be 10 or, y can be a further developed to be a formula based on the
> number of required inclusions to vary the steepness of the curve as the
> mempool size varies.
>
> In the next step, the random value must be:
> if random(101^y) < p then transaction is included;
>
> Regards,
> Damian Williamson
>
> --
> *From:* Damian Williamson <willt...@live.com.au>
> *Sent:* Saturday, 20 January 2018 10:25:43 AM
> *To:* Bitcoin Protocol Discussion
> *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
> Transaction Priority For Ordering Transactions In Blocks
>
>
> An example curve:
>
> The curve curently described here is ineffective at acheiving the
> requirements. It seems to be not nearly steep enough resulting in too many
> inclusions (as it happens, this may not metter - needs further evaluation)
> and, the lower end values seem problematically small but, results in a
> number between 100 for the highest fee BTC/KB and a small fraction of 1 for
> the lowest. This math needs to be improved.
>
>
> pf(tx) = sin2((fx-(fl-0.0001))/(fh-(fl-0.0001))*1.
> 570796326795)*100
>
>
> pf is the calculated priority number for the fee for tx the specifc valid
> transaction.
> fx is the fee in BTC/KB for the specific transaction.
> fl is the lowest valid fee in BTC/KB currently in the nodes mempool.
> fh is the highest valid fee in BTC/KB currently in the nodes mempool.
>
> --
> *From:* bitcoin-dev-boun...@lists.linuxfoundation.org <
> bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian
> Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
> *Sent:* Thursday, 4 January 2018 8:01:10 PM
> *To:* Bitcoin Protocol Discussion
> *Subject:* [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
> Transaction Priority For Ordering Transactions In Blocks
>
>
> This proposal has a new update, mostly minor edits. Additionally, I had a
> logic flaw in the hard fork / soft fork declaration statement. The specific
> terms of the CC-BY-SA-4.0 licence the document is published under have
> now been updated to include additional permissions available under the MIT
> licence.
>
>
> Recently, on Twitter:
>
> I am looking for a capable analyst/programmer to work on a BIP proposal as
> co-author. Will need to format several Full BIP's per these BIP process
> requirements: ( https://github.com/bitcoin/bips/blob/master/bip-0002.
> mediawiki ) from a BIP Proposal, being two initially for non-consensus
> full-interoperable pre-rollout on peer service layer & API/RPC layer and, a
> reference implementation for Bitcoin Core per: (
> https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md ).
> Interested parties please reply via this list thread: (
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/
> 2017-December/015485.html ) #Bitcoin #BIP
>
>
> Regards,
>
> Damian Williamson
>
>
> --
> *From:* bitcoin-dev-boun...@lists.linuxfoundation.org <
> bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian
> Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
> *Se

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

Tried a different approach for the curves, would appreciate it if someone has 
the energy to work on this and help me to resolve it a bit more scientifically:


p(tx) = (fx - (fl - 0.0001)) / (fh - (fl - 0.0001))) * 100) + 1) ^ 
y) + (wx - 0.9) / ((86400 * n) - 0.9)) * 100) + 1) ^ y)

p is the calculated priority number for tx the specific valid transaction.
fx is the fee in BTC/KB for the specific transaction.
fl is the lowest valid fee in BTC/KB currently in the nodes mempool.
fh is the highest valid fee in BTC/KB currently in the nodes mempool.
wx is the current wait in seconds for tx the specific valid transaction.
n is the number of days maximum wait consensus value.
y can be 10 or, y can be a further developed to be a formula based on the 
number of required inclusions to vary the steepness of the curve as the mempool 
size varies.

In the next step, the random value must be:
if random(101^y) < p then transaction is included;

Regards,
Damian Williamson



From: Damian Williamson <willt...@live.com.au>
Sent: Saturday, 20 January 2018 10:25:43 AM
To: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


An example curve:

The curve curently described here is ineffective at acheiving the requirements. 
It seems to be not nearly steep enough resulting in too many inclusions (as it 
happens, this may not metter - needs further evaluation) and, the lower end 
values seem problematically small but, results in a number between 100 for the 
highest fee BTC/KB and a small fraction of 1 for the lowest. This math needs to 
be improved.


pf(tx) = sin2((fx-(fl-0.0001))/(fh-(fl-0.0001))*1.570796326795)*100


pf is the calculated priority number for the fee for tx the specifc valid 
transaction.
fx is the fee in BTC/KB for the specific transaction.
fl is the lowest valid fee in BTC/KB currently in the nodes mempool.
fh is the highest valid fee in BTC/KB currently in the nodes mempool.


From: bitcoin-dev-boun...@lists.linuxfoundation.org 
<bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian Williamson 
via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Thursday, 4 January 2018 8:01:10 PM
To: Bitcoin Protocol Discussion
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


This proposal has a new update, mostly minor edits. Additionally, I had a logic 
flaw in the hard fork / soft fork declaration statement. The specific terms of 
the CC-BY-SA-4.0 licence the document is published under have now been updated 
to include additional permissions available under the MIT licence.


Recently, on Twitter:

I am looking for a capable analyst/programmer to work on a BIP proposal as 
co-author. Will need to format several Full BIP's per these BIP process 
requirements: ( https://github.com/bitcoin/bips/blob/master/bip-0002.mediawiki 
) from a BIP Proposal, being two initially for non-consensus full-interoperable 
pre-rollout on peer service layer & API/RPC layer and, a reference 
implementation for Bitcoin Core per: ( 
https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md ). Interested 
parties please reply via this list thread: ( 
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015485.html
 ) #Bitcoin #BIP


Regards,

Damian Williamson



From: bitcoin-dev-boun...@lists.linuxfoundation.org 
<bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian Williamson 
via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Monday, 1 January 2018 10:04 PM
To: bitcoin-dev@lists.linuxfoundation.org
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Happy New Year all.

This proposal has been further amended with several minor changes and a
few additions.

I believe that all known issues raised so far have been sufficiently
addressed. Either that or, I still have more work to do.

## BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For
Ordering Transactions In Blocks

Schema:
##
Document: BIP Proposal
Title: UTPFOTIB - Use Transaction Priority For Ordering Transactions In
Blocks
Published: 26-12-2017
Revised: 01-01-2018
Author: Damian Williamson <willt...@live.com.au>
Licence: Creative Commons Attribution-ShareAlike 4.0 International
License.
URL: http://thekingjameshrmh.tumblr.com/post/168948530950/bip-proposal-
utpfotib-use-transaction-priority-for-order
##

### 1. Abstract

This document proposes to address the issue of transactional
reliability in Bitcoin, where valid transactions may be stuck in the
transaction pool for extended periods or never confirm.

There are two key issues to be resolved to achieve this:

1.  The current transact

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

An example curve:

The curve curently described here is ineffective at acheiving the requirements. 
It seems to be not nearly steep enough resulting in too many inclusions (as it 
happens, this may not metter - needs further evaluation) and, the lower end 
values seem problematically small but, results in a number between 100 for the 
highest fee BTC/KB and a small fraction of 1 for the lowest. This math needs to 
be improved.


pf(tx) = sin2((fx-(fl-0.0001))/(fh-(fl-0.0001))*1.570796326795)*100


pf is the calculated priority number for the fee for tx the specifc valid 
transaction.
fx is the fee in BTC/KB for the specific transaction.
fl is the lowest valid fee in BTC/KB currently in the nodes mempool.
fh is the highest valid fee in BTC/KB currently in the nodes mempool.


From: bitcoin-dev-boun...@lists.linuxfoundation.org 
<bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian Williamson 
via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Thursday, 4 January 2018 8:01:10 PM
To: Bitcoin Protocol Discussion
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


This proposal has a new update, mostly minor edits. Additionally, I had a logic 
flaw in the hard fork / soft fork declaration statement. The specific terms of 
the CC-BY-SA-4.0 licence the document is published under have now been updated 
to include additional permissions available under the MIT licence.


Recently, on Twitter:

I am looking for a capable analyst/programmer to work on a BIP proposal as 
co-author. Will need to format several Full BIP's per these BIP process 
requirements: ( https://github.com/bitcoin/bips/blob/master/bip-0002.mediawiki 
) from a BIP Proposal, being two initially for non-consensus full-interoperable 
pre-rollout on peer service layer & API/RPC layer and, a reference 
implementation for Bitcoin Core per: ( 
https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md ). Interested 
parties please reply via this list thread: ( 
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015485.html
 ) #Bitcoin #BIP


Regards,

Damian Williamson



From: bitcoin-dev-boun...@lists.linuxfoundation.org 
<bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian Williamson 
via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Monday, 1 January 2018 10:04 PM
To: bitcoin-dev@lists.linuxfoundation.org
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Happy New Year all.

This proposal has been further amended with several minor changes and a
few additions.

I believe that all known issues raised so far have been sufficiently
addressed. Either that or, I still have more work to do.

## BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For
Ordering Transactions In Blocks

Schema:
##
Document: BIP Proposal
Title: UTPFOTIB - Use Transaction Priority For Ordering Transactions In
Blocks
Published: 26-12-2017
Revised: 01-01-2018
Author: Damian Williamson <willt...@live.com.au>
Licence: Creative Commons Attribution-ShareAlike 4.0 International
License.
URL: http://thekingjameshrmh.tumblr.com/post/168948530950/bip-proposal-
utpfotib-use-transaction-priority-for-order
##

### 1. Abstract

This document proposes to address the issue of transactional
reliability in Bitcoin, where valid transactions may be stuck in the
transaction pool for extended periods or never confirm.

There are two key issues to be resolved to achieve this:

1.  The current transaction bandwidth limit.
2.  The current ad-hoc methods of including transactions in blocks
resulting in variable and confusing confirmation times for valid
transactions, including transactions with a valid fee that may never
confirm.

It is important with any change to protect the value of fees as these
will eventually be the only payment that miners receive. Rather than an
auction model for limited bandwidth, the proposal results in a fee for
priority service auction model.

It would not be true to suggest that all feedback received so far has
been entirely positive although, most of it has been constructive.

The previous threads for this proposal are available here:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/s
ubject.html

In all parts of this proposal, references to a transaction, a valid
transaction, a transaction with a valid fee, a valid fee, etc. is
defined as any transaction that is otherwise valid with a fee of at
least 0.1000 BTC/KB as defined as the dust level, interpreting from
Bitcoin Core GUI. Transactions with a fee lower than this rate are
considered dust.

In all parts of this proposal, dust and zero-fee transactions are
always ignored and/or excluded unless specifically mentioned.

It is generally assumed that 

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

This proposal has a new update, mostly minor edits. Additionally, I had a logic 
flaw in the hard fork / soft fork declaration statement. The specific terms of 
the CC-BY-SA-4.0 licence the document is published under have now been updated 
to include additional permissions available under the MIT licence.


Recently, on Twitter:

I am looking for a capable analyst/programmer to work on a BIP proposal as 
co-author. Will need to format several Full BIP's per these BIP process 
requirements: ( https://github.com/bitcoin/bips/blob/master/bip-0002.mediawiki 
) from a BIP Proposal, being two initially for non-consensus full-interoperable 
pre-rollout on peer service layer & API/RPC layer and, a reference 
implementation for Bitcoin Core per: ( 
https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md ). Interested 
parties please reply via this list thread: ( 
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015485.html
 ) #Bitcoin #BIP


Regards,

Damian Williamson



From: bitcoin-dev-boun...@lists.linuxfoundation.org 
<bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian Williamson 
via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Monday, 1 January 2018 10:04 PM
To: bitcoin-dev@lists.linuxfoundation.org
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Happy New Year all.

This proposal has been further amended with several minor changes and a
few additions.

I believe that all known issues raised so far have been sufficiently
addressed. Either that or, I still have more work to do.

## BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For
Ordering Transactions In Blocks

Schema:
##
Document: BIP Proposal
Title: UTPFOTIB - Use Transaction Priority For Ordering Transactions In
Blocks
Published: 26-12-2017
Revised: 01-01-2018
Author: Damian Williamson <willt...@live.com.au>
Licence: Creative Commons Attribution-ShareAlike 4.0 International
License.
URL: http://thekingjameshrmh.tumblr.com/post/168948530950/bip-proposal-
utpfotib-use-transaction-priority-for-order
##

### 1. Abstract

This document proposes to address the issue of transactional
reliability in Bitcoin, where valid transactions may be stuck in the
transaction pool for extended periods or never confirm.

There are two key issues to be resolved to achieve this:

1.  The current transaction bandwidth limit.
2.  The current ad-hoc methods of including transactions in blocks
resulting in variable and confusing confirmation times for valid
transactions, including transactions with a valid fee that may never
confirm.

It is important with any change to protect the value of fees as these
will eventually be the only payment that miners receive. Rather than an
auction model for limited bandwidth, the proposal results in a fee for
priority service auction model.

It would not be true to suggest that all feedback received so far has
been entirely positive although, most of it has been constructive.

The previous threads for this proposal are available here:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/s
ubject.html

In all parts of this proposal, references to a transaction, a valid
transaction, a transaction with a valid fee, a valid fee, etc. is
defined as any transaction that is otherwise valid with a fee of at
least 0.1000 BTC/KB as defined as the dust level, interpreting from
Bitcoin Core GUI. Transactions with a fee lower than this rate are
considered dust.

In all parts of this proposal, dust and zero-fee transactions are
always ignored and/or excluded unless specifically mentioned.

It is generally assumed that miners currently prefer to include
transactions with higher fees.

### 2. The need for this proposal

We all must learn to admit that transaction bandwidth is still lurking
as a serious issue for the operation, reliability, safety, consumer
acceptance, uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day
target confirmation from the fee recommendation. That transaction has
still not confirmed after now more than six days - even waiting twice
as long seems quite reasonable to me (note for accuracy: it did
eventually confirm). That transaction is a valid transaction; it is not
rubbish, junk or, spam. Under the current model with transaction
bandwidth limitation, the longer a transaction waits, the less likely
it is ever to confirm due to rising transaction numbers and being
pushed back by transactions with rising fees.

I argue that no transactions with fees above the dust level are rubbish
or junk, only some zero fee transactions might be spam. Having an ever-
increasing number of valid transactions that do not confirm as more new
transactions with higher fees are created is the opposite of operating
a robust, reliable transaction system.

While the miners have 

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

Happy New Year all.

This proposal has been further amended with several minor changes and a
few additions.

I believe that all known issues raised so far have been sufficiently
addressed. Either that or, I still have more work to do.

## BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For
Ordering Transactions In Blocks

Schema:  
##  
Document: BIP Proposal  
Title: UTPFOTIB - Use Transaction Priority For Ordering Transactions In
Blocks  
Published: 26-12-2017  
Revised: 01-01-2018  
Author: Damian Williamson   
Licence: Creative Commons Attribution-ShareAlike 4.0 International
License.  
URL: http://thekingjameshrmh.tumblr.com/post/168948530950/bip-proposal-
utpfotib-use-transaction-priority-for-order  
##

### 1. Abstract

This document proposes to address the issue of transactional
reliability in Bitcoin, where valid transactions may be stuck in the
transaction pool for extended periods or never confirm.

There are two key issues to be resolved to achieve this:

1.  The current transaction bandwidth limit.
2.  The current ad-hoc methods of including transactions in blocks
resulting in variable and confusing confirmation times for valid
transactions, including transactions with a valid fee that may never
confirm.

It is important with any change to protect the value of fees as these
will eventually be the only payment that miners receive. Rather than an
auction model for limited bandwidth, the proposal results in a fee for
priority service auction model.

It would not be true to suggest that all feedback received so far has
been entirely positive although, most of it has been constructive.

The previous threads for this proposal are available here:  
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/s
ubject.html

In all parts of this proposal, references to a transaction, a valid
transaction, a transaction with a valid fee, a valid fee, etc. is
defined as any transaction that is otherwise valid with a fee of at
least 0.1000 BTC/KB as defined as the dust level, interpreting from
Bitcoin Core GUI. Transactions with a fee lower than this rate are
considered dust.

In all parts of this proposal, dust and zero-fee transactions are
always ignored and/or excluded unless specifically mentioned.

It is generally assumed that miners currently prefer to include
transactions with higher fees.

### 2. The need for this proposal

We all must learn to admit that transaction bandwidth is still lurking
as a serious issue for the operation, reliability, safety, consumer
acceptance, uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day
target confirmation from the fee recommendation. That transaction has
still not confirmed after now more than six days - even waiting twice
as long seems quite reasonable to me (note for accuracy: it did
eventually confirm). That transaction is a valid transaction; it is not
rubbish, junk or, spam. Under the current model with transaction
bandwidth limitation, the longer a transaction waits, the less likely
it is ever to confirm due to rising transaction numbers and being
pushed back by transactions with rising fees.

I argue that no transactions with fees above the dust level are rubbish
or junk, only some zero fee transactions might be spam. Having an ever-
increasing number of valid transactions that do not confirm as more new
transactions with higher fees are created is the opposite of operating
a robust, reliable transaction system.

While the miners have discovered a gold mine, it is the service they
provide that is valuable. If the service is unreliable they are not
worth the gold that they mine. This is reflected in the value of
Bitcoin.

Business cannot operate with a model where transactions may or may not
confirm. Even a business choosing a modest fee has no guarantee that
their valid transaction will not be shuffled down by new transactions
to the realm of never confirming after it is created. Consumers also
will not accept this model as Bitcoin expands. If Bitcoin cannot be a
reliable payment system for confirmed transactions then consumers, by
and large, will simply not accept the model once they understand.
Bitcoin will be a dirty payment system, and this will kill the value of
Bitcoin.

Under the current system, a minority of transactions will eventually be
the lucky few who have fees high enough to escape being pushed down the
list.

Once there are more than x transactions (transaction bandwidth limit)
every ten minutes, only those choosing twenty-minute confirmation (2
blocks) from the fee recommendations will have initially at most a
fifty percent chance of ever having their payment confirm by the time
2x transactions is reached. Presently, not even using fee
recommendations can ensure a sufficiently high fee is paid to ensure
transaction confirmation.

I also argue that the current auction model for limited transaction
bandwidth is wrong, is 

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[ZmnSCPxj via bitcoin-dev]

Good morning Damian,

I see you have modified your proposal to be purely driven by miners, with 
fullnodes not actually being able to create a strict "yes-or-no" answer as to 
block validity under your rules.  This implies that your rules cannot be 
enforced and that rational miners will ignore your proposal unless it brings in 
more money for them.  The fact that your proposal provides some mechanism to 
increase block size means that miners will be incentivized to falsify data (by 
making up their own transactions just above your fixed "dust size" threshhold 
whatever that threshhold may be -- and remember, miners get at least 12.5 BTC 
per block, so they can make a lot of little falsified transactions to justify 
every block size increase) until the block size increase per block is the 
maximum possible block size increase.

--

Let me then explain proof-of-work and the arrow of time in Physics.  It may 
seem a digression, but please, bear with me.

Proof-of-work proves that work was performed, and (crucially) that this work 
was done in the past.

This is important because of the arrow of time.

In principle, every physical interaction is reversible.  Visualize a video of 
two indivisible particles.  The two particles move towards each other, collide, 
and because of the collision, fly apart. If you ran this video in reverse, or 
in forward, it would not be distinguishable to you, as an outside observer, 
whether the video was running in reverse or not.  It seems at some level, time 
does not exist.

And yet time exists.

Consider another video, that of a vase being dropped on a hard surface.  The 
vase hits the surface and shatters.  Played in reverse, we can judge it as 
nonsensical: scattered pieces of ceramic spontaneously forming a vase and then 
flying upwards.  This orients our arrow of time: the arrow of time points from 
states of the universe where lesser entropy exists (the vase is whole) to where 
greater entropy exists (the vase is in many pieces).

Indeed, all measures of time are, directly or indirectly, measures of increases 
in entropy.  Consider a simple hourglass: you place it into a state of low 
entropy and high energy with most of the sand is in the upper part of the 
hourglass.  As sand falls, and more of that energy is lost into entropy, you 
judge that time passes.

Consider a proof-of-work algorithm: you place electrons into a state of low 
entropy and high energy.  As electrons go through the mining hardware, 
producing hashes that pass the difficulty requirement, the energy in those 
electrons is lost into entropy (heat), and from the hashes produced (which 
proves not only that work was done, but in particular, that entropy increased 
due to work being done), you judge that time passes.

--

Thus, the blockchain itself is already a service that provides a measure of 
time.  When a block commits to a transaction, then that transaction is known to 
have existed at that block height, at the latest.

Thus one idea, is to have each block commit to some view of the mempool.  If a 
transaction exists in this mempool-view, then you know that the transaction is 
at least that old, and can judge the age from this and use this to compute the 
"transaction priority".

Unfortunately, transferring the data to prove that the mempool-view is valid, 
is equivalent to always sweeping the entire mempool contents per block.  In 
that case you might as well not have a block size limit.

In addition, miners may still commit to a falsely-empty mempool and deny that 
your transaction is old and therefore priority and therefore will simply fill 
their blocks with transactions that have high feerates rather than high 
priority.  Thus feerate will still be the ultimate measure.

Rather than attempt this, perhaps developers should be encouraged to make use 
of existing mechanisms, RBF and CPFP, to allow transactions to be sped up by 
directly manipulating feerates, as priority (by your measure) is not 
practically computable.

Regards,
ZmnSCPxj___
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

I have needed to re-tac my intentions somewhat, there is still much
work to be done.

This is a request for assistance and further discussion of the re-
revised proposal. I am sure there are still issues to be resolved.

## BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering
Transactions In Blocks

Schema:  
##  
Document: BIP Proposal  
Title: UTPFOTIB - Use Transaction Priority For Ordering Transactions In
Blocks  
Date: 26-12-2017  
Author: Damian Williamson willt...@live.com.au  
Licence: Creative Commons Attribution-ShareAlike 4.0 International
License.  
URL: http://thekingjameshrmh.tumblr.com/post/168948530950/bip-proposal-
utpfotib-use-transaction-priority-for-order  
##  

### 1. Abstract

This document proposes to address the issue of transactional
reliability in Bitcoin, where valid transactions may be stuck in the
transaction pool for extended periods or never confirm.

There are two key issues to be resolved to achieve this:

1.  The current transaction bandwidth limit.
2.  The current ad-hoc methods of including transactions in blocks
resulting in variable and confusing confirmation times for valid
transactions, including transactions with a valid fee that may never
confirm.

It is important with any change to protect the value of fees as these
will eventually be the only payment that miners receive. Rather than an
auction model for limited bandwidth, the proposal results in a fee for
priority service auction model.

It would not be true to suggest that all feedback received so far has
been entirely positive although, most of it has been constructive.

The previous threads for this proposal are available here:  
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/s
ubject.html

In all parts of this proposal, references to a transaction, a valid
transaction, a transaction with a valid fee, a valid fee, etc. is
defined as any transaction that is otherwise valid with a fee of at
least 0.1000 BTC/KB as defined as the dust level, interpreting from
Bitcoin Core GUI. Transactions with a fee lower than this rate are
considered dust.

In all parts of this proposal, dust and zero-fee transactions are
always ignored and/or excluded unless specifically mentioned.

It is generally assumed that miners currently prefer to include
transactions with higher fees.

### 2. The need for this proposal

We all must learn to admit that transaction bandwidth is still lurking
as a serious issue for the operation, reliability, safety, consumer
acceptance, uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day
target confirmation from the fee recommendation. That transaction has
still not confirmed after now more than six days - even waiting twice
as long seems quite reasonable to me (note for accuracy: it did
eventually confirm). That transaction is a valid transaction; it is not
rubbish, junk or, spam. Under the current model with transaction
bandwidth limitation, the longer a transaction waits, the less likely
it is ever to confirm due to rising transaction numbers and being
pushed back by transactions with rising fees.

I argue that no transactions with fees above the dust level are rubbish
or junk, only some zero fee transactions might be spam. Having an ever-
increasing number of valid transactions that do not confirm as more new
transactions with higher fees are created is the opposite of operating
a robust, reliable transaction system.

Business cannot operate with a model where transactions may or may not
confirm. Even a business choosing a modest fee has no guarantee that
their valid transaction will not be shuffled down by new transactions
to the realm of never confirming after it is created. Consumers also
will not accept this model as Bitcoin expands. If Bitcoin cannot be a
reliable payment system for confirmed transactions then consumers, by
and large, will simply not accept the model once they understand.
Bitcoin will be a dirty payment system, and this will kill the value of
Bitcoin.

Under the current system, a minority of transactions will eventually be
the lucky few who have fees high enough to escape being pushed down the
list.

Once there are more than x transactions (transaction bandwidth limit)
every ten minutes, only those choosing twenty-minute confirmation (2
blocks) from the fee recommendations will have initially at most a
fifty percent chance of ever having their payment confirm when 2x
transactions is reached. Presently, not even using fee recommendations
can ensure a sufficiently high fee is paid to ensure transaction
confirmation.

I also argue that the current auction model for limited transaction
bandwidth is wrong, is not suitable for a reliable transaction system
and, is wrong for Bitcoin. All transactions with valid fees must
confirm in due time. Currently, Bitcoin is not a safe way to send
payments.

I do not believe that consumers and business are against paying fees,
even high 

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

>.. This system has flaws, they all do.


>The simple fact is that there is currently no known system that works as well 
>as the current system..


Alright, but, we seem to agree, the current system also has flaws. The 
transaction bandwidth limit is a serious issue for transactional reliability.


What you have proposed is interesting but seems to do nothing for the issue of 
transaction bandwidth, which seems to be approaching its threshold:

https://bitinfocharts.com/comparison/bitcoin-transactions.html


Regards,

Damian Williamson


From: Spartacus Rex <spartacusre...@gmail.com>
Sent: Saturday, 23 December 2017 5:07:49 AM
To: Damian Williamson; Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Hi Damian,

Thought I'd chip in.  This is a hard fork scenario. This system has flaws, they 
all do.

If you had a fixed fee per block, so that every txn in that block paid the same 
fee, that might make it easier to include all txns eventually, as you envisage.

The fee could be calculated as the average of the amount txns are prepared to 
pay in the last 1000 blocks.

A txn would say ' I'll pay up to X bitcoins ' and as long as that is more than 
the value required for the block your txn can be added. This is to ensure you 
don't pay more than you are willing.  It also ensures that putting an enormous 
fee will not ensure your txn is processed quickly..

Calculating what the outputs are given a variable fee needs a new mechanism all 
of it's own, but I'm sure it's possible.

The simple fact is that there is currently no known system that works as well 
as the current system..

But there are other systems.


On Dec 22, 2017 15:09, "Damian Williamson via bitcoin-dev" 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
 wrote:

If the cash value of Bitcoin was high enough and zero fee transactions were 
never accepted and not counted when calculating the transaction pool size then 
I do not think it would be such an issue. Why is it even possible to create 
zero fee transactions?


Regards,

Damian Williamson


From: 
bitcoin-dev-boun...@lists.linuxfoundation.org<mailto:bitcoin-dev-boun...@lists.linuxfoundation.org>
 
<bitcoin-dev-boun...@lists.linuxfoundation.org<mailto:bitcoin-dev-boun...@lists.linuxfoundation.org>>
 on behalf of Damian Williamson via bitcoin-dev 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
Sent: Tuesday, 19 December 2017 6:51 PM
To: Mark Friedenbach
Cc: 
bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


Thank you for your constructive feedback. I now see that the proposal 
introduces a potential issue.


>Finally in terms of the broad goal, having block size based on the number of 
>transactions is NOT something desirable in the first place, even if it did 
>work. That’s effectively the same as an infinite block size since anyone 
>anywhere can create transactions in the mempool at no cost.


Do you have any critical suggestion as to how transaction bandwidth limit could 
be addressed, it will eventually become an issue if nothing is changed 
regardless of how high fees go?


Regards,

Damian Williamson




From: Mark Friedenbach <m...@friedenbach.org<mailto:m...@friedenbach.org>>
Sent: Tuesday, 19 December 2017 3:08 AM
To: Damian Williamson
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Damian, you seem to be misunderstanding that either

(1) the strong form of your proposal requires validating the commitment to the 
mempool properties, in which case the mempool becomes consensus critical (an 
impossible requirement); or

(2) in the weak form where the current block is dependent on the commitment in 
the last block only it is becomes a miner-selected field they can freely 
parameterize with no repercussions for setting values totally independent of 
the actual mempool.

If you want to make the block size dependent on the properties of the mempool 
in a consensus critical way, flex cap achieves this. If you want to make the 
contents or properties of the mempool known to well-connected nodes, weak 
blocks achieves that. But you can’t stick the mempool in consensus because it 
fundamentally is not something the nodes have consensus over. That’s a 
chicken-and-the-egg assumption.

Finally in terms of the broad goal, having block size based on the number of 
transactions is NOT something desirable in the first place, even if it did 
work. That’s effectively the same as an infinite block size since

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Spartacus Rex via bitcoin-dev]

..What you have proposed is interesting but seems to do nothing for the
issue of transaction
bandwidth, which seems to be approaching its threshold:
..

This system just shows one way of changing the way a miner calculates txn
priority.

A miners should always do what makes him the most money, so an old txn will
never get priority if a newer one offering more fees comes along. This is
why some txns will never get confirmation.

In this system a txn cannot just pay more fees, as you all pay the same
fees in a block, so an old txn that has a high enough threshold will be
worth just as much to a miner as any txn coming later on.

This way you can be sure that your txn will confirm at some point, and not
just be relegated to the 'never' confirmed pile.


On Dec 24, 2017 03:44, "Damian Williamson" <willt...@live.com.au> wrote:

>.. This system has flaws, they all do.


>The simple fact is that there is currently no known system that works as
well as the current system..


Alright, but, we seem to agree, the current system also has flaws. The
transaction bandwidth limit is a serious issue for transactional
reliability.


What you have proposed is interesting but seems to do nothing for the issue
of transaction bandwidth, which seems to be approaching its threshold:

https://bitinfocharts.com/comparison/bitcoin-transactions.html


Regards,

Damian Williamson
--
*From:* Spartacus Rex <spartacusre...@gmail.com>
*Sent:* Saturday, 23 December 2017 5:07:49 AM
*To:* Damian Williamson; Bitcoin Protocol Discussion

*Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
Transaction Priority For Ordering Transactions In Blocks

Hi Damian,

Thought I'd chip in.  This is a hard fork scenario. This system has flaws,
they all do.

If you had a fixed fee per block, so that every txn in that block paid the
same fee, that might make it easier to include all txns eventually, as you
envisage.

The fee could be calculated as the average of the amount txns are prepared
to pay in the last 1000 blocks.

A txn would say ' I'll pay up to X bitcoins ' and as long as that is more
than the value required for the block your txn can be added. This is to
ensure you don't pay more than you are willing.  It also ensures that
putting an enormous fee will not ensure your txn is processed quickly..

Calculating what the outputs are given a variable fee needs a new mechanism
all of it's own, but I'm sure it's possible.

The simple fact is that there is currently no known system that works as
well as the current system..

But there are other systems.


On Dec 22, 2017 15:09, "Damian Williamson via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

If the cash value of Bitcoin was high enough and zero fee transactions were
never accepted and not counted when calculating the transaction pool size
then I do not think it would be such an issue. Why is it even possible to
create zero fee transactions?


Regards,

Damian Williamson

--
*From:* bitcoin-dev-boun...@lists.linuxfoundation.org <
bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian
Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
*Sent:* Tuesday, 19 December 2017 6:51 PM
*To:* Mark Friedenbach
*Cc:* bitcoin-dev@lists.linuxfoundation.org
*Subject:* [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction
Priority For Ordering Transactions In Blocks


Thank you for your constructive feedback. I now see that the proposal
introduces a potential issue.


>Finally in terms of the broad goal, having block size based on the number
of transactions is NOT something desirable in the first place, even if it
did work. That’s effectively the same as an infinite block size since
anyone anywhere can create transactions in the mempool at no cost.


Do you have any critical suggestion as to how transaction bandwidth limit
could be addressed, it will eventually become an issue if nothing is
changed regardless of how high fees go?


Regards,
Damian Williamson



--
*From:* Mark Friedenbach <m...@friedenbach.org>
*Sent:* Tuesday, 19 December 2017 3:08 AM
*To:* Damian Williamson
*Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
Transaction Priority For Ordering Transactions In Blocks

Damian, you seem to be misunderstanding that either

(1) the strong form of your proposal requires validating the commitment to
the mempool properties, in which case the mempool becomes consensus
critical (an impossible requirement); or

(2) in the weak form where the current block is dependent on the commitment
in the last block only it is becomes a miner-selected field they can freely
parameterize with no repercussions for setting values totally independent
of the actual mempool.

If you want to make the block size dependent on the properties of the
mempool in a consensus critical way, flex cap achieves th

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

I suppose what I intended is (2) the weak form but, what is essentially needed 
is (1) the strong form. The answer may be somewhere in-between.


I do not see that an entire consensus for the mempool is needed, each node just 
needs a loose understanding of the average number of non-zero fee transactions 
in the mempool.


As a pre-rollout, it would be possible to give each node a serial ID and, 
calculate the average number of non-zero fee transactions from the information 
it has and, say every ten minutes, distribute information it has about the 
number of transactions in the mempool. Each node would be able to form its own 
picture of the average number of non-zero fee transactions in the mempool.


At rollout, this information would be the basis a node would use when a block 
is solved to provide the next expected block size. This would still not stop 
cheating by providing especially a number lower than the proposal would allow 
for, to game the system and hike fees. If miners will not act in the long-term 
interest of the stability and operation of the system then they should be 
ignored. If most miners will adhere to the proposal then the average effect 
would be stability in the operation of the proposal, having a few or even 
several nodes posting low numbers for the number of transactions expected in 
the next expected block size would not destroy the operation. If some node 
posted an insanely high number for next expected block size that resulted in 
the mempool being emptied then the proposal would be offended but I do not 
actually care. If no number is posted, just create a block the appropriate size 
ensure conformity. Nodes that have not adopted the proposal could just continue 
to create 1MB blocks.


Actually, the operation could be simplified using the distributed information 
directly to just create blocks of the appropriate size with no need to provide 
next block size. Flexible block size.


The proposal should also specify a minimum number of transactions to include 
for the next block to give at a minimum a 1MB block.


I currently have no information on flex cap, do you have a link?


Regards,

Damian Williamson



From: Mark Friedenbach <m...@friedenbach.org>
Sent: Tuesday, 19 December 2017 3:08 AM
To: Damian Williamson
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Damian, you seem to be misunderstanding that either

(1) the strong form of your proposal requires validating the commitment to the 
mempool properties, in which case the mempool becomes consensus critical (an 
impossible requirement); or

(2) in the weak form where the current block is dependent on the commitment in 
the last block only it is becomes a miner-selected field they can freely 
parameterize with no repercussions for setting values totally independent of 
the actual mempool.

If you want to make the block size dependent on the properties of the mempool 
in a consensus critical way, flex cap achieves this. If you want to make the 
contents or properties of the mempool known to well-connected nodes, weak 
blocks achieves that. But you can’t stick the mempool in consensus because it 
fundamentally is not something the nodes have consensus over. That’s a 
chicken-and-the-egg assumption.

Finally in terms of the broad goal, having block size based on the number of 
transactions is NOT something desirable in the first place, even if it did 
work. That’s effectively the same as an infinite block size since anyone 
anywhere can create transactions in the mempool at no cost.

On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
 wrote:

I do not know why people make the leap that the proposal requires a consensus 
on the transaction pool. It does not.

It may be helpful to have the discussion from the previous thread linked here.
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html

Where I speak of validating that a block conforms to the broadcast next block 
size, I do not propose validating the number broadcast for the next block size 
itself, only that the next generated block is that size.

Regards,
Damian Williamson



From: Damian Williamson <willt...@live.com.au<mailto:willt...@live.com.au>>
Sent: Saturday, 16 December 2017 7:59 AM
To: Rhavar
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

There are really two separate problems to solve.


  1.  How does Bitcoin scale with fixed block size?
  2.  How do we ensure that all valid transactions are eventually included in 
the blockchain?

Those are the two issues that the proposal attempts to address. It makes sense 
to resolve these two problems toget

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Spartacus Rex via bitcoin-dev]

Hi Damian,

Thought I'd chip in.  This is a hard fork scenario. This system has flaws,
they all do.

If you had a fixed fee per block, so that every txn in that block paid the
same fee, that might make it easier to include all txns eventually, as you
envisage.

The fee could be calculated as the average of the amount txns are prepared
to pay in the last 1000 blocks.

A txn would say ' I'll pay up to X bitcoins ' and as long as that is more
than the value required for the block your txn can be added. This is to
ensure you don't pay more than you are willing.  It also ensures that
putting an enormous fee will not ensure your txn is processed quickly..

Calculating what the outputs are given a variable fee needs a new mechanism
all of it's own, but I'm sure it's possible.

The simple fact is that there is currently no known system that works as
well as the current system..

But there are other systems.


On Dec 22, 2017 15:09, "Damian Williamson via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> If the cash value of Bitcoin was high enough and zero fee transactions
> were never accepted and not counted when calculating the transaction pool
> size then I do not think it would be such an issue. Why is it even possible
> to create zero fee transactions?
>
>
> Regards,
>
> Damian Williamson
>
> --
> *From:* bitcoin-dev-boun...@lists.linuxfoundation.org <
> bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian
> Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
> *Sent:* Tuesday, 19 December 2017 6:51 PM
> *To:* Mark Friedenbach
> *Cc:* bitcoin-dev@lists.linuxfoundation.org
> *Subject:* [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
> Transaction Priority For Ordering Transactions In Blocks
>
>
> Thank you for your constructive feedback. I now see that the proposal
> introduces a potential issue.
>
>
> >Finally in terms of the broad goal, having block size based on the number
> of transactions is NOT something desirable in the first place, even if it
> did work. That’s effectively the same as an infinite block size since
> anyone anywhere can create transactions in the mempool at no cost.
>
>
> Do you have any critical suggestion as to how transaction bandwidth limit
> could be addressed, it will eventually become an issue if nothing is
> changed regardless of how high fees go?
>
>
> Regards,
> Damian Williamson
>
>
>
> --------------
> *From:* Mark Friedenbach <m...@friedenbach.org>
> *Sent:* Tuesday, 19 December 2017 3:08 AM
> *To:* Damian Williamson
> *Subject:* Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
> Transaction Priority For Ordering Transactions In Blocks
>
> Damian, you seem to be misunderstanding that either
>
> (1) the strong form of your proposal requires validating the commitment to
> the mempool properties, in which case the mempool becomes consensus
> critical (an impossible requirement); or
>
> (2) in the weak form where the current block is dependent on the
> commitment in the last block only it is becomes a miner-selected field they
> can freely parameterize with no repercussions for setting values totally
> independent of the actual mempool.
>
> If you want to make the block size dependent on the properties of the
> mempool in a consensus critical way, flex cap achieves this. If you want to
> make the contents or properties of the mempool known to well-connected
> nodes, weak blocks achieves that. But you can’t stick the mempool in
> consensus because it fundamentally is not something the nodes have
> consensus over. That’s a chicken-and-the-egg assumption.
>
> Finally in terms of the broad goal, having block size based on the number
> of transactions is NOT something desirable in the first place, even if it
> did work. That’s effectively the same as an infinite block size since
> anyone anywhere can create transactions in the mempool at no cost.
>
> On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> I do not know why people make the leap that the proposal requires a
> consensus on the transaction pool. It does not.
>
> It may be helpful to have the discussion from the previous thread linked
> here.
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/
> 2017-December/015370.html
>
> Where I speak of validating that a block conforms to the broadcast next
> block size, I do not propose validating the number broadcast for the next
> block size itself, only that the next generated block is that size.
>
> Regards,
> Damian Williamson
>
>
> --
> *From:* Damian Williamson <willt

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

If the cash value of Bitcoin was high enough and zero fee transactions were 
never accepted and not counted when calculating the transaction pool size then 
I do not think it would be such an issue. Why is it even possible to create 
zero fee transactions?


Regards,

Damian Williamson


From: bitcoin-dev-boun...@lists.linuxfoundation.org 
<bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian Williamson 
via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Tuesday, 19 December 2017 6:51 PM
To: Mark Friedenbach
Cc: bitcoin-dev@lists.linuxfoundation.org
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


Thank you for your constructive feedback. I now see that the proposal 
introduces a potential issue.


>Finally in terms of the broad goal, having block size based on the number of 
>transactions is NOT something desirable in the first place, even if it did 
>work. That’s effectively the same as an infinite block size since anyone 
>anywhere can create transactions in the mempool at no cost.


Do you have any critical suggestion as to how transaction bandwidth limit could 
be addressed, it will eventually become an issue if nothing is changed 
regardless of how high fees go?


Regards,

Damian Williamson




From: Mark Friedenbach <m...@friedenbach.org>
Sent: Tuesday, 19 December 2017 3:08 AM
To: Damian Williamson
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Damian, you seem to be misunderstanding that either

(1) the strong form of your proposal requires validating the commitment to the 
mempool properties, in which case the mempool becomes consensus critical (an 
impossible requirement); or

(2) in the weak form where the current block is dependent on the commitment in 
the last block only it is becomes a miner-selected field they can freely 
parameterize with no repercussions for setting values totally independent of 
the actual mempool.

If you want to make the block size dependent on the properties of the mempool 
in a consensus critical way, flex cap achieves this. If you want to make the 
contents or properties of the mempool known to well-connected nodes, weak 
blocks achieves that. But you can’t stick the mempool in consensus because it 
fundamentally is not something the nodes have consensus over. That’s a 
chicken-and-the-egg assumption.

Finally in terms of the broad goal, having block size based on the number of 
transactions is NOT something desirable in the first place, even if it did 
work. That’s effectively the same as an infinite block size since anyone 
anywhere can create transactions in the mempool at no cost.

On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
 wrote:

I do not know why people make the leap that the proposal requires a consensus 
on the transaction pool. It does not.

It may be helpful to have the discussion from the previous thread linked here.
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html

Where I speak of validating that a block conforms to the broadcast next block 
size, I do not propose validating the number broadcast for the next block size 
itself, only that the next generated block is that size.

Regards,
Damian Williamson



From: Damian Williamson <willt...@live.com.au<mailto:willt...@live.com.au>>
Sent: Saturday, 16 December 2017 7:59 AM
To: Rhavar
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

There are really two separate problems to solve.


  1.  How does Bitcoin scale with fixed block size?
  2.  How do we ensure that all valid transactions are eventually included in 
the blockchain?

Those are the two issues that the proposal attempts to address. It makes sense 
to resolve these two problems together. Using the proposed system for variable 
block sizes would solve the first problem but there would still be a whole 
bunch of never confirming transactions. I am not sure how to reliably solve the 
second problem at scale without first solving the first.

>* Every node has a (potentially) different mempool, you can't use it to decide 
>consensus values like the max block size.

I do not suggest a consensus. Depending on which node solves a block the value 
for next block size will be different. The consensus would be that blocks will 
adhere to the next block size value transmitted with the current block. It is 
easy to verify that the consensus is being adhered to once in place.

>* Increasing the entropy in a block to make it more unpredictable doesn't 
>really make sense.

Not

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

Thank you for your constructive feedback. I now see that the proposal 
introduces a potential issue.


>Finally in terms of the broad goal, having block size based on the number of 
>transactions is NOT something desirable in the first place, even if it did 
>work. That’s effectively the same as an infinite block size since anyone 
>anywhere can create transactions in the mempool at no cost.


Do you have any critical suggestion as to how transaction bandwidth limit could 
be addressed, it will eventually become an issue if nothing is changed 
regardless of how high fees go?


Regards,

Damian Williamson




From: Mark Friedenbach <m...@friedenbach.org>
Sent: Tuesday, 19 December 2017 3:08 AM
To: Damian Williamson
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Damian, you seem to be misunderstanding that either

(1) the strong form of your proposal requires validating the commitment to the 
mempool properties, in which case the mempool becomes consensus critical (an 
impossible requirement); or

(2) in the weak form where the current block is dependent on the commitment in 
the last block only it is becomes a miner-selected field they can freely 
parameterize with no repercussions for setting values totally independent of 
the actual mempool.

If you want to make the block size dependent on the properties of the mempool 
in a consensus critical way, flex cap achieves this. If you want to make the 
contents or properties of the mempool known to well-connected nodes, weak 
blocks achieves that. But you can’t stick the mempool in consensus because it 
fundamentally is not something the nodes have consensus over. That’s a 
chicken-and-the-egg assumption.

Finally in terms of the broad goal, having block size based on the number of 
transactions is NOT something desirable in the first place, even if it did 
work. That’s effectively the same as an infinite block size since anyone 
anywhere can create transactions in the mempool at no cost.

On Dec 16, 2017, at 8:14 PM, Damian Williamson via bitcoin-dev 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
 wrote:

I do not know why people make the leap that the proposal requires a consensus 
on the transaction pool. It does not.

It may be helpful to have the discussion from the previous thread linked here.
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html

Where I speak of validating that a block conforms to the broadcast next block 
size, I do not propose validating the number broadcast for the next block size 
itself, only that the next generated block is that size.

Regards,
Damian Williamson



From: Damian Williamson <willt...@live.com.au<mailto:willt...@live.com.au>>
Sent: Saturday, 16 December 2017 7:59 AM
To: Rhavar
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

There are really two separate problems to solve.


  1.  How does Bitcoin scale with fixed block size?
  2.  How do we ensure that all valid transactions are eventually included in 
the blockchain?

Those are the two issues that the proposal attempts to address. It makes sense 
to resolve these two problems together. Using the proposed system for variable 
block sizes would solve the first problem but there would still be a whole 
bunch of never confirming transactions. I am not sure how to reliably solve the 
second problem at scale without first solving the first.

>* Every node has a (potentially) different mempool, you can't use it to decide 
>consensus values like the max block size.

I do not suggest a consensus. Depending on which node solves a block the value 
for next block size will be different. The consensus would be that blocks will 
adhere to the next block size value transmitted with the current block. It is 
easy to verify that the consensus is being adhered to once in place.

>* Increasing the entropy in a block to make it more unpredictable doesn't 
>really make sense.

Not a necessary function, just an effect of using a probability-based 
distribution.

>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks 
>miners to ignore their best interests, and confirm transactions by "priority". 
> What are you going to do if a "malicious" miner decides to go after their 
>profits and order by what makes them the most money. Add "ordered by priority" 
>as a consensus requirement? And even if you miners can still sort their 
>mempool by fee, and then order the top 1MB by priority.

I entirely agree with your sentiment that Bitcoin must be incentive compatible. 
It is necessary.

It is in only miners immediate interest to make the most profitable b

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

Thank you for your constructive feedback. I now see that the proposal 
introduces a potential issue.


It is difficult to define then, what is a valid transaction? Clearly, my 
definition was insufficient.


Regards,

Damian Williamson



From: Chris Riley <cri...@gmail.com>
Sent: Monday, 18 December 2017 11:09 PM
To: Damian Williamson; Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

Regarding "problem" #2 where you say "How do we ensure that all valid 
transactions are eventually included in the blockchain?":  I do not believe 
that all people would (a) agree this is a problem or (b) that we do want to 
*ENSURE* that *ALL* valid transactions are eventually included in the 
blockchain.  There are many *valid* transactions that oftentimes miners do not 
(and should not) wish to require be confirmed and included in the blockchain.  
Spam transactions for example can be valid, but used to attack bitcoin by using 
no or low fee.  Any valid transaction MAY be included by a miner, but requiring 
it in some fashion at this point would open the network to other attack 
vectors.  Perhaps you meant it a different way.


On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev 
<bitcoin-dev@lists.linuxfoundation.org<mailto:bitcoin-dev@lists.linuxfoundation.org>>
 wrote:
>
> There are really two separate problems to solve.
>
>
> How does Bitcoin scale with fixed block size?
> How do we ensure that all valid transactions are eventually included in the 
> blockchain?
>
>
> Those are the two issues that the proposal attempts to address. It makes 
> sense to resolve these two problems together. Using the proposed system for 
> variable block sizes would solve the first problem but there would still be a 
> whole bunch of never confirming transactions. I am not sure how to reliably 
> solve the second problem at scale without first solving the first.
>
>
> >* Every node has a (potentially) different mempool, you can't use it to 
> >decide consensus values like the max block size.
>
>
> I do not suggest a consensus. Depending on which node solves a block the 
> value for next block size will be different. The consensus would be that 
> blocks will adhere to the next block size value transmitted with the current 
> block. It is easy to verify that the consensus is being adhered to once in 
> place.
>
> >* Increasing the entropy in a block to make it more unpredictable doesn't 
> >really make sense.
>
> Not a necessary function, just an effect of using a probability-based 
> distribution.
>
> >* Bitcoin should be roughly incentive compatible. Your proposal explicits 
> >asks miners to ignore their best interests, and confirm transactions by 
> >"priority".  What are you going to do if a "malicious" miner decides to go 
> >after their profits and order by what makes them the most money. Add 
> >"ordered by priority" as a consensus requirement? And even if you miners can 
> >still sort their mempool by fee, and then order the top 1MB by priority.
>
> I entirely agree with your sentiment that Bitcoin must be incentive 
> compatible. It is necessary.
>
> It is in only miners immediate interest to make the most profitable block 
> from the available transaction pool. As with so many other things, it is 
> necessary to partially ignore short-term gain for long-term benefit. It is in 
> miners and everybody's long-term interest to have a reliable transaction 
> service. A busy transaction service that confirms lots of transactions per 
> hour will become more profitable as demand increases and more users are 
> prepared to pay for priority. As it is there is currently no way to fully 
> scale because of the transaction bandwidth limit and that is problematic. If 
> all valid transactions must eventually confirm then there must be a way to 
> resolve that problem.
>
> Bitcoin deliberately removes traditional scale by ensuring blocks take ten 
> minutes on average to solve, an ingenious idea and, incentive compatible but, 
> fixed block sizes leaves us with a problem to solve when we want to scale.
>
> >If you could find a good solution that would allow you to know if miners 
> >were following your rule or not (and thus ignore it if it doesn't) then you 
> >wouldn't even need bitcoin in the first place.
>
> I am confident that the math to verify blocks based on the proposal can be 
> developed (and I think it will not be too complex for a mathematician with 
> the relevant experience), however, I am nowhere near experienced enough with 
> probability and statistical analysis to do it. Yes,

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Chris Riley via bitcoin-dev]

Regarding "problem" #2 where you say "How do we ensure that all valid
transactions are eventually included in the blockchain?":  I do not believe
that all people would (a) agree this is a problem or (b) that we do want to
*ENSURE* that *ALL* valid transactions are eventually included in the
blockchain.  There are many *valid* transactions that oftentimes miners do
not (and should not) wish to require be confirmed and included in the
blockchain.  Spam transactions for example can be valid, but used to attack
bitcoin by using no or low fee.  Any valid transaction MAY be included by a
miner, but requiring it in some fashion at this point would open the
network to other attack vectors.  Perhaps you meant it a different way.


On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> There are really two separate problems to solve.
>
>
> How does Bitcoin scale with fixed block size?
> How do we ensure that all valid transactions are eventually included in
the blockchain?
>
>
> Those are the two issues that the proposal attempts to address. It makes
sense to resolve these two problems together. Using the proposed system for
variable block sizes would solve the first problem but there would still be
a whole bunch of never confirming transactions. I am not sure how to
reliably solve the second problem at scale without first solving the first.
>
>
> >* Every node has a (potentially) different mempool, you can't use it to
decide consensus values like the max block size.
>
>
> I do not suggest a consensus. Depending on which node solves a block the
value for next block size will be different. The consensus would be that
blocks will adhere to the next block size value transmitted with the
current block. It is easy to verify that the consensus is being adhered to
once in place.
>
> >* Increasing the entropy in a block to make it more unpredictable
doesn't really make sense.
>
> Not a necessary function, just an effect of using a probability-based
distribution.
>
> >* Bitcoin should be roughly incentive compatible. Your proposal
explicits asks miners to ignore their best interests, and confirm
transactions by "priority".  What are you going to do if a "malicious"
miner decides to go after their profits and order by what makes them the
most money. Add "ordered by priority" as a consensus requirement? And even
if you miners can still sort their mempool by fee, and then order the top
1MB by priority.
>
> I entirely agree with your sentiment that Bitcoin must be incentive
compatible. It is necessary.
>
> It is in only miners immediate interest to make the most profitable block
from the available transaction pool. As with so many other things, it is
necessary to partially ignore short-term gain for long-term benefit. It is
in miners and everybody's long-term interest to have a reliable transaction
service. A busy transaction service that confirms lots of transactions per
hour will become more profitable as demand increases and more users are
prepared to pay for priority. As it is there is currently no way to fully
scale because of the transaction bandwidth limit and that is problematic.
If all valid transactions must eventually confirm then there must be a way
to resolve that problem.
>
> Bitcoin deliberately removes traditional scale by ensuring blocks take
ten minutes on average to solve, an ingenious idea and, incentive
compatible but, fixed block sizes leaves us with a problem to solve when we
want to scale.
>
> >If you could find a good solution that would allow you to know if miners
were following your rule or not (and thus ignore it if it doesn't) then you
wouldn't even need bitcoin in the first place.
>
> I am confident that the math to verify blocks based on the proposal can
be developed (and I think it will not be too complex for a mathematician
with the relevant experience), however, I am nowhere near experienced
enough with probability and statistical analysis to do it. Yes, if Bitcoin
doesn't then it might make another great opportunity for an altcoin but I
am not even nearly interested in promoting any altcoins.
>
>
> If not the proposal that I have put forward, then, hopefully, someone can
come up with a better solution. The important thing is that the issues are
resolved.
>
>
> Regards,
>
> Damian Williamson
>
>
>
> ________________
> From: Rhavar <rha...@protonmail.com>
> Sent: Saturday, 16 December 2017 3:38 AM
> To: Damian Williamson
> Cc: Bitcoin Protocol Discussion
> Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use
Transaction Priority For Ordering Transactions In Blocks
>
> > I understand that there would be technical issues to resolve in
implementation, but, are there no fundamental 

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

I do not know why people make the leap that the proposal requires a consensus 
on the transaction pool. It does not.


It may be helpful to have the discussion from the previous thread linked here.

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/015370.html


Where I speak of validating that a block conforms to the broadcast next block 
size, I do not propose validating the number broadcast for the next block size 
itself, only that the next generated block is that size.


Regards,

Damian Williamson



From: Damian Williamson <willt...@live.com.au>
Sent: Saturday, 16 December 2017 7:59 AM
To: Rhavar
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


There are really two separate problems to solve.


  1.  How does Bitcoin scale with fixed block size?
  2.  How do we ensure that all valid transactions are eventually included in 
the blockchain?


Those are the two issues that the proposal attempts to address. It makes sense 
to resolve these two problems together. Using the proposed system for variable 
block sizes would solve the first problem but there would still be a whole 
bunch of never confirming transactions. I am not sure how to reliably solve the 
second problem at scale without first solving the first.


>* Every node has a (potentially) different mempool, you can't use it to decide 
>consensus values like the max block size.

I do not suggest a consensus. Depending on which node solves a block the value 
for next block size will be different. The consensus would be that blocks will 
adhere to the next block size value transmitted with the current block. It is 
easy to verify that the consensus is being adhered to once in place.

>* Increasing the entropy in a block to make it more unpredictable doesn't 
>really make sense.

Not a necessary function, just an effect of using a probability-based 
distribution.

>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks 
>miners to ignore their best interests, and confirm transactions by "priority". 
> What are you going to do if a "malicious" miner decides to go after their 
>profits and order by what makes them the most money. Add "ordered by priority" 
>as a consensus requirement? And even if you miners can still sort their 
>mempool by fee, and then order the top 1MB by priority.

I entirely agree with your sentiment that Bitcoin must be incentive compatible. 
It is necessary.

It is in only miners immediate interest to make the most profitable block from 
the available transaction pool. As with so many other things, it is necessary 
to partially ignore short-term gain for long-term benefit. It is in miners and 
everybody's long-term interest to have a reliable transaction service. A busy 
transaction service that confirms lots of transactions per hour will become 
more profitable as demand increases and more users are prepared to pay for 
priority. As it is there is currently no way to fully scale because of the 
transaction bandwidth limit and that is problematic. If all valid transactions 
must eventually confirm then there must be a way to resolve that problem.

Bitcoin deliberately removes traditional scale by ensuring blocks take ten 
minutes on average to solve, an ingenious idea and, incentive compatible but, 
fixed block sizes leaves us with a problem to solve when we want to scale.

>If you could find a good solution that would allow you to know if miners were 
>following your rule or not (and thus ignore it if it doesn't) then you 
>wouldn't even need bitcoin in the first place.

I am confident that the math to verify blocks based on the proposal can be 
developed (and I think it will not be too complex for a mathematician with the 
relevant experience), however, I am nowhere near experienced enough with 
probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it 
might make another great opportunity for an altcoin but I am not even nearly 
interested in promoting any altcoins.


If not the proposal that I have put forward, then, hopefully, someone can come 
up with a better solution. The important thing is that the issues are resolved.


Regards,

Damian Williamson



From: Rhavar <rha...@protonmail.com>
Sent: Saturday, 16 December 2017 3:38 AM
To: Damian Williamson
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

> I understand that there would be technical issues to resolve in 
> implementation, but, are there no fundamental errors?

Unfortunately your proposal is really fundamentally broken, on a few levels. I 
think you might need to do a bit more research into how bitcoin works before 
coming up with such 

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

There are really two separate problems to solve.


  1.  How does Bitcoin scale with fixed block size?
  2.  How do we ensure that all valid transactions are eventually included in 
the blockchain?


Those are the two issues that the proposal attempts to address. It makes sense 
to resolve these two problems together. Using the proposed system for variable 
block sizes would solve the first problem but there would still be a whole 
bunch of never confirming transactions. I am not sure how to reliably solve the 
second problem at scale without first solving the first.


>* Every node has a (potentially) different mempool, you can't use it to decide 
>consensus values like the max block size.

I do not suggest a consensus. Depending on which node solves a block the value 
for next block size will be different. The consensus would be that blocks will 
adhere to the next block size value transmitted with the current block. It is 
easy to verify that the consensus is being adhered to once in place.

>* Increasing the entropy in a block to make it more unpredictable doesn't 
>really make sense.

Not a necessary function, just an effect of using a probability-based 
distribution.

>* Bitcoin should be roughly incentive compatible. Your proposal explicits asks 
>miners to ignore their best interests, and confirm transactions by "priority". 
> What are you going to do if a "malicious" miner decides to go after their 
>profits and order by what makes them the most money. Add "ordered by priority" 
>as a consensus requirement? And even if you miners can still sort their 
>mempool by fee, and then order the top 1MB by priority.

I entirely agree with your sentiment that Bitcoin must be incentive compatible. 
It is necessary.

It is in only miners immediate interest to make the most profitable block from 
the available transaction pool. As with so many other things, it is necessary 
to partially ignore short-term gain for long-term benefit. It is in miners and 
everybody's long-term interest to have a reliable transaction service. A busy 
transaction service that confirms lots of transactions per hour will become 
more profitable as demand increases and more users are prepared to pay for 
priority. As it is there is currently no way to fully scale because of the 
transaction bandwidth limit and that is problematic. If all valid transactions 
must eventually confirm then there must be a way to resolve that problem.

Bitcoin deliberately removes traditional scale by ensuring blocks take ten 
minutes on average to solve, an ingenious idea and, incentive compatible but, 
fixed block sizes leaves us with a problem to solve when we want to scale.

>If you could find a good solution that would allow you to know if miners were 
>following your rule or not (and thus ignore it if it doesn't) then you 
>wouldn't even need bitcoin in the first place.

I am confident that the math to verify blocks based on the proposal can be 
developed (and I think it will not be too complex for a mathematician with the 
relevant experience), however, I am nowhere near experienced enough with 
probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it 
might make another great opportunity for an altcoin but I am not even nearly 
interested in promoting any altcoins.


If not the proposal that I have put forward, then, hopefully, someone can come 
up with a better solution. The important thing is that the issues are resolved.


Regards,

Damian Williamson



From: Rhavar <rha...@protonmail.com>
Sent: Saturday, 16 December 2017 3:38 AM
To: Damian Williamson
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks

> I understand that there would be technical issues to resolve in 
> implementation, but, are there no fundamental errors?

Unfortunately your proposal is really fundamentally broken, on a few levels. I 
think you might need to do a bit more research into how bitcoin works before 
coming up with such improvements =)

But just some quick notes:

* Every node has a (potentially) different mempool, you can't use it to decide 
consensus values like the max block size.

* Increasing the entropy in a block to make it more unpredictable doesn't 
really make sense.

* Bitcoin should be roughly incentive compatible. Your proposal explicits asks 
miners to ignore their best interests, and confirm transactions by "priority".  
What are you going to do if a "malicious" miner decides to go after their 
profits and order by what makes them the most money. Add "ordered by priority" 
as a consensus requirement? And even if you miners can still sort their mempool 
by fee, and then order the top 1MB by priority.

If you could find a good solution that would allow you to know if miners were 
follow

Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Rhavar via bitcoin-dev]

> I understand that there would be technical issues to resolve in 
> implementation, but, are there no fundamental errors?

Unfortunately your proposal is really fundamentally broken, on a few levels. I 
think you might need to do a bit more research into how bitcoin works before 
coming up with such improvements =)

But just some quick notes:

* Every node has a (potentially) different mempool, you can't use it to decide 
consensus values like the max block size.

* Increasing the entropy in a block to make it more unpredictable doesn't 
really make sense.

* Bitcoin should be roughly incentive compatible. Your proposal explicits asks 
miners to ignore their best interests, and confirm transactions by "priority".  
What are you going to do if a "malicious" miner decides to go after their 
profits and order by what makes them the most money. Add "ordered by priority" 
as a consensus requirement? And even if you miners can still sort their mempool 
by fee, and then order the top 1MB by priority.

If you could find a good solution that would allow you to know if miners were 
following your rule or not (and thus ignore it if it doesn't) then you wouldn't 
even need bitcoin in the first place.

-Ryan

>  Original Message ----
> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
> Priority For Ordering Transactions In Blocks
> Local Time: December 15, 2017 3:42 AM
> UTC Time: December 15, 2017 9:42 AM
> From: bitcoin-dev@lists.linuxfoundation.org
> To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
>
> I should not take it that the lack of critical feedback to this revised 
> proposal is a glowing endorsement. I understand that there would be technical 
> issues to resolve in implementation, but, are there no fundamental errors?
>
> I suppose that it if is difficult to determine how long a transaction has 
> been waiting in the pool then, each node could simply keep track of when a 
> transaction was first seen. This may have implications for a verify routine, 
> however, for example, if a node was offline, how should it differentiate how 
> long each transaction was waiting in that case? If a node was restarted daily 
> would it always think that all transactions had been waiting in the pool less 
> than one day If each node keeps the current transaction pool in a file and 
> updates it, as transactions are included in blocks and, as new transactions 
> appear in the pool, then that would go some way to alleviate the issue, apart 
> from entirely new nodes. There should be no reason the contents of a 
> transaction pool files cannot be shared without agreement as to the 
> transaction pool between nodes, just as nodes transmit new transactions 
> freely.
>
> It has been questioned why miners could not cheat. For the question of how 
> many transactions to include in a block, I say it is a standoff and miners 
> will conform to the proposal, not wanting to leave transactions with valid 
> fees standing, and, not wanting to shrink the transaction pool. In any case, 
> if miners shrink the transaction pool then I am not immediately concerned 
> since it provides a more efficient service. For the question of including 
> transactions according to the proposal, I say if it is possible to keep track 
> of how long transactions are waiting in the pool so that they can be included 
> on a probability curve then it is possible to verify that blocks conform to 
> the proposal, since the input is a probability, the output should conform to 
> a probability curve.
>
> If someone has the necessary skill, would anyone be willing to develop the 
> math necessary for the proposal?
>
> Regards,
> Damian Williamson
>
> ---
>
> From: bitcoin-dev-boun...@lists.linuxfoundation.org 
> <bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian 
> Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
> Sent: Friday, 8 December 2017 8:01 AM
> To: bitcoin-dev@lists.linuxfoundation.org
> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
> Priority For Ordering Transactions In Blocks
>
> Good afternoon,
>
> The need for this proposal:
>
> We all must learn to admit that transaction bandwidth is still lurking as a 
> serious issue for the operation, reliability, safety, consumer acceptance, 
> uptake and, for the value of Bitcoin.
>
> I recently sent a payment which was not urgent so; I chose three-day target 
> confirmation from the fee recommendation. That transaction has still not 
> confirmed after now more than six days - even waiting twice as long seems 
> quite reasonable to me. That transaction is a valid transaction

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

I should not take it that the lack of critical feedback to this revised 
proposal is a glowing endorsement. I understand that there would be technical 
issues to resolve in implementation, but, are there no fundamental errors?

I suppose that it if is difficult to determine how long a transaction has been 
waiting in the pool then, each node could simply keep track of when a 
transaction was first seen. This may have implications for a verify routine, 
however, for example, if a node was offline, how should it differentiate how 
long each transaction was waiting in that case? If a node was restarted daily 
would it always think that all transactions had been waiting in the pool less 
than one day If each node keeps the current transaction pool in a file and 
updates it, as transactions are included in blocks and, as new transactions 
appear in the pool, then that would go some way to alleviate the issue, apart 
from entirely new nodes. There should be no reason the contents of a 
transaction pool files cannot be shared without agreement as to the transaction 
pool between nodes, just as nodes transmit new transactions freely.

It has been questioned why miners could not cheat. For the question of how many 
transactions to include in a block, I say it is a standoff and miners will 
conform to the proposal, not wanting to leave transactions with valid fees 
standing, and, not wanting to shrink the transaction pool. In any case, if 
miners shrink the transaction pool then I am not immediately concerned since it 
provides a more efficient service. For the question of including transactions 
according to the proposal, I say if it is possible to keep track of how long 
transactions are waiting in the pool so that they can be included on a 
probability curve then it is possible to verify that blocks conform to the 
proposal, since the input is a probability, the output should conform to a 
probability curve.


If someone has the necessary skill, would anyone be willing to develop the math 
necessary for the proposal?

Regards,
Damian Williamson


From: bitcoin-dev-boun...@lists.linuxfoundation.org 
<bitcoin-dev-boun...@lists.linuxfoundation.org> on behalf of Damian Williamson 
via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Friday, 8 December 2017 8:01 AM
To: bitcoin-dev@lists.linuxfoundation.org
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction 
Priority For Ordering Transactions In Blocks


Good afternoon,

The need for this proposal:

We all must learn to admit that transaction bandwidth is still lurking as a 
serious issue for the operation, reliability, safety, consumer acceptance, 
uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day target 
confirmation from the fee recommendation. That transaction has still not 
confirmed after now more than six days - even waiting twice as long seems quite 
reasonable to me. That transaction is a valid transaction; it is not rubbish, 
junk or, spam. Under the current model with transaction bandwidth limitation, 
the longer a transaction waits, the less likely it is ever to confirm due to 
rising transaction numbers and being pushed back by transactions with rising 
fees.

I argue that no transactions are rubbish or junk, only some zero fee 
transactions might be spam. Having an ever-increasing number of valid 
transactions that do not confirm as more new transactions with higher fees are 
created is the opposite of operating a robust, reliable transaction system.

Business cannot operate with a model where transactions may or may not confirm. 
Even a business choosing a modest fee has no guarantee that their valid 
transaction will not be shuffled down by new transactions to the realm of never 
confirming after it is created. Consumers also will not accept this model as 
Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed 
transactions then consumers, by and large, will simply not accept the model 
once they understand. Bitcoin will be a dirty payment system, and this will 
kill the value of Bitcoin.

Under the current system, a minority of transactions will eventually be the 
lucky few who have fees high enough to escape being pushed down the list.

Once there are more than x transactions (transaction bandwidth limit) every ten 
minutes, only those choosing twenty-minute confirmation (2 blocks) will have 
initially at most a fifty percent chance of ever having their payment confirm. 
Presently, not even using fee recommendations can ensure a sufficiently high 
fee is paid to ensure transaction confirmation.

I also argue that the current auction model for limited transaction bandwidth 
is wrong, is not suitable for a reliable transaction system and, is wrong for 
Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a 
safe way to send payments.

I do not believe that consumers 

[bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks

[Damian Williamson via bitcoin-dev]

Good afternoon,

The need for this proposal:

We all must learn to admit that transaction bandwidth is still lurking as a 
serious issue for the operation, reliability, safety, consumer acceptance, 
uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day target 
confirmation from the fee recommendation. That transaction has still not 
confirmed after now more than six days - even waiting twice as long seems quite 
reasonable to me. That transaction is a valid transaction; it is not rubbish, 
junk or, spam. Under the current model with transaction bandwidth limitation, 
the longer a transaction waits, the less likely it is ever to confirm due to 
rising transaction numbers and being pushed back by transactions with rising 
fees.

I argue that no transactions are rubbish or junk, only some zero fee 
transactions might be spam. Having an ever-increasing number of valid 
transactions that do not confirm as more new transactions with higher fees are 
created is the opposite of operating a robust, reliable transaction system.

Business cannot operate with a model where transactions may or may not confirm. 
Even a business choosing a modest fee has no guarantee that their valid 
transaction will not be shuffled down by new transactions to the realm of never 
confirming after it is created. Consumers also will not accept this model as 
Bitcoin expands. If Bitcoin cannot be a reliable payment system for confirmed 
transactions then consumers, by and large, will simply not accept the model 
once they understand. Bitcoin will be a dirty payment system, and this will 
kill the value of Bitcoin.

Under the current system, a minority of transactions will eventually be the 
lucky few who have fees high enough to escape being pushed down the list.

Once there are more than x transactions (transaction bandwidth limit) every ten 
minutes, only those choosing twenty-minute confirmation (2 blocks) will have 
initially at most a fifty percent chance of ever having their payment confirm. 
Presently, not even using fee recommendations can ensure a sufficiently high 
fee is paid to ensure transaction confirmation.

I also argue that the current auction model for limited transaction bandwidth 
is wrong, is not suitable for a reliable transaction system and, is wrong for 
Bitcoin. All transactions must confirm in due time. Currently, Bitcoin is not a 
safe way to send payments.

I do not believe that consumers and business are against paying fees, even high 
fees. What is required is operational reliability.

This great issue needs to be resolved for the safety and reliability of 
Bitcoin. The time to resolve issues in commerce is before they become great big 
issues. The time to resolve this issue is now. We must have the foresight to 
identify and resolve problems before they trip us over.  Simply doubling block 
sizes every so often is reactionary and is not a reliable permanent solution. I 
have written a BIP proposal for a technical solution but, need your help to 
write it up to an acceptable standard to be a full BIP.

I have formatted the following with markdown which is human readable so, I hope 
nobody minds. I have done as much with this proposal as I feel that I am able 
so far but continue to take your feedback.

# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactions 
In Blocks

## The problem:
Everybody wants value. Miners want to maximize revenue from fees (and we 
presume, to minimize block size). Consumers need transaction reliability and, 
(we presume) want low fees.

The current transaction bandwidth limit is a limiting factor for both. As the 
operational safety of transactions is limited, so is consumer confidence as 
they realize the issue and, accordingly, uptake is limited. Fees are 
artificially inflated due to bandwidth limitations while failing to provide a 
full confirmation service for all transactions.

Current fee recommendations provide no satisfaction for transaction reliability 
and, as Bitcoin scales, this will worsen.

Bitcoin must be a fully scalable and reliable service, providing full 
transaction confirmation for every valid transaction.

The possibility to send a transaction with a fee lower than one that is 
acceptable to allow eventual transaction confirmation should be removed from 
the protocol and also from the user interface.

## Solution summary:
Provide each transaction with an individual transaction priority each time 
before choosing transactions to include in the current block, the priority 
being a function of the fee paid (on a curve), and the time waiting in the 
transaction pool (also on a curve) out to n days (n=60 ?). The transaction 
priority to serve as the likelihood of a transaction being included in the 
current block, and for determining the order in which transactions are tried to 
see if they will be included.

Use a target block size. Determine the target block size using; current