subject:"BigAutoField"

Re: Defaulting to use BigAutoField in models

2020-07-12 Thread Tom Forbes

d not much else 
>>> has happened since then.
>>>
>>> As many of you are aware the max value a standard AutoField can hold is 
>>> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need. 
>>> But it’s often not, and you only find out at the worst possible time - out 
>>> of the blue at night and during a period of rapid growth. The process for 
>>> fixing this issue also becomes a lot harder as your data grows - when 
>>> you’ve hit the limit you’re looking at a multi-hour `ALTER TABLE` on 
>>> Postgres during which writes and reads are blocked, or a risky operation to 
>>> create a new table with the correct primary key and copy old data over in 
>>> batches. Basically if you’ve experienced this before you wouldn’t wish it 
>>> on your worst enemy.
>>>
>>> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements 
>>> welcome!) setting that _defaults_ to `BigAutoField`, with prominent 
>>> docs/release notes saying that to preserve the existing behaviour this 
>>> should be set to `AutoField`. I think this the only realistic way we can 
>>> implement this for new projects in a way that ensures it will be used 
>>> meaningfully and not forgotten about until it’s too late.
>>>
>>> Rails managed to do this migration somewhat painlessly due the big 
>>> differences between Rails and Django models. As Django migrations are 
>>> derived from the current model state so there’s no way I can think of to 
>>> express “make this auto-generated field a BigAutoField only if this model 
>>> is *new*”. The way I see it is that a global setting is very easy to 
>>> toggle and there is little chance of missing the large numbers of 
>>> migrations that would be generated during the Django update. Smaller 
>>> applications could apply the migrations with little issue and larger 
>>> applications would be able to opt-out (as well as be reminded that this is 
>>> a problem they could face!).
>>>
>>> Some specifics:
>>> - The setting would take any dotted path to a class, or a single class 
>>> name for a build in field. This would potentially solve [3], and could be 
>>> useful to people who want to default to other fields like UUIDs (or a 
>>> custom BigAutoField) for whatever reason
>>> - The setting would also be used for GenericForeignKeys, which right now 
>>> are backed by a PositiveIntegerField and so suffer from the same AutoField 
>>> limitations
>>>
>>> Any thoughts on this?
>>>
>>> Tom
>>>
>>> 1. 
>>> https://groups.google.com/d/msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
>>>
>>> 2. https://github.com/django/django/pull/8924/
>>>
>>> 3. https://code.djangoproject.com/ticket/56 
>>> <https://groups.google.com/d/msg/django-developers/VFXZpHnuEJc/bbefjX9yCQAJ>
>>>
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2c853449-c519-4605-be23-4faa21ed40d4n%40googlegroups.com.

Re: Defaulting to use BigAutoField in models

2020-06-28 Thread Tom Forbes


I spent some time last week experimenting with my patch, the biggest issue 
I could find was that under some situations we add explicit casts to array 
fields in Postgres based on the *expected* PK field type (i.e `WHERE 
[1,2,3]::int @> [1,2,3]::bigint`) which causes the query to be invalid. The 
casts all look redundant however working out when they are redundant is 
pretty complex, so I'm not sure if this approach has any legs.

> Regarding third party apps a solution could be to allow a per-app config 
option or encourage them to explicitly choose which primary key they use 
for their model but I'm not a big fan of baking logic in the migration 
framework to treat some fields in a special way.

Auto-generated M2M tables are an issue here, we would need to add some 
option to ManyToManyField() to allow passing in a PK field type. But I like 
the approach of just saying "if you ship migrations, be explicit about your 
primary keys" as it seems pretty simple.

Overall I think your two phase plan seems like a good approach  - lets do 
it?

On Thursday, 18 June 2020 at 15:18:35 UTC+1 charettes wrote:

> +1 from me as well but I don't think we should add any logic to the 
> migration framework to handle the transition.
>
> I think the release plan should be something along the following
>
> Phase 1:
> - New projects are generated with MODEL_DEFAULT_PK = 
> 'django.db.models.BigAutoField'
> - A system check or warning is emitted when the setting is not defined and 
> MODEL_DEFAULT_PK default to 'django.db.models.AutoField' during the 
> transition period. The warning should warn that the value will default to 
> 'django.db.models.AutoField' in a future version of Django.
> - An upgrade path is mentioned in the docs to mention that explicit ` id = 
> AutoField()` must be assigned to preserve previous behaviour and avoid 
> migrations. This one of the thing projects such as django-codemod could 
> really help with[0]
>
> Phase 2:
> - settings.MODEL_DEFAULT_PK now defaults to 'django.db.models.BigAutoField'
> - System check/warning about undefined MODEL_DEFAULT_PK setting is removed.
> - Let the migration framework generate migrations just like it normally 
> would for projects that didn't follow the documented migration path.
> - Optionally add a system check that warns about usage of `AutoField` 
> because of its possible overflow.
>
> > I think we should restrict the setting between normal and big auto 
> fields only. Allowing UUID's would be changing the type, with the potential 
> for havoc with code incompalities throughout django. It's also not possible 
> to migrate tables over to the new type.
>
> I'm not sure I agree here. For folks that are setting a up a new project 
> starting with UUIDField primary keys can be useful and if we're adding a 
> setting for this purpose I think we should it make it as flexible as 
> possible.
>
>
> [0] https://github.com/browniebroke/django-codemod
>
> Le jeudi 11 juin 2020 11:28:59 UTC-4, Tom Forbes a écrit :
>>
>> I’d like to re-propose switching Django to use BigAutoField’s rather than 
>> the current AutoField. This has been proposed[1] before (and a MR made[2]) 
>> but it was closed due to implementation issues and not much else has 
>> happened since then.
>>
>> As many of you are aware the max value a standard AutoField can hold is 
>> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need. 
>> But it’s often not, and you only find out at the worst possible time - out 
>> of the blue at night and during a period of rapid growth. The process for 
>> fixing this issue also becomes a lot harder as your data grows - when 
>> you’ve hit the limit you’re looking at a multi-hour `ALTER TABLE` on 
>> Postgres during which writes and reads are blocked, or a risky operation to 
>> create a new table with the correct primary key and copy old data over in 
>> batches. Basically if you’ve experienced this before you wouldn’t wish it 
>> on your worst enemy.
>>
>> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements 
>> welcome!) setting that _defaults_ to `BigAutoField`, with prominent 
>> docs/release notes saying that to preserve the existing behaviour this 
>> should be set to `AutoField`. I think this the only realistic way we can 
>> implement this for new projects in a way that ensures it will be used 
>> meaningfully and not forgotten about until it’s too late.
>>
>> Rails managed to do this migration somewhat painlessly due the big 
>> differences between Rails and Django models. As Django migrations are 
>> derived from the current model state so there’s no way I can think of to 
>> express “make this auto-generated field a BigAu

Re: Defaulting to use BigAutoField in models

2020-06-18 Thread charettes

Email got sent too fast

I meant

> The warning should warn that the value will default to 
'django.db.models.BigAutoField' in a future version of Django.

Regarding third party apps a solution could be to allow a per-app config 
option or encourage them to explicitly choose which primary key they use 
for their model but I'm not a big fan of baking logic in the migration 
framework to treat some fields in a special way.

Simon

Le jeudi 18 juin 2020 10:18:35 UTC-4, charettes a écrit :
>
> +1 from me as well but I don't think we should add any logic to the 
> migration framework to handle the transition.
>
> I think the release plan should be something along the following
>
> Phase 1:
> - New projects are generated with MODEL_DEFAULT_PK = 
> 'django.db.models.BigAutoField'
> - A system check or warning is emitted when the setting is not defined and 
> MODEL_DEFAULT_PK default to 'django.db.models.AutoField' during the 
> transition period. The warning should warn that the value will default to 
> 'django.db.models.AutoField' in a future version of Django.
> - An upgrade path is mentioned in the docs to mention that explicit ` id = 
> AutoField()` must be assigned to preserve previous behaviour and avoid 
> migrations. This one of the thing projects such as django-codemod could 
> really help with[0]
>
> Phase 2:
> - settings.MODEL_DEFAULT_PK now defaults to 'django.db.models.BigAutoField'
> - System check/warning about undefined MODEL_DEFAULT_PK setting is removed.
> - Let the migration framework generate migrations just like it normally 
> would for projects that didn't follow the documented migration path.
> - Optionally add a system check that warns about usage of `AutoField` 
> because of its possible overflow.
>
> > I think we should restrict the setting between normal and big auto 
> fields only. Allowing UUID's would be changing the type, with the potential 
> for havoc with code incompalities throughout django. It's also not possible 
> to migrate tables over to the new type.
>
> I'm not sure I agree here. For folks that are setting a up a new project 
> starting with UUIDField primary keys can be useful and if we're adding a 
> setting for this purpose I think we should it make it as flexible as 
> possible.
>
>
> [0] https://github.com/browniebroke/django-codemod
>
> Le jeudi 11 juin 2020 11:28:59 UTC-4, Tom Forbes a écrit :
>>
>> I’d like to re-propose switching Django to use BigAutoField’s rather than 
>> the current AutoField. This has been proposed[1] before (and a MR made[2]) 
>> but it was closed due to implementation issues and not much else has 
>> happened since then.
>>
>> As many of you are aware the max value a standard AutoField can hold is 
>> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need. 
>> But it’s often not, and you only find out at the worst possible time - out 
>> of the blue at night and during a period of rapid growth. The process for 
>> fixing this issue also becomes a lot harder as your data grows - when 
>> you’ve hit the limit you’re looking at a multi-hour `ALTER TABLE` on 
>> Postgres during which writes and reads are blocked, or a risky operation to 
>> create a new table with the correct primary key and copy old data over in 
>> batches. Basically if you’ve experienced this before you wouldn’t wish it 
>> on your worst enemy.
>>
>> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements 
>> welcome!) setting that _defaults_ to `BigAutoField`, with prominent 
>> docs/release notes saying that to preserve the existing behaviour this 
>> should be set to `AutoField`. I think this the only realistic way we can 
>> implement this for new projects in a way that ensures it will be used 
>> meaningfully and not forgotten about until it’s too late.
>>
>> Rails managed to do this migration somewhat painlessly due the big 
>> differences between Rails and Django models. As Django migrations are 
>> derived from the current model state so there’s no way I can think of to 
>> express “make this auto-generated field a BigAutoField only if this model 
>> is *new*”. The way I see it is that a global setting is very easy to 
>> toggle and there is little chance of missing the large numbers of 
>> migrations that would be generated during the Django update. Smaller 
>> applications could apply the migrations with little issue and larger 
>> applications would be able to opt-out (as well as be reminded that this is 
>> a problem they could face!).
>>
>> Some specifics:
>> - The setting would take any dotted path to a class, or a single class 
>> name for a build in field. This would potential

Re: Defaulting to use BigAutoField in models

2020-06-18 Thread charettes

+1 from me as well but I don't think we should add any logic to the 
migration framework to handle the transition.

I think the release plan should be something along the following

Phase 1:
- New projects are generated with MODEL_DEFAULT_PK = 
'django.db.models.BigAutoField'
- A system check or warning is emitted when the setting is not defined and 
MODEL_DEFAULT_PK default to 'django.db.models.AutoField' during the 
transition period. The warning should warn that the value will default to 
'django.db.models.AutoField' in a future version of Django.
- An upgrade path is mentioned in the docs to mention that explicit ` id = 
AutoField()` must be assigned to preserve previous behaviour and avoid 
migrations. This one of the thing projects such as django-codemod could 
really help with[0]

Phase 2:
- settings.MODEL_DEFAULT_PK now defaults to 'django.db.models.BigAutoField'
- System check/warning about undefined MODEL_DEFAULT_PK setting is removed.
- Let the migration framework generate migrations just like it normally 
would for projects that didn't follow the documented migration path.
- Optionally add a system check that warns about usage of `AutoField` 
because of its possible overflow.

> I think we should restrict the setting between normal and big auto fields 
only. Allowing UUID's would be changing the type, with the potential for 
havoc with code incompalities throughout django. It's also not possible to 
migrate tables over to the new type.

I'm not sure I agree here. For folks that are setting a up a new project 
starting with UUIDField primary keys can be useful and if we're adding a 
setting for this purpose I think we should it make it as flexible as 
possible.


[0] https://github.com/browniebroke/django-codemod

Le jeudi 11 juin 2020 11:28:59 UTC-4, Tom Forbes a écrit :
>
> I’d like to re-propose switching Django to use BigAutoField’s rather than 
> the current AutoField. This has been proposed[1] before (and a MR made[2]) 
> but it was closed due to implementation issues and not much else has 
> happened since then.
>
> As many of you are aware the max value a standard AutoField can hold is 
> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need. 
> But it’s often not, and you only find out at the worst possible time - out 
> of the blue at night and during a period of rapid growth. The process for 
> fixing this issue also becomes a lot harder as your data grows - when 
> you’ve hit the limit you’re looking at a multi-hour `ALTER TABLE` on 
> Postgres during which writes and reads are blocked, or a risky operation to 
> create a new table with the correct primary key and copy old data over in 
> batches. Basically if you’ve experienced this before you wouldn’t wish it 
> on your worst enemy.
>
> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements 
> welcome!) setting that _defaults_ to `BigAutoField`, with prominent 
> docs/release notes saying that to preserve the existing behaviour this 
> should be set to `AutoField`. I think this the only realistic way we can 
> implement this for new projects in a way that ensures it will be used 
> meaningfully and not forgotten about until it’s too late.
>
> Rails managed to do this migration somewhat painlessly due the big 
> differences between Rails and Django models. As Django migrations are 
> derived from the current model state so there’s no way I can think of to 
> express “make this auto-generated field a BigAutoField only if this model 
> is *new*”. The way I see it is that a global setting is very easy to 
> toggle and there is little chance of missing the large numbers of 
> migrations that would be generated during the Django update. Smaller 
> applications could apply the migrations with little issue and larger 
> applications would be able to opt-out (as well as be reminded that this is 
> a problem they could face!).
>
> Some specifics:
> - The setting would take any dotted path to a class, or a single class 
> name for a build in field. This would potentially solve [3], and could be 
> useful to people who want to default to other fields like UUIDs (or a 
> custom BigAutoField) for whatever reason
> - The setting would also be used for GenericForeignKeys, which right now 
> are backed by a PositiveIntegerField and so suffer from the same AutoField 
> limitations
>
> Any thoughts on this?
>
> Tom
>
> 1. 
> https://groups.google.com/d/msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
>
> 2. https://github.com/django/django/pull/8924/
>
> 3. https://code.djangoproject.com/ticket/56 
> <https://groups.google.com/d/msg/django-developers/VFXZpHnuEJc/bbefjX9yCQAJ>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To

Re: Defaulting to use BigAutoField in models

2020-06-18 Thread Adam Johnson

This would make the change only affect existing installs of third party
apps. I think documentation on using RunSQL() in a migration (in a project
app) could be warranted.

On Thu, 18 Jun 2020 at 14:50, Caio Ariede  wrote:

> What about third party apps?
>
> It would be great to migrate every AutoField to BigAutoField without the
> need to specify them manually.
>
>
> --
> Caio
>
>
>
> On 18 Jun 2020, at 09:40, Adam Johnson  wrote:
>
> Would we be able to provide instructions on how to migrate from SIGNED INT
>> to UNSIGNED BIG INT? Perhaps a management command that outputs SQL like
>> sqlmigrate?
>>
>
> It would only be needed to add this to the model:
> id = BigAutoField()
>
> Then migrations would pick it up, because the field is no longer
> auto_created.
>
> On Thu, 18 Jun 2020 at 13:24, Caio Ariede  wrote:
>
>>
>> On 17 Jun 2020, at 16:56, Adam Johnson  wrote:
>>
>> I think special casing the migrations framework is an avenue to explore,
>>> so I created this today and to my surprise it seems to work quite well:
>>> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839
>>> .
>>
>>
>> That is a pretty neat solution. I think it would work quite well.
>>
>>
>> I really like that solution as well, it’s pretty fair :)
>>
>>
>>
>> What about a way to warn users approaching the limit? I think a database
>> check that detects AutoFields where the next ID is >50% of the way through
>> would be a good solution. That's the kind of monitoring I've built in the
>> past.
>>
>> (50% might sound like it's near the limit, but the table is growing with
>> an exponential curve, even quite shallow, it's quite close in terms of
>> time.)
>>
>>
>> I think this is indeed a good to have.
>>
>> Would we be able to provide instructions on how to migrate from SIGNED
>> INT to UNSIGNED BIG INT? Perhaps a management command that outputs SQL like
>> sqlmigrate?
>>
>>
>> --
>> Caio
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/5ECD26CA-339E-4954-A0BA-E7D8296E6988%40gmail.com
>> <https://groups.google.com/d/msgid/django-developers/5ECD26CA-339E-4954-A0BA-E7D8296E6988%40gmail.com?utm_medium=email_source=footer>
>> .
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM0W-8ywE%3D3fHpdoMn3%3Dw-3J%2BN2e_nEgJHaNYia-3Xv%2BWw%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAMyDDM0W-8ywE%3D3fHpdoMn3%3Dw-3J%2BN2e_nEgJHaNYia-3Xv%2BWw%40mail.gmail.com?utm_medium=email_source=footer>
> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/AFE3161D-6571-46B1-AAD3-C21E4A9CE0C0%40gmail.com
> <https://groups.google.com/d/msgid/django-developers/AFE3161D-6571-46B1-AAD3-C21E4A9CE0C0%40gmail.com?utm_medium=email_source=footer>
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM2PYgN6Nnyk%3D2khTzZELBJbDotueQOAV%3DfJFv4zOA0KPA%40mail.gmail.com.

Re: Defaulting to use BigAutoField in models

2020-06-18 Thread Caio Ariede

What about third party apps?

It would be great to migrate every AutoField to BigAutoField without the need 
to specify them manually.


--
Caio



> On 18 Jun 2020, at 09:40, Adam Johnson  wrote:
> 
> Would we be able to provide instructions on how to migrate from SIGNED INT to 
> UNSIGNED BIG INT? Perhaps a management command that outputs SQL like 
> sqlmigrate?
> 
> It would only be needed to add this to the model:
> id = BigAutoField() 
> 
> Then migrations would pick it up, because the field is no longer auto_created.
> 
> On Thu, 18 Jun 2020 at 13:24, Caio Ariede  <mailto:caio.ari...@gmail.com>> wrote:
> 
>> On 17 Jun 2020, at 16:56, Adam Johnson > <mailto:m...@adamj.eu>> wrote:
>> 
>> I think special casing the migrations framework is an avenue to explore, so 
>> I created this today and to my surprise it seems to work quite well: 
>> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839
>>  
>> <https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839>.
>> 
>> That is a pretty neat solution. I think it would work quite well.
> 
> I really like that solution as well, it’s pretty fair :)
> 
> 
>> 
>> What about a way to warn users approaching the limit? I think a database 
>> check that detects AutoFields where the next ID is >50% of the way through 
>> would be a good solution. That's the kind of monitoring I've built in the 
>> past.
>> 
>> (50% might sound like it's near the limit, but the table is growing with an 
>> exponential curve, even quite shallow, it's quite close in terms of time.)
>> 
> 
> I think this is indeed a good to have.
> 
> Would we be able to provide instructions on how to migrate from SIGNED INT to 
> UNSIGNED BIG INT? Perhaps a management command that outputs SQL like 
> sqlmigrate?
> 
> 
> --
> Caio
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> <mailto:django-developers+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/5ECD26CA-339E-4954-A0BA-E7D8296E6988%40gmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/5ECD26CA-339E-4954-A0BA-E7D8296E6988%40gmail.com?utm_medium=email_source=footer>.
> 
> 
> -- 
> Adam
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> <mailto:django-developers+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAMyDDM0W-8ywE%3D3fHpdoMn3%3Dw-3J%2BN2e_nEgJHaNYia-3Xv%2BWw%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/django-developers/CAMyDDM0W-8ywE%3D3fHpdoMn3%3Dw-3J%2BN2e_nEgJHaNYia-3Xv%2BWw%40mail.gmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/AFE3161D-6571-46B1-AAD3-C21E4A9CE0C0%40gmail.com.

Re: Defaulting to use BigAutoField in models

2020-06-18 Thread Adam Johnson

>
> Would we be able to provide instructions on how to migrate from SIGNED INT
> to UNSIGNED BIG INT? Perhaps a management command that outputs SQL like
> sqlmigrate?
>

It would only be needed to add this to the model:
id = BigAutoField()

Then migrations would pick it up, because the field is no longer
auto_created.

On Thu, 18 Jun 2020 at 13:24, Caio Ariede  wrote:

>
> On 17 Jun 2020, at 16:56, Adam Johnson  wrote:
>
> I think special casing the migrations framework is an avenue to explore,
>> so I created this today and to my surprise it seems to work quite well:
>> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839
>> .
>
>
> That is a pretty neat solution. I think it would work quite well.
>
>
> I really like that solution as well, it’s pretty fair :)
>
>
>
> What about a way to warn users approaching the limit? I think a database
> check that detects AutoFields where the next ID is >50% of the way through
> would be a good solution. That's the kind of monitoring I've built in the
> past.
>
> (50% might sound like it's near the limit, but the table is growing with
> an exponential curve, even quite shallow, it's quite close in terms of
> time.)
>
>
> I think this is indeed a good to have.
>
> Would we be able to provide instructions on how to migrate from SIGNED INT
> to UNSIGNED BIG INT? Perhaps a management command that outputs SQL like
> sqlmigrate?
>
>
> --
> Caio
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5ECD26CA-339E-4954-A0BA-E7D8296E6988%40gmail.com
> <https://groups.google.com/d/msgid/django-developers/5ECD26CA-339E-4954-A0BA-E7D8296E6988%40gmail.com?utm_medium=email_source=footer>
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0W-8ywE%3D3fHpdoMn3%3Dw-3J%2BN2e_nEgJHaNYia-3Xv%2BWw%40mail.gmail.com.

Re: Defaulting to use BigAutoField in models

2020-06-18 Thread Caio Ariede


> On 17 Jun 2020, at 16:56, Adam Johnson  wrote:
> 
> I think special casing the migrations framework is an avenue to explore, so I 
> created this today and to my surprise it seems to work quite well: 
> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839 
> .
> 
> That is a pretty neat solution. I think it would work quite well.

I really like that solution as well, it’s pretty fair :)


> 
> What about a way to warn users approaching the limit? I think a database 
> check that detects AutoFields where the next ID is >50% of the way through 
> would be a good solution. That's the kind of monitoring I've built in the 
> past.
> 
> (50% might sound like it's near the limit, but the table is growing with an 
> exponential curve, even quite shallow, it's quite close in terms of time.)
> 

I think this is indeed a good to have.

Would we be able to provide instructions on how to migrate from SIGNED INT to 
UNSIGNED BIG INT? Perhaps a management command that outputs SQL like sqlmigrate?


--
Caio

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5ECD26CA-339E-4954-A0BA-E7D8296E6988%40gmail.com.

Re: Defaulting to use BigAutoField in models

2020-06-17 Thread Adam Johnson

>
> I think special casing the migrations framework is an avenue to explore,
> so I created this today and to my surprise it seems to work quite well:
> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839
> .


That is a pretty neat solution. I think it would work quite well.

What about a way to warn users approaching the limit? I think a database
check that detects AutoFields where the next ID is >50% of the way through
would be a good solution. That's the kind of monitoring I've built in the
past.

(50% might sound like it's near the limit, but the table is growing with an
exponential curve, even quite shallow, it's quite close in terms of time.)

I’m really not sure how we would do that: wouldn’t it require a pretty big
> refactor? It would definitely be cleaner.


Yeah I haven't thought it through. Given the small size of your patch and
the lack of expectation for any similar changes, I think special casing the
migrations framework is a fair tradeoff.

On Wed, 17 Jun 2020 at 13:51, Tom Forbes  wrote:

> I think we should re-open ticket 31007[1] as there was a consensus in 2017
> during the previous attempt at this[2] and things have not really changed -
> a 32bit primary key is still not a sensible default given the calamity that
> can occur when you hit the end of it’s range.
>
> The sticky point is the implementation - how do we migrate new projects
> whilst minimising disruption to existing projects (who either won’t ever
> hit the limit, or have too much data to migrate wholesale). I think special
> casing the migrations framework is an avenue to explore, so I created this
> today and to my surprise it seems to work quite well:
> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839
> .
>
> The obvious effect of this is that for old projects the Django model
> thinks its primary key is 64 bit while the underlying table is only 32 bit.
> I can’t think of much practical impact of this approach:
> - Python 3 doesn’t distinguish between 32/64 bit numbers so code will
> continue to work as-is.
> - Third party migrations that ship with 32 bit columns won’t be impacted,
> and it’s up to them to create migrations if they want to.
> - Adding `id = BigAutoField(primary_key=True)` to a model will create an
> explicit AlterField migration, when projects are able to migrate.
>
> Am I missing something?
>
> I haven't thought this through, but maybe it's possible to restrict the
> change to inside AutoField, rather than create a new field class. If its
> db_parameters() method / db_type() could receive enough context from the
> migration history, it could potentially determine its own database type.
>
>
> I’m really not sure how we would do that: wouldn’t it require a pretty big
> refactor? It would definitely be cleaner.
>
> 1. https://code.djangoproject.com/ticket/31007
> 2.
> https://groups.google.com/forum/#!msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
>
>
> On 12 Jun 2020, at 12:11, Adam Johnson  wrote:
>
> I haven't thought this through, but maybe it's possible to restrict the
> change to inside AutoField, rather than create a new field class. If its
> db_parameters() method / db_type() could receive enough context from the
> migration history, it could potentially determine its own database type.
> For those who want to fixed field sizes, BigAutoField and SmallAutoField
> could be left, and MediumAutoField added.
>
> (Whilst poking around I think I spotted a small cleanup:
> https://code.djangoproject.com/ticket/31698 )
>
> On Fri, 12 Jun 2020 at 11:40, Tom Forbes  wrote:
>
>> I think we should restrict the setting between normal and big auto fields
>> only. Allowing UUID's would be changing the type, with the potential for
>> havoc with code incompalities throughout django. It's also not possible to
>> migrate tables over to the new type.
>>
>>
>> That’s a really good point, I got a bit swept up with the idea to realise
>> that it would break a lot of things if applied generally!
>>
>> The autodetector knows if a model is new. It could be that during one
>> version Django outputs BigAutoField for fields added in CreateModel only.
>>
>>
>> We could make every automatic PK become a BigAutoField but include some
>> logic in the migrations framework to *ignore* changes between an
>> auto_created AutoField -> BigAutoField? This would ignore all existing
>> models in your migration history, but all completely new models would
>> receive BigAutoField PKs from the start. Users who explicitly add a
>> `BigAutoField` to a model with a previously created `AutoField` would see a
>> migration created as the `auto_created` flag is not set. It would also play
>>

Re: Defaulting to use BigAutoField in models

2020-06-17 Thread Tom Forbes

I think we should re-open ticket 31007[1] as there was a consensus in 2017 
during the previous attempt at this[2] and things have not really changed - a 
32bit primary key is still not a sensible default given the calamity that can 
occur when you hit the end of it’s range. 

The sticky point is the implementation - how do we migrate new projects whilst 
minimising disruption to existing projects (who either won’t ever hit the 
limit, or have too much data to migrate wholesale). I think special casing the 
migrations framework is an avenue to explore, so I created this today and to my 
surprise it seems to work quite well: 
https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839 
<https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839>.

The obvious effect of this is that for old projects the Django model thinks its 
primary key is 64 bit while the underlying table is only 32 bit. I can’t think 
of much practical impact of this approach: 
- Python 3 doesn’t distinguish between 32/64 bit numbers so code will continue 
to work as-is. 
- Third party migrations that ship with 32 bit columns won’t be impacted, and 
it’s up to them to create migrations if they want to. 
- Adding `id = BigAutoField(primary_key=True)` to a model will create an 
explicit AlterField migration, when projects are able to migrate.

Am I missing something?

> I haven't thought this through, but maybe it's possible to restrict the 
> change to inside AutoField, rather than create a new field class. If its 
> db_parameters() method / db_type() could receive enough context from the 
> migration history, it could potentially determine its own database type.

I’m really not sure how we would do that: wouldn’t it require a pretty big 
refactor? It would definitely be cleaner.

1. https://code.djangoproject.com/ticket/31007 
<https://code.djangoproject.com/ticket/31007>
2. 
https://groups.google.com/forum/#!msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
 
<https://groups.google.com/forum/#!msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ>


> On 12 Jun 2020, at 12:11, Adam Johnson  wrote:
> 
> I haven't thought this through, but maybe it's possible to restrict the 
> change to inside AutoField, rather than create a new field class. If its 
> db_parameters() method / db_type() could receive enough context from the 
> migration history, it could potentially determine its own database type. For 
> those who want to fixed field sizes, BigAutoField and SmallAutoField could be 
> left, and MediumAutoField added.
> 
> (Whilst poking around I think I spotted a small cleanup: 
> https://code.djangoproject.com/ticket/31698 
> <https://code.djangoproject.com/ticket/31698> )
> 
> On Fri, 12 Jun 2020 at 11:40, Tom Forbes  <mailto:t...@tomforb.es>> wrote:
>> I think we should restrict the setting between normal and big auto fields 
>> only. Allowing UUID's would be changing the type, with the potential for 
>> havoc with code incompalities throughout django. It's also not possible to 
>> migrate tables over to the new type.
> 
> That’s a really good point, I got a bit swept up with the idea to realise 
> that it would break a lot of things if applied generally!
> 
>> The autodetector knows if a model is new. It could be that during one 
>> version Django outputs BigAutoField for fields added in CreateModel only.
> 
> We could make every automatic PK become a BigAutoField but include some logic 
> in the migrations framework to ignore changes between an auto_created 
> AutoField -> BigAutoField? This would ignore all existing models in your 
> migration history, but all completely new models would receive BigAutoField 
> PKs from the start. Users who explicitly add a `BigAutoField` to a model with 
> a previously created `AutoField` would see a migration created as the 
> `auto_created` flag is not set. It would also play nicely with foreign keys 
> which also need to be 8 bytes.
> 
> This would take care of third party apps with migrations and not require a 
> setting but the downside is that model state will be slightly different from 
> the database. All models would have `BigAutoField`’s whereas the database 
> would only have an `AutoField`. This feels slightly iffy to me even though I 
> cannot think of a case where it would have any practical effect.
> 
>> On 11 Jun 2020, at 19:22, Adam Johnson > <mailto:m...@adamj.eu>> wrote:
>> 
>> Big +1 on solving this from me.
>> 
>> - The setting would take any dotted path to a class, or a single class name 
>> for a build in field. This would potentially solve [3], and could be useful 
>> to people who want to default to other fields like UUIDs (or a custom 
>> BigAutoField) for whatever reason
>> 
>> I think we

Re: Defaulting to use BigAutoField in models

2020-06-12 Thread Adam Johnson

I haven't thought this through, but maybe it's possible to restrict the
change to inside AutoField, rather than create a new field class. If its
db_parameters() method / db_type() could receive enough context from the
migration history, it could potentially determine its own database type.
For those who want to fixed field sizes, BigAutoField and SmallAutoField
could be left, and MediumAutoField added.

(Whilst poking around I think I spotted a small cleanup:
https://code.djangoproject.com/ticket/31698 )

On Fri, 12 Jun 2020 at 11:40, Tom Forbes  wrote:

> I think we should restrict the setting between normal and big auto fields
> only. Allowing UUID's would be changing the type, with the potential for
> havoc with code incompalities throughout django. It's also not possible to
> migrate tables over to the new type.
>
>
> That’s a really good point, I got a bit swept up with the idea to realise
> that it would break a lot of things if applied generally!
>
> The autodetector knows if a model is new. It could be that during one
> version Django outputs BigAutoField for fields added in CreateModel only.
>
>
> We could make every automatic PK become a BigAutoField but include some
> logic in the migrations framework to *ignore* changes between an
> auto_created AutoField -> BigAutoField? This would ignore all existing
> models in your migration history, but all completely new models would
> receive BigAutoField PKs from the start. Users who explicitly add a
> `BigAutoField` to a model with a previously created `AutoField` would see a
> migration created as the `auto_created` flag is not set. It would also play
> nicely with foreign keys which also need to be 8 bytes.
>
> This would take care of third party apps with migrations and not require a
> setting but the downside is that model state will be slightly different
> from the database. All models would have `BigAutoField`’s whereas the
> database would only have an `AutoField`. This feels slightly iffy to me
> even though I cannot think of a case where it would have any practical
> effect.
>
> On 11 Jun 2020, at 19:22, Adam Johnson  wrote:
>
> Big +1 on solving this from me.
>
> - The setting would take any dotted path to a class, or a single class
>> name for a build in field. This would potentially solve [3], and could be
>> useful to people who want to default to other fields like UUIDs (or a
>> custom BigAutoField) for whatever reason
>>
>
> I think we should restrict the setting between normal and big auto fields
> only. Allowing UUID's would be changing the type, with the potential for
> havoc with code incompalities throughout django. It's also not possible to
> migrate tables over to the new type.
>
> What do you think the solution is for third-party apps? They ship their
> own migrations and can't really be tied to project state.
>
> As Django migrations are derived from the current model state so there’s
>> no way I can think of to express “make this auto-generated field a
>> BigAutoField only if this model is *new*”.
>>
>
> The autodetector knows if a model is new. It could be that during one
> version Django outputs BigAutoField for fields added in CreateModel only.
>
> On Thu, 11 Jun 2020 at 16:28, Tom Forbes  wrote:
>
>> I’d like to re-propose switching Django to use BigAutoField’s rather than
>> the current AutoField. This has been proposed[1] before (and a MR made[2])
>> but it was closed due to implementation issues and not much else has
>> happened since then.
>>
>> As many of you are aware the max value a standard AutoField can hold is
>> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need.
>> But it’s often not, and you only find out at the worst possible time - out
>> of the blue at night and during a period of rapid growth. The process for
>> fixing this issue also becomes a lot harder as your data grows - when
>> you’ve hit the limit you’re looking at a multi-hour `ALTER TABLE` on
>> Postgres during which writes and reads are blocked, or a risky operation to
>> create a new table with the correct primary key and copy old data over in
>> batches. Basically if you’ve experienced this before you wouldn’t wish it
>> on your worst enemy.
>>
>> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements
>> welcome!) setting that _defaults_ to `BigAutoField`, with prominent
>> docs/release notes saying that to preserve the existing behaviour this
>> should be set to `AutoField`. I think this the only realistic way we can
>> implement this for new projects in a way that ensures it will be used
>> meaningfully and not forgotten about until it’s too late.
>>
>> Rails managed to

Re: Defaulting to use BigAutoField in models

2020-06-12 Thread Tom Forbes

> I think we should restrict the setting between normal and big auto fields 
> only. Allowing UUID's would be changing the type, with the potential for 
> havoc with code incompalities throughout django. It's also not possible to 
> migrate tables over to the new type.

That’s a really good point, I got a bit swept up with the idea to realise that 
it would break a lot of things if applied generally!

> The autodetector knows if a model is new. It could be that during one version 
> Django outputs BigAutoField for fields added in CreateModel only.

We could make every automatic PK become a BigAutoField but include some logic 
in the migrations framework to ignore changes between an auto_created AutoField 
-> BigAutoField? This would ignore all existing models in your migration 
history, but all completely new models would receive BigAutoField PKs from the 
start. Users who explicitly add a `BigAutoField` to a model with a previously 
created `AutoField` would see a migration created as the `auto_created` flag is 
not set. It would also play nicely with foreign keys which also need to be 8 
bytes.

This would take care of third party apps with migrations and not require a 
setting but the downside is that model state will be slightly different from 
the database. All models would have `BigAutoField`’s whereas the database would 
only have an `AutoField`. This feels slightly iffy to me even though I cannot 
think of a case where it would have any practical effect.

> On 11 Jun 2020, at 19:22, Adam Johnson  wrote:
> 
> Big +1 on solving this from me.
> 
> - The setting would take any dotted path to a class, or a single class name 
> for a build in field. This would potentially solve [3], and could be useful 
> to people who want to default to other fields like UUIDs (or a custom 
> BigAutoField) for whatever reason
> 
> I think we should restrict the setting between normal and big auto fields 
> only. Allowing UUID's would be changing the type, with the potential for 
> havoc with code incompalities throughout django. It's also not possible to 
> migrate tables over to the new type.
> 
> What do you think the solution is for third-party apps? They ship their own 
> migrations and can't really be tied to project state.
> 
> As Django migrations are derived from the current model state so there’s no 
> way I can think of to express “make this auto-generated field a BigAutoField 
> only if this model is new”.
> 
> The autodetector knows if a model is new. It could be that during one version 
> Django outputs BigAutoField for fields added in CreateModel only.
> 
> On Thu, 11 Jun 2020 at 16:28, Tom Forbes  <mailto:t...@tomforb.es>> wrote:
> I’d like to re-propose switching Django to use BigAutoField’s rather than the 
> current AutoField. This has been proposed[1] before (and a MR made[2]) but it 
> was closed due to implementation issues and not much else has happened since 
> then.
> 
> As many of you are aware the max value a standard AutoField can hold is 
> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need. 
> But it’s often not, and you only find out at the worst possible time - out of 
> the blue at night and during a period of rapid growth. The process for fixing 
> this issue also becomes a lot harder as your data grows - when you’ve hit the 
> limit you’re looking at a multi-hour `ALTER TABLE` on Postgres during which 
> writes and reads are blocked, or a risky operation to create a new table with 
> the correct primary key and copy old data over in batches. Basically if 
> you’ve experienced this before you wouldn’t wish it on your worst enemy.
> 
> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements welcome!) 
> setting that _defaults_ to `BigAutoField`, with prominent docs/release notes 
> saying that to preserve the existing behaviour this should be set to 
> `AutoField`. I think this the only realistic way we can implement this for 
> new projects in a way that ensures it will be used meaningfully and not 
> forgotten about until it’s too late.
> 
> Rails managed to do this migration somewhat painlessly due the big 
> differences between Rails and Django models. As Django migrations are derived 
> from the current model state so there’s no way I can think of to express 
> “make this auto-generated field a BigAutoField only if this model is new”. 
> The way I see it is that a global setting is very easy to toggle and there is 
> little chance of missing the large numbers of migrations that would be 
> generated during the Django update. Smaller applications could apply the 
> migrations with little issue and larger applications would be able to opt-out 
> (as well as be reminded that this is a problem they could face!).
> 
> Some specifics:
> - The s

Re: Defaulting to use BigAutoField in models

2020-06-11 Thread Adam Johnson

Big +1 on solving this from me.

- The setting would take any dotted path to a class, or a single class name
> for a build in field. This would potentially solve [3], and could be useful
> to people who want to default to other fields like UUIDs (or a custom
> BigAutoField) for whatever reason
>

I think we should restrict the setting between normal and big auto fields
only. Allowing UUID's would be changing the type, with the potential for
havoc with code incompalities throughout django. It's also not possible to
migrate tables over to the new type.

What do you think the solution is for third-party apps? They ship their own
migrations and can't really be tied to project state.

As Django migrations are derived from the current model state so there’s no
> way I can think of to express “make this auto-generated field a
> BigAutoField only if this model is *new*”.
>

The autodetector knows if a model is new. It could be that during one
version Django outputs BigAutoField for fields added in CreateModel only.

On Thu, 11 Jun 2020 at 16:28, Tom Forbes  wrote:

> I’d like to re-propose switching Django to use BigAutoField’s rather than
> the current AutoField. This has been proposed[1] before (and a MR made[2])
> but it was closed due to implementation issues and not much else has
> happened since then.
>
> As many of you are aware the max value a standard AutoField can hold is
> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need.
> But it’s often not, and you only find out at the worst possible time - out
> of the blue at night and during a period of rapid growth. The process for
> fixing this issue also becomes a lot harder as your data grows - when
> you’ve hit the limit you’re looking at a multi-hour `ALTER TABLE` on
> Postgres during which writes and reads are blocked, or a risky operation to
> create a new table with the correct primary key and copy old data over in
> batches. Basically if you’ve experienced this before you wouldn’t wish it
> on your worst enemy.
>
> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements
> welcome!) setting that _defaults_ to `BigAutoField`, with prominent
> docs/release notes saying that to preserve the existing behaviour this
> should be set to `AutoField`. I think this the only realistic way we can
> implement this for new projects in a way that ensures it will be used
> meaningfully and not forgotten about until it’s too late.
>
> Rails managed to do this migration somewhat painlessly due the big
> differences between Rails and Django models. As Django migrations are
> derived from the current model state so there’s no way I can think of to
> express “make this auto-generated field a BigAutoField only if this model
> is *new*”. The way I see it is that a global setting is very easy to
> toggle and there is little chance of missing the large numbers of
> migrations that would be generated during the Django update. Smaller
> applications could apply the migrations with little issue and larger
> applications would be able to opt-out (as well as be reminded that this is
> a problem they could face!).
>
> Some specifics:
> - The setting would take any dotted path to a class, or a single class
> name for a build in field. This would potentially solve [3], and could be
> useful to people who want to default to other fields like UUIDs (or a
> custom BigAutoField) for whatever reason
> - The setting would also be used for GenericForeignKeys, which right now
> are backed by a PositiveIntegerField and so suffer from the same AutoField
> limitations
>
> Any thoughts on this?
>
> Tom
>
> 1.
> https://groups.google.com/d/msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
>
> 2. https://github.com/django/django/pull/8924/
>
> 3. https://code.djangoproject.com/ticket/56
> <https://groups.google.com/d/msg/django-developers/VFXZpHnuEJc/bbefjX9yCQAJ>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/435EC704-3EF6-4EF4-BF85-175AE29C01F5%40tomforb.es
> <https://groups.google.com/d/msgid/django-developers/435EC704-3EF6-4EF4-BF85-175AE29C01F5%40tomforb.es?utm_medium=email_source=footer>
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1EbbffnydYkVZELcvX3d5y%3DprOCi-b0naYDsP0PRSOwA%40mail.gmail.com.

Re: Defaulting to use BigAutoField in models

2020-06-11 Thread Caio Ariede

For the record, there was a related discussion a few months ago:

https://groups.google.com/d/msg/django-developers/VFXZpHnuEJc/bbefjX9yCQAJ


> On 11 Jun 2020, at 12:28, Tom Forbes  wrote:
> 
> nsures it will be used meaningfully and not forgotten about until it’s too 
> late.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4F05621B-F004-44B5-901E-29962D531804%40gmail.com.

Defaulting to use BigAutoField in models

2020-06-11 Thread Tom Forbes

I’d like to re-propose switching Django to use BigAutoField’s rather than the
current AutoField. This has been proposed[1] before (and a MR made[2]) but it
was closed due to implementation issues and not much else has happened since
then.

As many of you are aware the max value a standard AutoField can hold is
2,147,483,647 (2.1 billion) which sounds like more than you can ever need. But
it’s often not, and you only find out at the worst possible time - out of the
blue at night and during a period of rapid growth. The process for fixing this
issue also becomes a lot harder as your data grows - when you’ve hit the limit
you’re looking at a multi-hour `ALTER TABLE` on Postgres during which writes
and reads are blocked, or a risky operation to create a new table with the
correct primary key and copy old data over in batches. Basically if you’ve
experienced this before you wouldn’t wish it on your worst enemy.

I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements welcome!)
setting that _defaults_ to `BigAutoField`, with prominent docs/release notes
saying that to preserve the existing behaviour this should be set to
`AutoField`. I think this the only realistic way we can implement this for new
projects in a way that ensures it will be used meaningfully and not forgotten
about until it’s too late.

Rails managed to do this migration somewhat painlessly due the big differences
between Rails and Django models. As Django migrations are derived from the
current model state so there’s no way I can think of to express “make this
auto-generated field a BigAutoField only if this model is new”. The way I see
it is that a global setting is very easy to toggle and there is little chance
of missing the large numbers of migrations that would be generated during the
Django update. Smaller applications could apply the migrations with little
issue and larger applications would be able to opt-out (as well as be reminded
that this is a problem they could face!).

Some specifics:
- The setting would take any dotted path to a class, or a single class name for
a build in field. This would potentially solve [3], and could be useful to
people who want to default to other fields like UUIDs (or a custom
BigAutoField) for whatever reason
- The setting would also be used for GenericForeignKeys, which right now are
backed by a PositiveIntegerField and so suffer from the same AutoField
limitations

Any thoughts on this?

Tom

1. https://groups.google.com/d/msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
<https://groups.google.com/d/msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ>

2. https://github.com/django/django/pull/8924/
<https://github.com/django/django/pull/8924/>

3. https://code.djangoproject.com/ticket/56
<https://groups.google.com/d/msg/django-developers/VFXZpHnuEJc/bbefjX9yCQAJ>

44 matches

Mail list logo