Re: [Django] #34171: QuerySet.bulk_create() crashes on mixed case columns in unique_fields/update_fields. (was: ON CONFLICT sql does not work with column aliases)

2022-11-20 Thread Django 
#34171: QuerySet.bulk_create() crashes on mixed case columns in
unique_fields/update_fields.
-+-
 Reporter:  joshbrooks   |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  4.1
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * severity:  Normal => Release blocker
 * cc: Chih Sean Hsu (added)
 * component:  Uncategorized => Database layer (models, ORM)
 * type:  Uncategorized => Bug
 * stage:  Unreviewed => Accepted


Comment:

 Thanks for the report! Bug in 0f6946495a8ec955b471ca1baaf408ceb53d4796. We
 should use columns instead of field names, e.g.
 {{{#!diff
 diff --git a/django/db/models/query.py b/django/db/models/query.py
 index de49e1c58c..fcf0a0616c 100644
 --- a/django/db/models/query.py
 +++ b/django/db/models/query.py
 @@ -798,6 +798,10 @@ class QuerySet(AltersData):
  self._prepare_for_bulk_create(objs)
  with transaction.atomic(using=self.db, savepoint=False):
  objs_with_pk, objs_without_pk = partition(lambda o: o.pk is
 None, objs)
 +if update_fields:
 +update_fields = [self.model._meta.get_field(name) for
 name in update_fields]
 +if unique_fields:
 +unique_fields = [self.model._meta.get_field(name) for
 name in unique_fields]
  if objs_with_pk:
  returned_columns = self._batched_insert(
  objs_with_pk,
 diff --git a/django/db/models/sql/compiler.py
 b/django/db/models/sql/compiler.py
 index 0562a71dd1..caf36382b5 100644
 --- a/django/db/models/sql/compiler.py
 +++ b/django/db/models/sql/compiler.py
 @@ -1725,8 +1725,8 @@ class SQLInsertCompiler(SQLCompiler):
  on_conflict_suffix_sql =
 self.connection.ops.on_conflict_suffix_sql(
  fields,
  self.query.on_conflict,
 -self.query.update_fields,
 -self.query.unique_fields,
 +(f.column for f in self.query.update_fields),
 +(f.column for f in self.query.unique_fields),
  )
  if (
  self.returning_fields
 }}}

 Would you like to prepare a patch? (regression tests are required)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070184990db190-b8c598c7-c0d1-40d7-ac0a-75c1bed81c41-00%40eu-central-1.amazonses.com.

[Django] #34171: ON CONFLICT sql does not work with column aliases

2022-11-20 Thread Django 
#34171: ON CONFLICT sql does not work with column aliases
-+
   Reporter:  joshbrooks |  Owner:  nobody
   Type:  Uncategorized  | Status:  new
  Component:  Uncategorized  |Version:  4.1
   Severity:  Normal |   Keywords:
   Triage Stage:  Unreviewed |  Has patch:  0
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
  UI/UX:  0  |
-+
 Not sure exactly how to phrase this, but when I I'm calling `bulk_update`
 on the manager for a class with `db_column` set on fields the SQL is
 invalid. Ellipses indicate other fields excluded for clarity.

 {{{
 class ActivityBlackListed(models.Model):
 """
 Originally sourced from Activity_BlackListed in /home/josh
 /PNDS_Interim_MIS-Data.accdb (13 records)
 """

 class Meta:
 db_table = "Activity_BlackListed"

 blacklistid = models.IntegerField(primary_key=True,
 db_column="BlacklistID")
 sectorid = models.IntegerField(null=True, blank=True,
 db_column="SectorID")
 ...
 }}}


 {{{qs.bulk_create(instances, update_conflicts=True,
 update_fields=["sectorid", ...], unique_fields=["blacklistid"])}}}

 The "INSERT" code does take into account the db_columns
 {{{
 INSERT INTO "Activity_BlackListed" ("BlacklistID",...) VALUES (%s,  ...),
 }}}


 The code which is generated for "ON CONFLICT" uses the field name and not
 the db_column which leads to a syntax error
 {{{
 'ON CONFLICT("blacklistid") DO UPDATE SET "sectorid" =
 EXCLUDED."sectorid", ...
 }}}

 PostgreSQL returns {{{ERROR:  column "blacklistid" does not exist at
 character 1508}}}

 What should be generated is I think:
 {{{
 'ON CONFLICT("BlacklistID") DO UPDATE SET "SectorID" =
 EXCLUDED."SectorID", ...
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018498f515e7-b5ccecd9-ca2b-4860-bc42-013a6e101cfe-00%40eu-central-1.amazonses.com.

Re: [Django] #29049: Add slicing notation to F expressions

2022-11-20 Thread Django 
#29049: Add slicing notation to F expressions
-+-
 Reporter:  Matthew Pava |Owner:  Abhinav
 |  Yadav
 Type:  New feature  |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  slice F  | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070184986f02bc-077b4f41-5d96-4b5f-8074-663b3b615b47-00%40eu-central-1.amazonses.com.

Re: [Django] #34170: Mitigate the BREACH attack

2022-11-20 Thread Django 
#34170: Mitigate the BREACH attack
-+-
 Reporter:  Andreas Pelme|Owner:  Andreas
 |  Pelme
 Type:  New feature  |   Status:  assigned
Component:  HTTP handling|  Version:  dev
 Severity:  Normal   |   Resolution:
 Keywords:  breach, htb, gzip| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  1
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_docs:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070184986e4fef-3ffc48ad-9f37-433d-8b7c-8c390bc21394-00%40eu-central-1.amazonses.com.

Re: [Django] #34170: Mitigate the BREACH attack

2022-11-20 Thread Django 
#34170: Mitigate the BREACH attack
-+-
 Reporter:  Andreas Pelme|Owner:  Andreas
 |  Pelme
 Type:  New feature  |   Status:  assigned
Component:  HTTP handling|  Version:  dev
 Severity:  Normal   |   Resolution:
 Keywords:  breach, htb, gzip| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Nick Pope):

 * keywords:   => breach, htb, gzip
 * owner:  nobody => Andreas Pelme


Old description:

> The BREACH attach (https://breachattack.com/) was published in 2013. The
> Django project responded soon after
> (https://www.djangoproject.com/weblog/2013/aug/06/breach-and-django/)
> suggesting users to basically stop using gzip. CSRF masking was
> implemented in 2016 (#20869).
>
> In April 2022, a paper called "Heal The Breach" was published, suggesting
> a mitigation that does not depend on masking specific tokens or injecting
> data into HTML. It is rather a generic and effective mitigation. It
> suggests adding randomness to the compressed response by injecting random
> bytes in the gzip filename field of the gzip stream:
> https://ieeexplore.ieee.org/document/9754554
>
> Telling users to disable gzip is not great for bandwidth consumption. I
> propose that Django should implement "Heal The Breach" with sensible
> default.

New description:

 The BREACH attack (https://breachattack.com/) was published in 2013. The
 Django project responded soon after
 (https://www.djangoproject.com/weblog/2013/aug/06/breach-and-django/)
 suggesting users to basically stop using gzip. CSRF masking was
 implemented in 2016 (#20869).

 In April 2022, a paper called "Heal The Breach" was published, suggesting
 a mitigation that does not depend on masking specific tokens or injecting
 data into HTML. It is rather a generic and effective mitigation. It
 suggests adding randomness to the compressed response by injecting random
 bytes in the gzip filename field of the gzip stream:
 https://ieeexplore.ieee.org/document/9754554

 Telling users to disable gzip is not great for bandwidth consumption. I
 propose that Django should implement "Heal The Breach" with sensible
 default.

--

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070184973c066a-ce80de05-6aa3-49c9-9914-3f38930ad0b4-00%40eu-central-1.amazonses.com.

Re: [Django] #34170: Mitigate the BREACH attack

2022-11-20 Thread Django 
#34170: Mitigate the BREACH attack
---+
 Reporter:  Andreas Pelme  |Owner:  nobody
 Type:  New feature|   Status:  assigned
Component:  HTTP handling  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Nick Pope):

 * needs_better_patch:  0 => 1
 * has_patch:  0 => 1
 * version:  4.1 => dev
 * stage:  Unreviewed => Accepted


Comment:

 [https://github.com/django/django/pull/16311 PR].

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070184973a8bfd-b0073b77-2002-44e2-acd0-0838044990aa-00%40eu-central-1.amazonses.com.

[Django] #34170: Mitigate the BREACH attack

2022-11-20 Thread Django 
#34170: Mitigate the BREACH attack
-+--
   Reporter:  Andreas Pelme  |  Owner:  nobody
   Type:  New feature| Status:  assigned
  Component:  HTTP handling  |Version:  4.1
   Severity:  Normal |   Keywords:
   Triage Stage:  Unreviewed |  Has patch:  0
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
  UI/UX:  0  |
-+--
 The BREACH attach (https://breachattack.com/) was published in 2013. The
 Django project responded soon after
 (https://www.djangoproject.com/weblog/2013/aug/06/breach-and-django/)
 suggesting users to basically stop using gzip. CSRF masking was
 implemented in 2016 (#20869).

 In April 2022, a paper called "Heal The Breach" was published, suggesting
 a mitigation that does not depend on masking specific tokens or injecting
 data into HTML. It is rather a generic and effective mitigation. It
 suggests adding randomness to the compressed response by injecting random
 bytes in the gzip filename field of the gzip stream:
 https://ieeexplore.ieee.org/document/9754554

 Telling users to disable gzip is not great for bandwidth consumption. I
 propose that Django should implement "Heal The Breach" with sensible
 default.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018496caa6f8-5d14ac32-241c-4c8f-b025-91821843ddce-00%40eu-central-1.amazonses.com.

Re: [Django] #33879: timesince - wrong results for 11 months + several weeks

2022-11-20 Thread Django 
#33879: timesince - wrong results for 11 months + several weeks
-+-
 Reporter:  אורי |Owner:
 Type:   |  GianpaoloBranca
  Cleanup/optimization   |   Status:  assigned
Component:  Utilities|  Version:  dev
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by GianpaoloBranca):

 * needs_better_patch:  1 => 0


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701849686f792-66b232fd-e8ad-4a34-bed9-3712c36906d4-00%40eu-central-1.amazonses.com.

Re: [Django] #33879: timesince - wrong results for 11 months + several weeks

2022-11-20 Thread Django 
#33879: timesince - wrong results for 11 months + several weeks
-+-
 Reporter:  אורי |Owner:
 Type:   |  GianpaoloBranca
  Cleanup/optimization   |   Status:  assigned
Component:  Utilities|  Version:  dev
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-

Comment (by GianpaoloBranca):

 I updated the function with a different algorithm that covers cases
 related to month duration that were not covered by the month average.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018496868b88-6e908c37-cc96-4f51-8e9b-f9c5787cffc1-00%40eu-central-1.amazonses.com.

Re: [Django] #34160: Django 4.1 Expression contains mixed types for (Big/Small)IntegerFields.

2022-11-20 Thread Django 
#34160: Django 4.1 Expression contains mixed types for (Big/Small)IntegerFields.
-+-
 Reporter:  Martin Lehoux|Owner:  Martin
 |  Lehoux
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  4.1
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Simon Charette):

 Martin, the only way I see on how this can be solved is to revisit how
 `BaseExpression._resolve_output_field` deals with variadic ''merge''
 expressions (e.g. `Case`, `Coalesce`) in a way that tries to be a bit more
 clever with regards to type integer types handling. You could give it a
 shot but be aware that it's going to be exploration work, certainly a nice
 way to learn more about the ORM works but not necessarily one that will
 lead to a fool proof solution.

 In the mean time, I think that the only solution is effectively to add
 explicit `output_field`.

 I'd be happy to review your exploration work but just a small disclaimer
 that I'm not sure what the proper solution might be.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070184962e32c4-9ff0a45e-33a8-4dc6-bf06-7bb98ee166bc-00%40eu-central-1.amazonses.com.

Re: [Django] #32577: Add support for `UUIDAutoField` `DEFAULT_AUTO_FIELD`

2022-11-20 Thread Django 
#32577: Add support for `UUIDAutoField` `DEFAULT_AUTO_FIELD`
-+-
 Reporter:  Tomasz Wójcik|Owner:  (none)
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mathieu Poussin):

 > "So yes, having a sequence PK (that stays hidden) and a GUID UK (that
 customers see) can be a good option"

 It depends, on centralized databases this may be fine because everything
 is local, however on distributed systems (for example cockroachdb), having
 to handle incremental integer is a much slower (and exponentially slower
 with more nodes), because you basically need to coordinate all the nodes
 that can write data to stop, get the last sequence value, insert your
 line, increment the sequence and then unlock it (so you lock insert on the
 table during this time, even on others nodes from the cluster).

 Example: https://github.com/cockroachdb/cockroach/issues/41258

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018495a205a7-da3bc0fd-eae9-4242-81c2-ec28630409e8-00%40eu-central-1.amazonses.com.