Re: Change Local Memory Cache to Use LRU #28977

[Josh Smeaton]

To lend some weight to this, I've implemented an LRU loc mem cache and have 
done some benchmarking. There are some graphs in the 
readme: https://github.com/kogan/django-lrucache-backend - which I've 
written a little about 
https://devblog.kogan.com/blog/a-smarter-local-memory-django-cache-backend 
(I don't think my particular implementation is useful to core, as it relies 
on a 3rd party C lib, but using OrderedDict is cool!).

You can also get a very nice bump in throughput if you eliminate the 
`validate_key` check, which does a character by character check of the key 
to avoid issues with particular characters in memcache.

We don't want to be promoting locmemcache too much, but that said, if we 
can provide a better default then we should in my opinion.

On Friday, 5 January 2018 10:12:39 UTC+11, Grant Jenks wrote:
>
> Hi all--
>
> Long time user, first time poster, here. Thank you all for Django!
>
> The current local memory cache (locmem) in Django uses a pseudo-random 
> culling strategy. Rather than random, the OrderedDict data type can be used 
> to implement an LRU eviction policy. A prototype implementation is already 
> used by functools.lru_cache and Python 3 now supports 
> OrderedDict.move_to_end and OrderedDict.popitem to ease the implementation.
>
> I have created an example set of changes at 
> https://github.com/grantjenks/django/tree/ticket_28977 in commit 
> https://github.com/grantjenks/django/commit/b06574f6713d4b7d367d7a11e0268fb62f5fd1d1
>
> Is there a consensus as to the value of these changes?
>
> Sincerely,
> Grant Jenks
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fdb91901-c378-4258-9201-d24a9f5f103e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[Josh Smeaton]

I wasn't aware of this new feature Shai, thanks for pointing it out!

For this particular case I'd prefer locking to be bound to a particular 
queryset rather than the database as a whole. I would also expect it to 
fail loudly when accessing a non-fetched related object (book.author), 
which can be a common cause of pain.

I'm also still very interested in auto-prefetch Adam, is there any help I 
can lend?

On Thursday, 4 January 2018 17:32:19 UTC+11, Shai Berger wrote:
>
> Hi all, 
>
> Django 2.0 has a new feature[1] which allows you to say "I expect no 
> actual 
> database queries in this piece of code". It is not designed to stop a 
> specific 
> queryset from spawning requests, so getting it to do exactly what's asked 
> for 
> in this thread may be a little involved, but if your goal is to prevent 
> surprise queries, I think it is actually better than sealing a single 
> queryset. 
>
> HTH, 
> Shai 
>
> [1] https://docs.djangoproject.com/en/2.0/topics/db/instrumentation/ 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7441a83e-e459-4612-8dd7-cff48f5af7f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[charettes]

Shai,

`execute_wrapper` can be useful to prevent any queries from
happening at all but I believe there's merit in wanting to
prevent objects retrieved from a queryset from executing ghost
queries during manipulations of the returned `Model` instances
while allowing other unrelated queries to be performed.

For example, what if a database based cache backend needs to be
called during a queryset serialization, or that the `ContentType`
in-memory cache is not warmed up during serialization, or that
any other kind of query, that is not linked to the manipulated
queryset needs to be executed during the serialization process.

Adam,

While I understand the desire to make life easier for Django
newcomers through automated ORM optimization I feel like there
should still be a way to help developers familar with existing
manual optimization techniques to avoid unexpected queries when
they've clearly defined what they expect the ORM to do.

I feel like a few contributors in the thread you've linked to
express similar concerns.

e.g.

> I completely agree that visibility of this problem is a major
> issue, and would really welcome work on improving this,
> especially in DEBUG mode. One option might be a method that
> replaces lazy loads with exceptions.

- Luke Plant[0]

Best,
Simon

[0] 
https://groups.google.com/d/msg/django-developers/EplZGj-ejvg/cHyFdoaXCAAJ

Le jeudi 4 janvier 2018 05:55:21 UTC-5, Adam Johnson a écrit :
>
> 1. I don't think the seal API on the QuerySet is enough, there are other 
> ways implicit queries can be triggered without a QuerySet, e.g. 
> Book(author_id=1).author . Shai's suggestion of using a query execution 
> blocker is probably the best approach. If you were really worried you could 
> even implement it the other way round, have a permanent block and then 
> 'unlock it' around blocks of code you expect to make queries.
>
> 2. https://github.com/YPlan/django-perf-rec is assertNumQueries on 
> steroids and we used it at YPlan to great success :)
>
> 3. auto-prefetch would get rid of a lot of N+1 query problems: 
> https://groups.google.com/forum/#!topic/django-developers/EplZGj-ejvg
>
> On 4 January 2018 at 06:28, Shai Berger  
> wrote:
>
>> Hi all,
>>
>> Django 2.0 has a new feature[1] which allows you to say "I expect no 
>> actual
>> database queries in this piece of code". It is not designed to stop a 
>> specific
>> queryset from spawning requests, so getting it to do exactly what's asked 
>> for
>> in this thread may be a little involved, but if your goal is to prevent
>> surprise queries, I think it is actually better than sealing a single
>> queryset.
>>
>> HTH,
>> Shai
>>
>> [1] https://docs.djangoproject.com/en/2.0/topics/db/instrumentation/
>>
>
>
>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/44eac846-97ee-481a-8136-4388eaf62a39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Change Local Memory Cache to Use LRU #28977

[Grant Jenks]

Hi all--

Long time user, first time poster, here. Thank you all for Django!

The current local memory cache (locmem) in Django uses a pseudo-random 
culling strategy. Rather than random, the OrderedDict data type can be used 
to implement an LRU eviction policy. A prototype implementation is already 
used by functools.lru_cache and Python 3 now supports 
OrderedDict.move_to_end and OrderedDict.popitem to ease the implementation.

I have created an example set of changes 
at https://github.com/grantjenks/django/tree/ticket_28977 in 
commit 
https://github.com/grantjenks/django/commit/b06574f6713d4b7d367d7a11e0268fb62f5fd1d1

Is there a consensus as to the value of these changes?

Sincerely,
Grant Jenks

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1f9d2225-18e8-46f3-9311-b07177c4baca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: UnicodeDecodeError at /manager/procesar_csv/ 'utf8' codec can't decode byte 0xd1 in position 0: invalid continuation byte

[Adam Johnson]

This mailing list is for the development of Django itself, not for support.
Use the django-users mailing list for that, or IRC #django on freenode, or
a site like Stack Overflow.

On 4 January 2018 at 16:01, Hernán Darío Zapata Villa 
wrote:

> Good morning everybody
>
> I have a mistake about the encondnig with utf8 with a csv file , when
> select the file and public them, django shows this mistake:
>
> UnicodeDecodeError at /manager/procesar_csv/  'utf8' codec can't decode
> byte 0xd1 in position 0: invalid continuation byte
>
> guys, how can fix this? i am an amateur in django, I was looking for a
> solution in different formus(stackoverlfow, devcde, github) but nothing
> working!.
>
> Thanks for the attetion.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/7c0af574-e03c-4a61-933c-
> efeb3138cf32%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0UhZbfXpgmRsHhbJ7T2HV7K2ZOimH7zaOpVfRmz98aGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

UnicodeDecodeError at /manager/procesar_csv/ 'utf8' codec can't decode byte 0xd1 in position 0: invalid continuation byte

[Hernán Darío Zapata Villa]

Good morning everybody

I have a mistake about the encondnig with utf8 with a csv file , when 
select the file and public them, django shows this mistake:

UnicodeDecodeError at /manager/procesar_csv/  'utf8' codec can't decode 
byte 0xd1 in position 0: invalid continuation byte

guys, how can fix this? i am an amateur in django, I was looking for a 
solution in different formus(stackoverlfow, devcde, github) but nothing 
working!.

Thanks for the attetion.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7c0af574-e03c-4a61-933c-efeb3138cf32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [contrib.staticfiles] Change verbosity level of collectstatic command's outputs

[Aymeric Augustin]

Hello,

The ticket tracker wasn't _obviously_ the wrong place :-) but the mailing
list is a better way to draw the attention of many people to an issue that
requires discussion.

I agree that the default logging is needlessly verbose and that's annoying.

Printing information about every file should be moved to level 2. Aggregate
information (e.g. N files copied, M files postprocessed) should remain at
level 1.

Best regards,

-- 
Aymeric.


2018-01-04 11:10 GMT+01:00 Tanguy Nodet :

> This is a repost of this ticket
> , which was obviously in the
> wrong place.
>
> I have a suggestion concerning the logging of the *collectstatic *command.
> Presently, with the default verbosity level (1), individual information on
> each copied or linked file are printed to *sdtout*, resulting in a log
> looking like this:
> Copying  '/path/to/the/file'
> # or
> Linking '/path/to/the/file'
> # [...dozens if not hundreds of other lines...]
> 187 static files copied to '/var/www/my_site/static'.
>
> Now providing the command is launched at every startup of the server (and
> that you restart your server often, for example in development), this can
> result in pretty heavy and "polluted" server logs.
>
> If you use verbosity level 0, no output whatsoever is printed (obviously).
> Plus, in several cases, the output for verbosity levels 1 and 2 are the
> same.
> Looking at the ​collectstatic source code
> ,
> only skipped operations (for post-processing, linking, copy, deletion with
> *--clear*) and deletion (of previously copied files that got modified)
> are logged with a verbosity level of 2.
>
> In my opinion, it would be nice to shift the lines concerning individual
> files to a verbosity level of 2, and keep the summary at a verbosity level
> of 1.
> I think it would also bring consistency with other commands, such as
> *loaddata*, which only displays its summary (at a verbosity level of 1):
> Installed 42 object(s) from 7 fixture(s)
>
> What are your thoughts?
> Cheers!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/266d9ab5-e916-4eb4-9a24-
> 52ac0a264d00%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANE-7mUmkK%2B5V7bV_qpbi5CR2hYwvQqVrjo1mtmwY1HkY2TetQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[contrib.staticfiles] Change verbosity level of collectstatic command's outputs

[Tanguy Nodet]

This is a repost of this ticket 
, which was obviously in the 
wrong place.

I have a suggestion concerning the logging of the *collectstatic *command.
Presently, with the default verbosity level (1), individual information on 
each copied or linked file are printed to *sdtout*, resulting in a log 
looking like this:
Copying  '/path/to/the/file'
# or
Linking '/path/to/the/file'
# [...dozens if not hundreds of other lines...]
187 static files copied to '/var/www/my_site/static'.

Now providing the command is launched at every startup of the server (and 
that you restart your server often, for example in development), this can 
result in pretty heavy and "polluted" server logs.

If you use verbosity level 0, no output whatsoever is printed (obviously).
Plus, in several cases, the output for verbosity levels 1 and 2 are the 
same.
Looking at the ​collectstatic source code 
,
 
only skipped operations (for post-processing, linking, copy, deletion with 
*--clear*) and deletion (of previously copied files that got modified) are 
logged with a verbosity level of 2.

In my opinion, it would be nice to shift the lines concerning individual 
files to a verbosity level of 2, and keep the summary at a verbosity level 
of 1.
I think it would also bring consistency with other commands, such as 
*loaddata*, which only displays its summary (at a verbosity level of 1):
Installed 42 object(s) from 7 fixture(s)

What are your thoughts?
Cheers!

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/266d9ab5-e916-4eb4-9a24-52ac0a264d00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[Adam Johnson]

1. I don't think the seal API on the QuerySet is enough, there are other
ways implicit queries can be triggered without a QuerySet, e.g.
Book(author_id=1).author . Shai's suggestion of using a query execution
blocker is probably the best approach. If you were really worried you could
even implement it the other way round, have a permanent block and then
'unlock it' around blocks of code you expect to make queries.

2. https://github.com/YPlan/django-perf-rec is assertNumQueries on steroids
and we used it at YPlan to great success :)

3. auto-prefetch would get rid of a lot of N+1 query problems:
https://groups.google.com/forum/#!topic/django-developers/EplZGj-ejvg

On 4 January 2018 at 06:28, Shai Berger  wrote:

> Hi all,
>
> Django 2.0 has a new feature[1] which allows you to say "I expect no actual
> database queries in this piece of code". It is not designed to stop a
> specific
> queryset from spawning requests, so getting it to do exactly what's asked
> for
> in this thread may be a little involved, but if your goal is to prevent
> surprise queries, I think it is actually better than sealing a single
> queryset.
>
> HTH,
> Shai
>
> [1] https://docs.djangoproject.com/en/2.0/topics/db/instrumentation/
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0E5CFQjuK3u6-npiBdZh5EcjHDP%2BciFoG7T%3DQ4nyd2AQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.