date:20171206

memory leak in django app?

2017-12-06 Thread Etienne Robillard


Hi all,

I'm struggling to understand how django/python may allocate and 
unallocate memory when used with uWSGI.


I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 
2 CPUs.


Is that enough RAM memory for a uWSGI/Gevent based WSGI app running Django?

I'm running uWSGI with the --gevent switch in order to allow cooperative 
multithreading, but my free RAM memory is always decreasing when nginx 
is running.


How can I debug memory allocation in a Django/uWSGI app?

I defined also in my sitecustomize.py "gc.enable()" to allow garbage 
collection, but it does not appears to make any differences.


Can you recommend any libraries to debug/profile memory allocation in 
Python 2.7 ?


Is Django more memory efficient with --pymalloc or by using the default 
linux malloc() ?


Thank you in advance,

Etienne

--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1e6a9dfa-5b8b-194b-b860-084fc6887890%40yandex.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enable https for a production django application

2017-12-06 Thread Tim Chase

Sorry, forgot to mention OV ("Organization Validation") certs.
There's a good comparison here

https://www.ssl.com/article/dv-ov-and-ev-certificates/

Additionally, you *can* self-sign your own cert but it will throw up
an "untrusted cert" warning to your users.  This used to be a viable
option for free certs before Let's Encrypt but now there's really no
reason to have self-signed certs.

-tkc


On 2017-12-04 07:57, Tim Chase wrote:
> On 2017-12-04 01:15, BIJAL MANIAR wrote:
> > Do we need to purchase SSL certificate? Any links on that would be
> > helpful.  
> 
> There are two types of cert: DV ("Domain Validation" merely lets you
> know that you're securely talking with the domain you think you are)
> and EV ("Extended Validation" where they actually check up on you,
> verifying your identity as a business/individual).
> 
> 
> 
> You can get free DV certs through several providers, but setting it
> up with Let's Encrypt is broadly supported on a variety of
> platforms.  They're only valid for a fairly short period of time (90
> days, IIRC) but there are good tools to automate the renewal of the
> cert.
> 
>   https://letsencrypt.org
> 
> For an EV cert, you have to pay and submit various forms of
> documentation.  I've heard good things from multiple people about
> obtaining theirs from DigiCert and multiple complaints about Comodo.
> 
>   https://www.digicert.com/
> 
> Hope this helps,
> 
> -tkc
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group. To unsubscribe from this group and
> stop receiving emails from it, send an email to
> django-users+unsubscr...@googlegroups.com. To post to this group,
> send email to django-users@googlegroups.com. Visit this group at
> https://groups.google.com/group/django-users. To view this
> discussion on the web visit
> https://groups.google.com/d/msgid/django-users/20171204075738.2f9c25f9%40bigbox.christie.dr.
> For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20171206061023.69d183d4%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

Re: memory leak in django app?

2017-12-06 Thread Antonis Christofides

Hello,

the amount of memory you need depends on what Django does and how many workers
(instances of Django) you run (which usually depends on how many requests you
are getting and how I/O intensive your Django application is). For many
applications, 512 MB is enough.

Why are you worried? The only symptom you describe is that your free memory is
decreasing. This is absolutely normal. The operating system doesn't like RAM
that is sitting down doing nothing, so it will do its best to make free RAM
nearly zero. Whenever there's much RAM available, it uses more for its caches.

How much memory is your Django app consuming? You can find out by executing
"top" and pressing "M" to sort by memory usage.

Regards,

Antonis

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 14:04, Etienne Robillard wrote:
> Hi all,
>
> I'm struggling to understand how django/python may allocate and unallocate
> memory when used with uWSGI.
>
> I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 2 
> CPUs.
>
> Is that enough RAM memory for a uWSGI/Gevent based WSGI app running Django?
>
> I'm running uWSGI with the --gevent switch in order to allow cooperative
> multithreading, but my free RAM memory is always decreasing when nginx is
> running.
>
> How can I debug memory allocation in a Django/uWSGI app?
>
> I defined also in my sitecustomize.py "gc.enable()" to allow garbage
> collection, but it does not appears to make any differences.
>
> Can you recommend any libraries to debug/profile memory allocation in Python
> 2.7 ?
>
> Is Django more memory efficient with --pymalloc or by using the default linux
> malloc() ?
>
> Thank you in advance,
>
> Etienne
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/f58de0a9-8a9d-cbf5-e7c9-99b898d657a2%40djangodeployment.com.
For more options, visit https://groups.google.com/d/optout.

Re: memory leak in django app?

2017-12-06 Thread Etienne Robillard

Hi Antonis,

Thank you for your reply. I installed the htop utility and found that 2
of my 4 uWSGI processes are using 882M (42.7%) of resident memory each.
Theses two processes takes about (85%) of the available RAM memory!
That can explain why I get "out of memory" errors when no more memory is
available for sshd.

I tried to debug memory allocation with guppy following instructions
here: https://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/

But I get "hp.Nothing" when I attempt to get the heap stats from the
master uWSGI process.

Example:
>>> hp.setref()
>>> hp.heap()
hp.Nothing

Any help would be appreciated!

Etienne

Le 2017-12-06 à 07:53, Antonis Christofides a écrit :

Hello,

the amount of memory you need depends on what Django does and how many workers
(instances of Django) you run (which usually depends on how many requests you
are getting and how I/O intensive your Django application is). For many
applications, 512 MB is enough.

Why are you worried? The only symptom you describe is that your free memory is
decreasing. This is absolutely normal. The operating system doesn't like RAM
that is sitting down doing nothing, so it will do its best to make free RAM
nearly zero. Whenever there's much RAM available, it uses more for its caches.

How much memory is your Django app consuming? You can find out by executing
"top" and pressing "M" to sort by memory usage.

Regards,

Antonis

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 14:04, Etienne Robillard wrote:

Hi all,

I'm struggling to understand how django/python may allocate and unallocate
memory when used with uWSGI.

I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 2 CPUs.

Is that enough RAM memory for a uWSGI/Gevent based WSGI app running Django?

I'm running uWSGI with the --gevent switch in order to allow cooperative
multithreading, but my free RAM memory is always decreasing when nginx is
running.

How can I debug memory allocation in a Django/uWSGI app?

I defined also in my sitecustomize.py "gc.enable()" to allow garbage
collection, but it does not appears to make any differences.

Can you recommend any libraries to debug/profile memory allocation in Python
2.7 ?

Is Django more memory efficient with --pymalloc or by using the default linux
malloc() ?

Thank you in advance,

Etienne

--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/

--
You received this message because you are subscribed to the Google Groups "Django
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/f12958ba-acaa-6bf3-1792-50f260ad6b63%40yandex.com.
For more options, visit https://groups.google.com/d/optout.

Re: Strange query when using annotate and count

2017-12-06 Thread Cristiano Coelho

After testing for a while, there really doesn't seem to be a good way to do 
this. .

Annotation is required since the query is filtered based on the annotated 
value, so any attempt to clear annotations would fail. Although a lookup 
could be used for the filtering.
But a custom lookup is not possible since the query also sorts by the 
annotated value, so an annotation is required.

This means that I would need to use a custom lookup for count queries, and 
annotation for result queries, which would mean a lot of duplicated code 
and also that the Django Paginator class can't be used since it expects 
only one queryset.


Are there really no options to make a count with a custom function to skip 
the group by clause (and so the inner query)? I mean there's no group by at 
all with the original query, it's just when count is used.

To summarize, the original query is a bit more complex than the one 
described on top, reason annotate is needed, and it looks more like:

q = 
Vulnerability.objects.annotate(score=WordTrigramCustomSimilarity('test','summary')).filter(score__gt=0.6).order_by('-score')

Which is then sent to a paginator, that ends up almost adding 3 times more 
database query time do to the horrid count query generated.


El viernes, 24 de noviembre de 2017, 18:04:21 (UTC-3), Cristiano Coelho 
escribió:
>
> Hello Simon,
>
> You are right, the score is really not meant to be attached so a custom 
> lookup might work, if it wasn't for the issue that an order_by clause would 
> fail without the annotation. 
>
> So it seems it's either update and duplicate a lot of code or find a very 
> ugly work around, I was hoping I missed some flag or implementation detail 
> on the custom function to tell the query builder to avoid all the grouping 
> and subquery stuff, will research more...
>
>
> El viernes, 24 de noviembre de 2017, 14:39:27 (UTC-3), Simon Charette 
> escribió:
>>
>> Hello Cristiano,
>>
>> If you are solely using this annotation for querying purpose, that means 
>> you are
>> not expecting a `score` attribute to be attached to your `Vulnerability` 
>> instances,
>> you could try defining your function as a lookup on CharField/TextField 
>> instead.
>>
>> You could then use it only when required by passing a tuple of values to 
>> it.
>>
>> e.g.
>>
>> Vulnerability.objects.filter(summary__wordsimilarity_gt=('test', 
>> threshold))
>>
>> If you need the value annotated I'm afraid the best solution would be to 
>> use
>> a custom paginator class. You could give a try at detecting whether or not
>> an annotation is referenced before removing it from query.annotations and
>> report your finding on the aforementioned ticket but I expect this to be
>> relatively hard to do correctly. Still it would make implementing 
>> `alias()`
>> way easier and provide help for users in the situation as you.
>>
>> Best,
>> Simon
>>
>> Le vendredi 24 novembre 2017 08:14:10 UTC-5, Cristiano Coelho a écrit :
>>>
>>> Hello Simon,
>>>
>>> That private API is good to know, but now that I think of it would still 
>>> not work for me, since my queryset is passed to a paginator and that's the 
>>> class that does both the count and actual queryset execution, so need a 
>>> queryset that can have both the annotation but also clears it if count is 
>>> called so it doesn't create the redundant sub queries.
>>>
>>> I'm wondering what's better, should I try to resolve this at the manager 
>>> level overriding count? I feel like this might cause issues if the 
>>> annotation is actually legit (ie with an aggregate) and it needs the 
>>> subquery after all.
>>> The other option is to subclass the paginator class with a special one 
>>> that does this annotation clearing before running count.
>>>
>>> Even with these cases, if the annotated value is used later with a 
>>> filter query I can't really simply removed, but the sub queries and extra 
>>> function calls really doesn't make sense to be there when doing a count, so 
>>> it seems that all the options are quite bad and hackish.
>>>
>>> Any other options?
>>>
>>>
>>> El viernes, 24 de noviembre de 2017, 1:12:07 (UTC-3), Simon Charette 
>>> escribió:

 Hello Cristiano,

 > Isn't there a way (even if it's hackish) to simply clear the 
 annotations (and order by)? I know querysets are smart enough to not 
 include the order by clause if there's a count.

 The ordering is cleared on `count()` because it shouldn't interfere 
 with the result in any way. You can clear annotations using 
 `queryset.query.annotations.clear()` but be aware that it is a private API 
 that could change under your feet. Make sure to make a copy of the 
 queryset 
 (e.g. copy = queryset.all()) before performing this change as it will 
 alter 
 the queryset in place.

 Best,
 Simon
  
 Le jeudi 23 novembre 2017 22:41:41 UTC-5, Cristiano Coelho a écrit :
>
> Hello Simon, thanks for the response.
>
> The above c

RE: django 2.0 url to path

2017-12-06 Thread Matthew Pava

Is settings.DEBUG set to True?

From: django-users@googlegroups.com [mailto:django-users@googlegroups.com] On 
Behalf Of ???
Sent: Tuesday, December 5, 2017 9:00 PM
To: Django users
Subject: django 2.0 url to path

i try to use path instead of url
but there are some problem

following is my code


oauth2_endpoint_views = ([
   path('authorize/', oauth2_views.AuthorizationView.as_view(), 
name='authorize'),
   path('token/', oauth2_views.TokenView.as_view(), name='token'),
   path('revoke-token/', oauth2_views.RevokeTokenView.as_view, 
name='revoke-token'),
], 'oauth2')



if settings.DEBUG:



   oauth2_endpoint_views += ([
   path('application/', oauth2_views.ApplicationList.as_view(), 
name='list'),
   path('application/register', 
oauth2_views.ApplicationRegistration.as_view(), name='register'),
   path('application//', oauth2_views.ApplicationDetail.as_view(), 
name='detail'),
   path('application//delete/', 
oauth2_views.ApplicationDetail.as_view(), name='delete'),
   path('application//update/', 
oauth2_views.ApplicationUpdate.as_view(), name='update'),
   path('authorized-tokens/', 
oauth2_views.AuthorizedTokensListView.as_view(), name='authorized-token-list'),
   path('authorized-tokens//delete/', 
oauth2_views.AuthorizedTokenDeleteView.as_view(), 
name='authorized-token-delete'),
   ], 'oauth2')





but i can't found page application, application/register/ 

how can i solve the problem...

plz help me


--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
django-users+unsubscr...@googlegroups.com.
To post to this group, send email to 
django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0dcd4fe4-baa5-4850-9e2b-5f87fc2b5d50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2bfa7f0d9dae4d699a096b17a5b793c2%40ISS1.ISS.LOCAL.
For more options, visit https://groups.google.com/d/optout.

RE: django 2.0 url to path

2017-12-06 Thread Matthew Pava

Oh, and it looks like you need to add a “/” to the end of your 
'application/register' path.

From: django-users@googlegroups.com [mailto:django-users@googlegroups.com] On 
Behalf Of Matthew Pava
Sent: Wednesday, December 6, 2017 8:26 AM
To: 'django-users@googlegroups.com'
Subject: RE: django 2.0 url to path

Is settings.DEBUG set to True?

From: django-users@googlegroups.com 
[mailto:django-users@googlegroups.com] On Behalf Of ???
Sent: Tuesday, December 5, 2017 9:00 PM
To: Django users
Subject: django 2.0 url to path

i try to use path instead of url
but there are some problem

following is my code


oauth2_endpoint_views = ([
   path('authorize/', oauth2_views.AuthorizationView.as_view(), 
name='authorize'),
   path('token/', oauth2_views.TokenView.as_view(), name='token'),
   path('revoke-token/', oauth2_views.RevokeTokenView.as_view, 
name='revoke-token'),
], 'oauth2')



if settings.DEBUG:



   oauth2_endpoint_views += ([
   path('application/', oauth2_views.ApplicationList.as_view(), 
name='list'),
   path('application/register', 
oauth2_views.ApplicationRegistration.as_view(), name='register'),
   path('application//', oauth2_views.ApplicationDetail.as_view(), 
name='detail'),
   path('application//delete/', 
oauth2_views.ApplicationDetail.as_view(), name='delete'),
   path('application//update/', 
oauth2_views.ApplicationUpdate.as_view(), name='update'),
   path('authorized-tokens/', 
oauth2_views.AuthorizedTokensListView.as_view(), name='authorized-token-list'),
   path('authorized-tokens//delete/', 
oauth2_views.AuthorizedTokenDeleteView.as_view(), 
name='authorized-token-delete'),
   ], 'oauth2')





but i can't found page application, application/register/ 

how can i solve the problem...

plz help me


--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
django-users+unsubscr...@googlegroups.com.
To post to this group, send email to 
django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0dcd4fe4-baa5-4850-9e2b-5f87fc2b5d50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
django-users+unsubscr...@googlegroups.com.
To post to this group, send email to 
django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2bfa7f0d9dae4d699a096b17a5b793c2%40ISS1.ISS.LOCAL.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/da2a5e3d191945fdb679fe2f1f27edce%40ISS1.ISS.LOCAL.
For more options, visit https://groups.google.com/d/optout.

Re: memory leak in django app?

2017-12-06 Thread Antonis Christofides

Does this happen only in production? What about when you run a development
server? What is the memory usage of your development server?

Antonis Christofides
http://djangodeployment.com


On 2017-12-06 15:05, Etienne Robillard wrote:
> Hi Antonis,
>
> Thank you for your reply. I installed the htop utility and found that 2 of my
> 4 uWSGI processes are using 882M (42.7%) of resident memory each. Theses two
> processes takes about  (85%) of the available RAM memory! That can explain why
> I get "out of memory" errors when no more memory is available for sshd.
>
> I tried to debug memory allocation with guppy following instructions here:
> https://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/
>
> But I get "hp.Nothing" when I attempt to get the heap stats from the master
> uWSGI process.
>
> Example:
> >>> hp.setref()
> >>> hp.heap()
> hp.Nothing
>
> Any help would be appreciated!
>
> Etienne
>
>
> Le 2017-12-06 à 07:53, Antonis Christofides a écrit :
>> Hello,
>>
>> the amount of memory you need depends on what Django does and how many 
>> workers
>> (instances of Django) you run (which usually depends on how many requests you
>> are getting and how I/O intensive your Django application is). For many
>> applications, 512 MB is enough.
>>
>> Why are you worried? The only symptom you describe is that your free memory 
>> is
>> decreasing. This is absolutely normal. The operating system doesn't like RAM
>> that is sitting down doing nothing, so it will do its best to make free RAM
>> nearly zero. Whenever there's much RAM available, it uses more for its 
>> caches.
>>
>> How much memory is your Django app consuming? You can find out by executing
>> "top" and pressing "M" to sort by memory usage.
>>
>> Regards,
>>
>> Antonis
>>
>> Antonis Christofides
>> http://djangodeployment.com
>>
>> On 2017-12-06 14:04, Etienne Robillard wrote:
>>> Hi all,
>>>
>>> I'm struggling to understand how django/python may allocate and unallocate
>>> memory when used with uWSGI.
>>>
>>> I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 2 
>>> CPUs.
>>>
>>> Is that enough RAM memory for a uWSGI/Gevent based WSGI app running Django?
>>>
>>> I'm running uWSGI with the --gevent switch in order to allow cooperative
>>> multithreading, but my free RAM memory is always decreasing when nginx is
>>> running.
>>>
>>> How can I debug memory allocation in a Django/uWSGI app?
>>>
>>> I defined also in my sitecustomize.py "gc.enable()" to allow garbage
>>> collection, but it does not appears to make any differences.
>>>
>>> Can you recommend any libraries to debug/profile memory allocation in Python
>>> 2.7 ?
>>>
>>> Is Django more memory efficient with --pymalloc or by using the default 
>>> linux
>>> malloc() ?
>>>
>>> Thank you in advance,
>>>
>>> Etienne
>>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0c2788c4-d6c2-3e5c-1eb4-222a3a9620ce%40djangodeployment.com.
For more options, visit https://groups.google.com/d/optout.

Re: memory leak in django app?

2017-12-06 Thread Etienne Robillard

Hi Antonis,

My development server appears unaffected by this problem. Plus I can get
heap stats using guppy, which is pretty cool. :)

Both my production and development servers have __debug__ enabled in
Python 2.7.13.

However when using ab -c 100 to benchmark my nginx server I get:

SSL handshake failed (1).

Do you have any ideas how to configure nginx to allow a minimum of 100
concurrent connections when using SSL encryption?

I have worker_connections set to 512 in my nginx.conf

Regards,

Etienne

Le 2017-12-06 à 12:17, Antonis Christofides a écrit :

Does this happen only in production? What about when you run a development
server? What is the memory usage of your development server?

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 15:05, Etienne Robillard wrote:

Hi Antonis,

Thank you for your reply. I installed the htop utility and found that 2 of my
4 uWSGI processes are using 882M (42.7%) of resident memory each. Theses two
processes takes about (85%) of the available RAM memory! That can explain why
I get "out of memory" errors when no more memory is available for sshd.

I tried to debug memory allocation with guppy following instructions here:
https://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/

But I get "hp.Nothing" when I attempt to get the heap stats from the master
uWSGI process.

Example:

hp.setref()
hp.heap()

hp.Nothing

Any help would be appreciated!

Etienne

Le 2017-12-06 à 07:53, Antonis Christofides a écrit :

Hello,

How much memory is your Django app consuming? You can find out by executing
"top" and pressing "M" to sort by memory usage.

Regards,

Antonis

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 14:04, Etienne Robillard wrote:

Hi all,

I'm struggling to understand how django/python may allocate and unallocate
memory when used with uWSGI.

I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 2 CPUs.

Is that enough RAM memory for a uWSGI/Gevent based WSGI app running Django?

I'm running uWSGI with the --gevent switch in order to allow cooperative
multithreading, but my free RAM memory is always decreasing when nginx is
running.

How can I debug memory allocation in a Django/uWSGI app?

I defined also in my sitecustomize.py "gc.enable()" to allow garbage
collection, but it does not appears to make any differences.

Can you recommend any libraries to debug/profile memory allocation in Python
2.7 ?

Is Django more memory efficient with --pymalloc or by using the default linux
malloc() ?

Thank you in advance,

Etienne

--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/

--
You received this message because you are subscribed to the Google Groups "Django
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/d9039dfa-c3b0-955b-44fc-b42cc7e49fd5%40yandex.com.
For more options, visit https://groups.google.com/d/optout.

Re: Strange query when using annotate and count

2017-12-06 Thread Cristiano Coelho

Update. Found a work around that gets rid of the unnecessary group by (and 
hence speeds up the query, from ~200ms to ~100ms in my use case). Simply 
adding a .values('pk').
Bad thing, every model needs a custom manager and it will still use an 
inner query, still trying to figure out the side effects of it, nothing so 
far...

class FasterCountQuerySet(QuerySet):
> def count(self):
> return super(FasterCountQuerySet, self.values('pk')).count()
>


Makes me wonder if the original count code can be made "smarter" in order 
to prevent the unnecessary group by, or if this simply a bug


El miércoles, 6 de diciembre de 2017, 10:28:22 (UTC-3), Cristiano Coelho 
escribió:
>
> After testing for a while, there really doesn't seem to be a good way to 
> do this. .
>
> Annotation is required since the query is filtered based on the annotated 
> value, so any attempt to clear annotations would fail. Although a lookup 
> could be used for the filtering.
> But a custom lookup is not possible since the query also sorts by the 
> annotated value, so an annotation is required.
>
> This means that I would need to use a custom lookup for count queries, and 
> annotation for result queries, which would mean a lot of duplicated code 
> and also that the Django Paginator class can't be used since it expects 
> only one queryset.
>
>
> Are there really no options to make a count with a custom function to skip 
> the group by clause (and so the inner query)? I mean there's no group by at 
> all with the original query, it's just when count is used.
>
> To summarize, the original query is a bit more complex than the one 
> described on top, reason annotate is needed, and it looks more like:
>
> q = 
> Vulnerability.objects.annotate(score=WordTrigramCustomSimilarity('test','summary')).filter(score__gt=0.6).order_by('-score')
>
> Which is then sent to a paginator, that ends up almost adding 3 times more 
> database query time do to the horrid count query generated.
>
>
> El viernes, 24 de noviembre de 2017, 18:04:21 (UTC-3), Cristiano Coelho 
> escribió:
>>
>> Hello Simon,
>>
>> You are right, the score is really not meant to be attached so a custom 
>> lookup might work, if it wasn't for the issue that an order_by clause would 
>> fail without the annotation. 
>>
>> So it seems it's either update and duplicate a lot of code or find a very 
>> ugly work around, I was hoping I missed some flag or implementation detail 
>> on the custom function to tell the query builder to avoid all the grouping 
>> and subquery stuff, will research more...
>>
>>
>> El viernes, 24 de noviembre de 2017, 14:39:27 (UTC-3), Simon Charette 
>> escribió:
>>>
>>> Hello Cristiano,
>>>
>>> If you are solely using this annotation for querying purpose, that means 
>>> you are
>>> not expecting a `score` attribute to be attached to your `Vulnerability` 
>>> instances,
>>> you could try defining your function as a lookup on CharField/TextField 
>>> instead.
>>>
>>> You could then use it only when required by passing a tuple of values to 
>>> it.
>>>
>>> e.g.
>>>
>>> Vulnerability.objects.filter(summary__wordsimilarity_gt=('test', 
>>> threshold))
>>>
>>> If you need the value annotated I'm afraid the best solution would be to 
>>> use
>>> a custom paginator class. You could give a try at detecting whether or 
>>> not
>>> an annotation is referenced before removing it from query.annotations and
>>> report your finding on the aforementioned ticket but I expect this to be
>>> relatively hard to do correctly. Still it would make implementing 
>>> `alias()`
>>> way easier and provide help for users in the situation as you.
>>>
>>> Best,
>>> Simon
>>>
>>> Le vendredi 24 novembre 2017 08:14:10 UTC-5, Cristiano Coelho a écrit :

 Hello Simon,

 That private API is good to know, but now that I think of it would 
 still not work for me, since my queryset is passed to a paginator and 
 that's the class that does both the count and actual queryset execution, 
 so 
 need a queryset that can have both the annotation but also clears it if 
 count is called so it doesn't create the redundant sub queries.

 I'm wondering what's better, should I try to resolve this at the 
 manager level overriding count? I feel like this might cause issues if the 
 annotation is actually legit (ie with an aggregate) and it needs the 
 subquery after all.
 The other option is to subclass the paginator class with a special one 
 that does this annotation clearing before running count.

 Even with these cases, if the annotated value is used later with a 
 filter query I can't really simply removed, but the sub queries and extra 
 function calls really doesn't make sense to be there when doing a count, 
 so 
 it seems that all the options are quite bad and hackish.

 Any other options?


 El viernes, 24 de noviembre de 2017, 1:12:07 (UTC-3), Simon Charette 
 escribió:
>
>>>

Re: Polls website from documentation

2017-12-06 Thread Langton Chikukwa

It worked thank you


On Wed, Dec 6, 2017 at 2:22 AM, Langton Chikukwa 
wrote:

> Thank you let me try that
>
>
>
>
>
> *From: *yingi keme 
> *Sent: *Wednesday, December 6, 2017 2:19 AM
> *To: *django-users@googlegroups.com
> *Subject: *Re: Polls website from documentation
>
>
>
> Your url is not pointing to the url pattern specified
>
>
>
> Try this:
>
>
>
> http://localhost:8000/polls
>
>
>
> What u are doing is http://localhost:8000/admin/polls.
>
>
>
> Then i noticed you deleted the admin url pattern in you urls.py file. You
> will definitely need to use the admin page pretty soon.
>
> Yingi Kem
>
>
> On 5 Dec 2017, at 11:22 PM, lmchikukwa  wrote:
>
> I am a beginner in the django framework. I am having a problem with a page
> not found error. The code for both the urls' files are attached and the
> output from the site is show.
>
> Please help me understand what I dd wrong because I do not know.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/91994018-827d-49f0-b920-63332ffe90bb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/48C6CBC9-2084-4ECF-9A1F-4B55133A3610%40gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAOf6RczEwpbSpD3erYQvtNX523mK5xTzcNBnzHWn4vZsanvk0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Django - Paginate with another GET request; not working with just pagination page number

2017-12-06 Thread Jack

I'm building a page with pagination and a filter form (2 GET requests).  If 
the URL includes both pagination and filter results, something like 
`/questions/?page=2&all_questions=on`, it works fine.  It also works if it 
just has filter results, something like `/questions/?all_questions=on`.

However if it only has the pagination page result, something like 
`/questions/?page=1`, no results are shown.

So I figured I need to do something in the views so that if there is only 
page number in the URL, a default filter will be given.  I know I probably 
need to add something to the Try and Except section of pagination, but I'm 
stumped as the actual code I need to write.


def questions_index(request):

user = request.user
form = QuestionFilterForm(request.GET or None)
question_list = []
if not form.is_bound:
question_list = Question.objects.all().order_by('-date_created')
if form.is_valid():
if form.cleaned_data['all_questions'] | 
(form.cleaned_data['general_questions'] & 
form.cleaned_data['location_all_gta']) == True:
question_list = 
Question.objects.all().order_by('-date_created')
elif form.cleaned_data['location_all_gta'] == True:
question_list += 
Question.objects.filter(question_type=1).order_by('-date_created')
else:
if form.cleaned_data['general_questions'] == True:
question_list += 
Question.objects.filter(question_type=2).order_by('-date_created')
if form.cleaned_data['location_toronto'] == True:
question_list += 
Question.objects.filter(location='1').order_by('-date_created')

paginator = Paginator(question_list, 15)

# Pagination
page = request.GET.get('page')
try:
questions = paginator.page(page)
except PageNotAnInteger:
# If page is not an integer, deliver first page.
questions = paginator.page(1)
except EmptyPage:
# If page is out of range (e.g. ), deliver last page of 
results.
questions = paginator.page(paginator.num_pages)

### I need to write something here...
except (url has no filter result)  
give default filter 

return render(request, 'questions_index.html', {'questions': 
questions, 'user': user, 'form': form})

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2a2f05ae-d71a-419b-9df0-394e1e129fcb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: memory leak in django app?

2017-12-06 Thread Antonis Christofides

The "SSL handshake failed" doesn't seem related to the alleged memory leak.

Now, you say you have this problem only in production. If you restart uwsgi,
does memory usage go immediately up? Does it go after a few requests? How many
requests? Do you have the same data in development and production? If, for
example, your app loads a whole db table in memory, and this db table has a
million records in production and only a few in development, there would be a
difference.

Regards,

Antonis

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 19:38, Etienne Robillard wrote:
> Hi Antonis,
>
> My development server appears unaffected by this problem. Plus I can get heap
> stats using guppy, which is pretty cool. :)
>
> Both my production and development servers have __debug__ enabled in Python
> 2.7.13.
>
> However when using ab -c 100 to benchmark my nginx server I get:
>
> SSL handshake failed (1).
>
> Do you have any ideas how to configure nginx to allow a minimum of 100
> concurrent connections when using SSL encryption?
>
> I have worker_connections set to 512 in my nginx.conf
>
> Regards,
>
> Etienne
>
>
> Le 2017-12-06 à 12:17, Antonis Christofides a écrit :
>> Does this happen only in production? What about when you run a development
>> server? What is the memory usage of your development server?
>>
>> Antonis Christofides
>> http://djangodeployment.com
>>
>>
>> On 2017-12-06 15:05, Etienne Robillard wrote:
>>> Hi Antonis,
>>>
>>> Thank you for your reply. I installed the htop utility and found that 2 of 
>>> my
>>> 4 uWSGI processes are using 882M (42.7%) of resident memory each. Theses two
>>> processes takes about  (85%) of the available RAM memory! That can explain 
>>> why
>>> I get "out of memory" errors when no more memory is available for sshd.
>>>
>>> I tried to debug memory allocation with guppy following instructions here:
>>> https://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/
>>>
>>> But I get "hp.Nothing" when I attempt to get the heap stats from the master
>>> uWSGI process.
>>>
>>> Example:
>> hp.setref()
>> hp.heap()
>>> hp.Nothing
>>>
>>> Any help would be appreciated!
>>>
>>> Etienne
>>>
>>>
>>> Le 2017-12-06 à 07:53, Antonis Christofides a écrit :
 Hello,

 the amount of memory you need depends on what Django does and how many 
 workers
 (instances of Django) you run (which usually depends on how many requests 
 you
 are getting and how I/O intensive your Django application is). For many
 applications, 512 MB is enough.

 Why are you worried? The only symptom you describe is that your free 
 memory is
 decreasing. This is absolutely normal. The operating system doesn't like 
 RAM
 that is sitting down doing nothing, so it will do its best to make free RAM
 nearly zero. Whenever there's much RAM available, it uses more for its 
 caches.

 How much memory is your Django app consuming? You can find out by executing
 "top" and pressing "M" to sort by memory usage.

 Regards,

 Antonis

 Antonis Christofides
 http://djangodeployment.com

 On 2017-12-06 14:04, Etienne Robillard wrote:
> Hi all,
>
> I'm struggling to understand how django/python may allocate and unallocate
> memory when used with uWSGI.
>
> I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 2
> CPUs.
>
> Is that enough RAM memory for a uWSGI/Gevent based WSGI app running 
> Django?
>
> I'm running uWSGI with the --gevent switch in order to allow cooperative
> multithreading, but my free RAM memory is always decreasing when nginx is
> running.
>
> How can I debug memory allocation in a Django/uWSGI app?
>
> I defined also in my sitecustomize.py "gc.enable()" to allow garbage
> collection, but it does not appears to make any differences.
>
> Can you recommend any libraries to debug/profile memory allocation in 
> Python
> 2.7 ?
>
> Is Django more memory efficient with --pymalloc or by using the default 
> linux
> malloc() ?
>
> Thank you in advance,
>
> Etienne
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4f43e326-313f-7808-09fc-c7087b46b0a7%40djangodeployment.com.
For more options, visit https://groups.google.com/d/optout.

Re: memory leak in django app?

2017-12-06 Thread Etienne Robillard

Hi Antonis,

Le 2017-12-06 à 16:23, Antonis Christofides a écrit :

The "SSL handshake failed" doesn't seem related to the alleged memory leak.
Correct. I forgot to specify which SSL protocol to use. Using ab -f
TLSV1 solved that problem.

Now, you say you have this problem only in production. If you restart uwsgi,
does memory usage go immediately up? Does it go after a few requests? How many
requests? Do you have the same data in development and production? If, for
example, your app loads a whole db table in memory, and this db table has a
million records in production and only a few in development, there would be a
difference.
The problem started to appear in production after a few days without
restarting the main uWSGI process.

Also, the database in use is ZODB and is not affected by this issue at all!

So, I'm guessing this is a problem with uWSGI, not Django. For example,
running the development server with wsgiref on my local host do not take
more than 3.5% of rss memory with DEBUG=True.

Regards,

Etienne

Regards,

Antonis

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 19:38, Etienne Robillard wrote:

Hi Antonis,

My development server appears unaffected by this problem. Plus I can get heap
stats using guppy, which is pretty cool. :)

Both my production and development servers have __debug__ enabled in Python
2.7.13.

However when using ab -c 100 to benchmark my nginx server I get:

SSL handshake failed (1).

Do you have any ideas how to configure nginx to allow a minimum of 100
concurrent connections when using SSL encryption?

I have worker_connections set to 512 in my nginx.conf

Regards,

Etienne

Le 2017-12-06 à 12:17, Antonis Christofides a écrit :

Does this happen only in production? What about when you run a development
server? What is the memory usage of your development server?

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 15:05, Etienne Robillard wrote:

Hi Antonis,

Thank you for your reply. I installed the htop utility and found that 2 of my
4 uWSGI processes are using 882M (42.7%) of resident memory each. Theses two
processes takes about (85%) of the available RAM memory! That can explain why
I get "out of memory" errors when no more memory is available for sshd.

I tried to debug memory allocation with guppy following instructions here:
https://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/

But I get "hp.Nothing" when I attempt to get the heap stats from the master
uWSGI process.

Example:

hp.setref()
hp.heap()

hp.Nothing

Any help would be appreciated!

Etienne

Le 2017-12-06 à 07:53, Antonis Christofides a écrit :

Hello,

How much memory is your Django app consuming? You can find out by executing
"top" and pressing "M" to sort by memory usage.

Regards,

Antonis

Antonis Christofides
http://djangodeployment.com

On 2017-12-06 14:04, Etienne Robillard wrote:

Hi all,

I'm struggling to understand how django/python may allocate and unallocate
memory when used with uWSGI.

I have a Debian system running Python 2.7 and uwsgi with 2GB of RAM and 2
CPUs.

Is that enough RAM memory for a uWSGI/Gevent based WSGI app running Django?

I'm running uWSGI with the --gevent switch in order to allow cooperative
multithreading, but my free RAM memory is always decreasing when nginx is
running.

How can I debug memory allocation in a Django/uWSGI app?

I defined also in my sitecustomize.py "gc.enable()" to allow garbage
collection, but it does not appears to make any differences.

Can you recommend any libraries to debug/profile memory allocation in Python
2.7 ?

Is Django more memory efficient with --pymalloc or by using the default linux
malloc() ?

Thank you in advance,

Etienne

--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/

--
You received this message because you are subscribed to the Google Groups "Django
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/c8941fcd-871e-cc41-c2b3-3d07c6a0d193%40yandex.com.
For more options, visit https://groups.google.com/d/optout.

ValueError source code string cannot contain null bytes

2017-12-06 Thread Rajeshwaran

Hi All,

I am facing this "ValueError: source code string cannot contain null bytes" 
error when i create a sqlite3 database and run the application.

Steps that i do :


   1. Create database using db browser for SQLITE3
   2. Create model by running "python manage.py inspectdb > models.py"
   3. Run the application by "python manage.py runserver"
   
Could someone please help me with this issue.

Thanks,
Rajeshwaran

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0a7812b2-f788-4eaf-8c1a-663e573a82a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ValueError: source code string cannot contain null bytes

2017-12-06 Thread Rajeshwaran

Hi All,

I am facing this "ValueError: source code string cannot contain null bytes" 
error when ever i create a splite3 database and use it.

Below are the steps i follow.

   1. Create the database using db browser.
   2. Run python "manage.py inspectdb > models.py"
   3. I run the server with "python manage.py runserver"
   
I get this error when I run the server. Could someone please help me in 
this.

Thanks,
Rajeshwaran

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/7c1a9275-a5d5-4a9f-bc31-52cb256afb73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

memory leak in django app?

Re: Enable https for a production django application

Re: memory leak in django app?

Re: memory leak in django app?

Re: Strange query when using annotate and count

RE: django 2.0 url to path

RE: django 2.0 url to path

Re: memory leak in django app?

Re: memory leak in django app?

Re: Strange query when using annotate and count

Re: Polls website from documentation

Django - Paginate with another GET request; not working with just pagination page number

Re: memory leak in django app?

Re: memory leak in django app?

ValueError source code string cannot contain null bytes

ValueError: source code string cannot contain null bytes

16 matches

Site Navigation

Mail list logo

Footer information