Re: Scaling/Parallel patterns for a View executing complex database transaction

[Stephen J. Butler]

If it's just key-value storage, adding a memcached layer sounds like a
good thing to investigate. Do the tuples frequently change?

On Thu, May 14, 2015 at 11:30 PM, Me Sulphur  wrote:
> Hi Russ,
>
> Thanks! While there are some pointers that we can pick up from your answer,
> let me add some more details.
>
> * It is a large key-value pair table (approaching ~ 1 bn rows) with an MD5
> key and JSON  for value. The look ups depend on the business logic but are
> necessary. Nevertheless, there are no more than 10-12 queries executed by
> combining key look ups using WHERE IN queries and pg is surprisingly good
> with IN queries.
>
> * Pre-computation is theoretically possible but the permutations can be very
> limiting. Over the top of my head, I think it will be M^n where M and n are
> very large (and growing) integers.
>
> * You are right, celery+polling can be done but the API is already deployed
> and in use in environments where there is strong resistance to rework a code
> base which is stable and changing them (consumers of the API) is beyond
> control.
>
>
> Thanks!
>
>
> On Friday, 15 May 2015 06:06:57 UTC+5:30, Russell Keith-Magee wrote:
>>
>>
>> On Thu, May 14, 2015 at 6:03 PM, Me Sulphur  wrote:
>>>
>>> Stack: Django 1.7 + Postgres 9.3 + Linux (No caching)
>>>
>>> Our application has a view which is called/executed very frequently. The
>>> view receives, parses and responds a JSON.
>>>
>>> In between request and response, there are about 3-5 inserts and around
>>> 1200-5000 look ups depending upon some if..else business logic. At around
>>> 2-4 seconds the view is very slow.
>>>
>>> However, a lot of the look ups (which are the bottlenecks) can be
>>> parallelized. But I do not know how can I do the same within a
>>> request-response cycle.
>>>
>>> If it was a web UI, I could use celery+polling, since it a
>>> machine-machine API call, the parallelisation has to be possible within a
>>> View's life cycle.
>>>
>>> If parallelisation is not possible, what alternatives do I have for
>>> scaling and reducing response time.
>>>
>> The short answer is "it depends".
>>
>> There isn't a single answer - everything will depend on the specifics of
>> your problem space. All I can offer is some vague suggestions of places you
>> might be able to look for some extra speed.
>>
>>  * Do you *really* need to do 1200-5000 lookups? It's faster to do 1 query
>> returning 10 rows than 10 queries returning 1 row each. Can you optimise the
>> queries on the database to minimise the number of queries needed? Depending
>> on circumstances, it may even be faster to do 1 query returning 15 rows, and
>> then post-process in the view to throw away the 5 rows you don't need.
>>
>>  * Better still - can you optimize the database structure so that
>> 1200-5000 calls aren't needed? Modern relational databases are *really* good
>> at query optimisation - if you give it the right query, and the right
>> database.
>>
>>  * Can you work your algorithm another way? In a previous job, I worked on
>> a tool that would look at a database of thousands of news articles received
>> in a given day, and then, for each of thousands of users, work out which
>> articles were "interesting" so they could be sent out in a daily alert
>> email. The obvious algorithm for this is "for user in users:
>> find_articles(user)" - but, for a variety of reasons, it turned out that
>> doing "for article in articles: find_users(article)" was almost 2 orders of
>> magnitude of faster. The less obvious algorithm allowed much greater
>> caching, and massively cut down the number of queries that were required.
>> The tradeoff was a lot more memory (memory vs speed is almost always the
>> tradeoff), and it wasn't only faster if you computed the results for *all*
>> users at the same time - but this was an daily offline process, so these
>> were limitations we were willing to accept.
>>
>>  * To that end - is there anything that can be precomputed? Can you cache
>> pieces of the response? Is there anything you can put into a memory store,
>> rather than the database. Databases are great, but if you have a small
>> amount of frequently re-used, easily keyed data, it may be better to put
>> that data into a location where it can be obtained quickly, rather than
>> hitting the database.
>>
>>  * If you *must* parallelize, and your algorithm is conducive to it,
>> threads are probably your best option - work out what part of your algorithm
>> can be parallelized, and put each part in a thread, and merge the results
>> once all the threads complete. If you're on Python 3, look into the
>> concurrent.futures module (or the "futures" module if you're on Python 2) to
>> help make this easier to manage. However, threads aren't magic fairy dust -
>> my "limited knowledge of your situation" guess is that parallelization won't
>> help you. If you've got a frequently executed view doing thousands of
>> database calls, I'm going to guess 

Re: Scaling/Parallel patterns for a View executing complex database transaction

[Me Sulphur]

Hi Russ,

Thanks! While there are some pointers that we can pick up from your answer, 
let me add some more details.

* It is a large key-value pair table (approaching ~ 1 bn rows) with an MD5 
key and JSON  for value. The look ups depend on the business logic but are 
necessary. Nevertheless, there are no more than 10-12 queries executed by 
combining key look ups using WHERE IN queries and pg is surprisingly good 
with IN queries.

* Pre-computation is theoretically possible but the permutations can be 
very limiting. Over the top of my head, I think it will be M^n where M and 
n are very large (and growing) integers.

* You are right, celery+polling can be done but the API is already deployed 
and in use in environments where there is strong resistance to rework a 
code base which is stable and changing them (consumers of the API) is 
beyond control.


Thanks!

 
On Friday, 15 May 2015 06:06:57 UTC+5:30, Russell Keith-Magee wrote:
>
>
> On Thu, May 14, 2015 at 6:03 PM, Me Sulphur  > wrote:
>
>> Stack: Django 1.7 + Postgres 9.3 + Linux (No caching)
>>
>> Our application has a view which is called/executed very frequently. The 
>> view receives, parses and responds a JSON.
>>
>> In between request and response, there are about 3-5 inserts and around 
>> 1200-5000 look ups depending upon some if..else business logic. At around 
>> 2-4 seconds the view is very slow. 
>>
>> However, a lot of the look ups (which are the bottlenecks) can be 
>> parallelized. But I do not know how can I do the same within a 
>> request-response cycle.
>>
>> If it was a web UI, I could use celery+polling, since it a 
>> machine-machine API call, the parallelisation has to be possible within a 
>> View's life cycle.
>>
>> If parallelisation is not possible, what alternatives do I have for 
>> scaling and reducing response time.
>>
>> The short answer is "it depends".
>
> There isn't a single answer - everything will depend on the specifics of 
> your problem space. All I can offer is some vague suggestions of places you 
> might be able to look for some extra speed.
>
>  * Do you *really* need to do 1200-5000 lookups? It's faster to do 1 query 
> returning 10 rows than 10 queries returning 1 row each. Can you optimise 
> the queries on the database to minimise the number of queries needed? 
> Depending on circumstances, it may even be faster to do 1 query returning 
> 15 rows, and then post-process in the view to throw away the 5 rows you 
> don't need.
>
>  * Better still - can you optimize the database structure so that 
> 1200-5000 calls aren't needed? Modern relational databases are *really* 
> good at query optimisation - if you give it the right query, and the right 
> database.
>
>  * Can you work your algorithm another way? In a previous job, I worked on 
> a tool that would look at a database of thousands of news articles received 
> in a given day, and then, for each of thousands of users, work out which 
> articles were "interesting" so they could be sent out in a daily alert 
> email. The obvious algorithm for this is "for user in users: 
> find_articles(user)" - but, for a variety of reasons, it turned out that 
> doing "for article in articles: find_users(article)" was almost 2 orders of 
> magnitude of faster. The less obvious algorithm allowed much greater 
> caching, and massively cut down the number of queries that were required. 
> The tradeoff was a lot more memory (memory vs speed is almost always the 
> tradeoff), and it wasn't only faster if you computed the results for *all* 
> users at the same time - but this was an daily offline process, so these 
> were limitations we were willing to accept.
>
>  * To that end - is there anything that can be precomputed? Can you cache 
> pieces of the response? Is there anything you can put into a memory store, 
> rather than the database. Databases are great, but if you have a small 
> amount of frequently re-used, easily keyed data, it may be better to put 
> that data into a location where it can be obtained quickly, rather than 
> hitting the database.
>
>  * If you *must* parallelize, and your algorithm is conducive to it, 
> threads are probably your best option - work out what part of your 
> algorithm can be parallelized, and put each part in a thread, and merge the 
> results once all the threads complete. If you're on Python 3, look into the 
> concurrent.futures module (or the "futures" module if you're on Python 2) 
> to help make this easier to manage. However, threads aren't magic fairy 
> dust - my "limited knowledge of your situation" guess is that 
> parallelization won't help you. If you've got a frequently executed view 
> doing thousands of database calls, I'm going to guess the database is 
> already a bit of a bottleneck; adding 10 threads per request is going to 
> increase the database load and make performance *worse*, not better, and if 
> it's a high traffic view, at some point, you're going to hit the limit of 
> the 

Re: best way change template base my app

[carlos]

Thank Andre Luis :)

On Wed, May 13, 2015 at 1:41 PM, André Luiz  wrote:

> extends should always be on first line of file and it accepts a variable
> as argument so you can define which template should be used on your view.
>
> # views.py
> def home(request):
> if request.session.type == 1:
>  base = 'base.html'
> else:
> base = 'base_other.html'
>
> context = {
> 'base': base,
> }
>
> return render(request, 'template.html', context)
>
> # template.html
> {% extends base %}
>
> if you need it on every template maybe you should use a context processor
> to define your base variable.
>
> 2015-05-13 15:11 GMT-03:00 carlos :
>
>> Hi, is posible change my base template with if tag
>> example:
>>
>> {% if request.session.type = '1' %}
>> {% extends "base.html"%}
>> {% else %}
>>{% extends "base_other.html"%}
>> {%endif%}
>>
>> i have a error like that
>>
>> TemplateSyntaxError at /monito/cult/
>>
>> Invalid block tag: 'endif'
>>
>>
>> will to do change the base template my app
>>
>> Cheers
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAM-7rO3DONtDe_d%2BHeBef446Mr%3DvSiKK7z_W8pyNyPq7Pbt3PA%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAL6LtmfZTMpfG%3Djh_uMtp63hZ%2BDsmz92_8UP81x0zuGfqp%2BO0w%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAM-7rO39yTCA%3DW7o5ovd2GGvwKpbQBkFJmf74x7iaT%2BPZXawpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Fwd: What is the ideal web server to use with Django?

[Andrew Farrell]

Ashkat,

I second the recommendation that Ashkat go with gunicorn+nginx for the same
reason Avraham does. Digital Ocean has a good walkthrough of how to set
that up here

.


One thing though is I would make sure to do it in stages when you are just
starting out. That way you always have some piece working and when some
thing isn't working you have a guess where that is:
1) First run through the tutorial with just a sqlite database and the
built-in webserver on port 8000.
2) Get nginx so that it is able to serve a static html file and you can see
its' logs (probably in /var/log/nginx)
3) Get nginx to be able to proxy_pass from port 80 to the built-in
webserver on port 8000
4) Get gunicorn able to serve on port 8000
5) Get nginx able to proxy to gunicorn on port 8000

And then you can also get the postgres database working, first by being
able to connect to it with manage.py dbshell, then by running migrations
and the built-in webserver and then the full stack.

Welcome aboard the django train,
Andrew


-- Forwarded message --
From: Avraham Serour 
Date: Thu, May 14, 2015 at 4:40 AM
Subject: Re: What is the ideal web server to use with Django?
To: django-users@googlegroups.com


My main reason for recommending nginx is the config file, they are simpler
than apache,
when developing an application you shouldn't spend much time configuring
the web server,
only after you reached so much traffic that it would make sense to spend
time with it
On May 14, 2015 12:26 PM, "Tom Evans"  wrote:

> On Thu, May 14, 2015 at 4:36 AM, reduxionist 
> wrote:
> > The question you asked Tom was "Doesn't Apache create new process for
> each
> > request [thus eating memory when serving large amounts of static files
> > during traffic peaks]?", and the reason that Tom correctly answers "No"
> is
> > that as far as "serving large amounts of static files" goes you should be
> > using mpm-worker (multi-threaded Apache) which most definitely does not
> > spawn a new process for each request.
> >
> > The reason for those search results is that mpm-prefork does, however,
> spawn
> > a process per request,
>
> No, really, it does not. It only spawns a new process when there are
> no available workers to process an incoming request, and you have not
> reached the maximum number of workers that you have configured it to
> start. You can configure it to start all the worker processes you want
> when it starts up, and never to kill them off, and it will never spawn
> a new process.
>
> Apache processes are small, unless you do daft things like embed your
> web application in each worker process (mod_php style). This is the
> main complaint "Apache is eating all my memory" - it isn't, your web
> application you've embedded into Apache is eating all your memory.
>
> All of this is irrelevant for django, because with Apache you should
> use mod_wsgi in daemon mode, which separates out your web application
> processes from the web server.
>
> > but it is only needed for non-thread-safe
> > environments (most notoriously mod_php) and you shouldn't have to use it
> as
> > long as you've been a good coder and avoided global state in your Django
> app
> > (e.g. keep request-specific shared-state thread-local).
> >
> > I think the reason a lot of people seem to run mpm-prefork is just that
> it's
> > the default multi-processing module for Apache on most (all?) *nix
> platforms
> > and they don't know any better.
>
> Quite. We run a pair of Apache 2.4 reverse proxies in front of all of
> our (400+) domains, serving around 40 million requests per day,
> providing SSL termination and static file serving. We use event MPM
> and we have it scaled to support a peak of 2048 simultaneous
> connections. Load on the server never goes above 0.2, memory usage
> never goes above 1GB for the entire OS + applications, the rest of the
> RAM is used by the OS to cache the aforementioned static files.
>
> On our app servers we typically use Apache with worker MPM and
> mod_wsgi, although we have a few nginx+uwsgi sites, and I would dearly
> love some time to play around with a circusd + chausette + celery
> setup.
>
> The choice of web server is, these days, irrelevant. If it uses too
> much memory or can't handle enough users, it is never the fault of the
> web server, but instead of your application and/or configuration.
> Which is why I return to my original advice:
>
> > I am new to Django. I am building a app which will have to handle several
> > concurrent requests. Which web server is suitable for this?
>
> Any and all.
>
> Leave the fanboyism to the phone guys.
>
> Cheers
>
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, 

Re: Scaling/Parallel patterns for a View executing complex database transaction

[Russell Keith-Magee]

On Thu, May 14, 2015 at 6:03 PM, Me Sulphur  wrote:

> Stack: Django 1.7 + Postgres 9.3 + Linux (No caching)
>
> Our application has a view which is called/executed very frequently. The
> view receives, parses and responds a JSON.
>
> In between request and response, there are about 3-5 inserts and around
> 1200-5000 look ups depending upon some if..else business logic. At around
> 2-4 seconds the view is very slow.
>
> However, a lot of the look ups (which are the bottlenecks) can be
> parallelized. But I do not know how can I do the same within a
> request-response cycle.
>
> If it was a web UI, I could use celery+polling, since it a machine-machine
> API call, the parallelisation has to be possible within a View's life cycle.
>
> If parallelisation is not possible, what alternatives do I have for
> scaling and reducing response time.
>
> The short answer is "it depends".

There isn't a single answer - everything will depend on the specifics of
your problem space. All I can offer is some vague suggestions of places you
might be able to look for some extra speed.

 * Do you *really* need to do 1200-5000 lookups? It's faster to do 1 query
returning 10 rows than 10 queries returning 1 row each. Can you optimise
the queries on the database to minimise the number of queries needed?
Depending on circumstances, it may even be faster to do 1 query returning
15 rows, and then post-process in the view to throw away the 5 rows you
don't need.

 * Better still - can you optimize the database structure so that 1200-5000
calls aren't needed? Modern relational databases are *really* good at query
optimisation - if you give it the right query, and the right database.

 * Can you work your algorithm another way? In a previous job, I worked on
a tool that would look at a database of thousands of news articles received
in a given day, and then, for each of thousands of users, work out which
articles were "interesting" so they could be sent out in a daily alert
email. The obvious algorithm for this is "for user in users:
find_articles(user)" - but, for a variety of reasons, it turned out that
doing "for article in articles: find_users(article)" was almost 2 orders of
magnitude of faster. The less obvious algorithm allowed much greater
caching, and massively cut down the number of queries that were required.
The tradeoff was a lot more memory (memory vs speed is almost always the
tradeoff), and it wasn't only faster if you computed the results for *all*
users at the same time - but this was an daily offline process, so these
were limitations we were willing to accept.

 * To that end - is there anything that can be precomputed? Can you cache
pieces of the response? Is there anything you can put into a memory store,
rather than the database. Databases are great, but if you have a small
amount of frequently re-used, easily keyed data, it may be better to put
that data into a location where it can be obtained quickly, rather than
hitting the database.

 * If you *must* parallelize, and your algorithm is conducive to it,
threads are probably your best option - work out what part of your
algorithm can be parallelized, and put each part in a thread, and merge the
results once all the threads complete. If you're on Python 3, look into the
concurrent.futures module (or the "futures" module if you're on Python 2)
to help make this easier to manage. However, threads aren't magic fairy
dust - my "limited knowledge of your situation" guess is that
parallelization won't help you. If you've got a frequently executed view
doing thousands of database calls, I'm going to guess the database is
already a bit of a bottleneck; adding 10 threads per request is going to
increase the database load and make performance *worse*, not better, and if
it's a high traffic view, at some point, you're going to hit the limit of
the number of threads your server can handle.

 * Lastly, I'd challenge your assertion that this can't be done using a
celery + poll approach because it's a machine API. The fact that it's a
machine consuming the API doesn't matter; it's just a matter of what the
machine consumes. The public API for a "long running data processing
service" should be a 2 call API: (a) submit a processing request, and (b)
check the status of a specific processing request. If a human is consuming
this API, it can be put into a nice single-page app with an AJAX call
checking the status of the request. If it's a machine, you can make exactly
the same calls; you just treat it as a sequential sequence of calls with a
"while not finished: poll(); sleep(5)" loop. Of course, changing the API
style won't make your algorithm run any faster - it will just take the load
off your web server and allow you to offload processing to other servers.

I hope some of this helps.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving 

differences of admin/auth/user between django 1.6.2 and 1.6.5

[josephine]

On my localhost where I have django 1.6.2, the admin/auth/user/ page has a 
search box and filter; when I click a specific user for "Change user", I 
can change the user's password using
admin/auth/user/3/password/;
On a different machine that has django 1.6. 5, the admin/auth/user/ page 
does not have the search box and filter.  When I click a user, there is no 
link to send me to the change password form. If I manually use the 
admin/auth/user/3/password/ link, it generates 404 error.
However, when I check the relevant source code of class UserAdmin, 1.6.2 
and 1.6.5 seem to the same, e.g.
they both have the following:

def get_urls(self):
from django.conf.urls import patterns
return patterns('',
(r'^(\d+)/password/$',
 self.admin_site.admin_view(self.user_change_password))
) + super(UserAdmin, self).get_urls()

So why doesn't the password link show up on 1.6.5?  Is there any setting 
that's disabling the UserAdmin defined in contrib/auth/admin.py?
My code has not customized UserAdmin.
A friend of mine is using 1.6.11 and has the same problem.


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/7311722f-401c-41dc-a7da-08216a0a13c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: data migrations for large tables on Postgres

[Vernon D. Cole]

I was thinking of something like that ... but I am not a good enough 
plpgsql programmer to figure out how to do the chunking in SQL.  I needed a 
way to do the data calculations a thousand rows at a time, rather than all 
ninety-million in one gulp.  So I have Python do the chunking and SQL do 
the update.  

There was also the documentation aspect: "where do I store the SQL code and 
the instructions for how to do the migration manually?"  This way, 
everything is contained in the migration module.  The SQL is stored in 
Python triple-quoted string literals, along with the code to run it, and 
the database connection information comes right out of the django settings 
module.

On Thursday, May 14, 2015 at 4:16:43 PM UTC-6, John Fabiani wrote:
>
>  As a newbie - would it be better to use pgadmin (or psql) to make the 
> changes and migrate --fake so that Django would be happy?
>
> Johnf
>
> On 05/14/2015 02:43 PM, Vernon D. Cole wrote:
>  
> I have learned the hard way this week that data migrations in django 1.8, 
> as wonderful has they are, do not scale.
>
> My test data table is now sitting at about 90,000,000 rows.  I was able to 
> add a "null=True" field in an instant, as documented.  Then came my attempt 
> to fill it -- I tried using RunSQL, as suggested, and the migration ran for 
> more than a day before crashing. The entire migration is treated as a 
> single transaction, so none of the work was committed. My next attempt was 
> to use RunPython with manual transaction control, but that cannot be done 
> because the "atomic=False" argument to RunPython does not work on a 
> Postgres database. 
>
> My final solution (which should be done propagating my new field about 
> three days from now) was to have the migration module spawn a copy of 
> itself as a no-wait subprocess. It then runs as a main program, it opens 
> its own connection to the database, and does the conversion a chunk at a 
> time with manual transaction control. 
>
> I think that my solution may be of benefit to others who might be able to 
> adapt my code to their own situation -- but I am not sure how to publish 
> it.  It is not a module, so publication on PyPi or such would be wrong.  
> What would be effective?
>
>  -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users...@googlegroups.com .
> To post to this group, send email to django...@googlegroups.com 
> .
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/58bbdd64-2b6d-491f-9429-3cba8a54d94f%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/dd2abf3f-0981-43cc-ac39-8c05a730f3ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: data migrations for large tables on Postgres

[john]

As a newbie - would it be better to use pgadmin (or psql) to make the 
changes and migrate --fake so that Django would be happy?


Johnf

On 05/14/2015 02:43 PM, Vernon D. Cole wrote:
I have learned the hard way this week that data migrations in django 
1.8, as wonderful has they are, do not scale.


My test data table is now sitting at about 90,000,000 rows.  I was 
able to add a "null=True" field in an instant, as documented.  Then 
came my attempt to fill it -- I tried using RunSQL, as suggested, and 
the migration ran for more than a day before crashing. The entire 
migration is treated as a single transaction, so none of the work was 
committed. My next attempt was to use RunPython with manual 
transaction control, but that cannot be done because the 
"atomic=False" argument to RunPython does not work on a Postgres 
database.


My final solution (which should be done propagating my new field about 
three days from now) was to have the migration module spawn a copy of 
itself as a no-wait subprocess. It then runs as a main program, it 
opens its own connection to the database, and does the conversion a 
chunk at a time with manual transaction control.


I think that my solution may be of benefit to others who might be able 
to adapt my code to their own situation -- but I am not sure how to 
publish it.  It is not a module, so publication on PyPi or such would 
be wrong.  What would be effective?


--
You received this message because you are subscribed to the Google 
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-users+unsubscr...@googlegroups.com 
.
To post to this group, send email to django-users@googlegroups.com 
.

Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/58bbdd64-2b6d-491f-9429-3cba8a54d94f%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1EB3.3030001%40jfcomputer.com.
For more options, visit https://groups.google.com/d/optout.

data migrations for large tables on Postgres

[Vernon D. Cole]

I have learned the hard way this week that data migrations in django 1.8, 
as wonderful has they are, do not scale.

My test data table is now sitting at about 90,000,000 rows.  I was able to 
add a "null=True" field in an instant, as documented.  Then came my attempt 
to fill it -- I tried using RunSQL, as suggested, and the migration ran for 
more than a day before crashing. The entire migration is treated as a 
single transaction, so none of the work was committed. My next attempt was 
to use RunPython with manual transaction control, but that cannot be done 
because the "atomic=False" argument to RunPython does not work on a 
Postgres database. 

My final solution (which should be done propagating my new field about 
three days from now) was to have the migration module spawn a copy of 
itself as a no-wait subprocess. It then runs as a main program, it opens 
its own connection to the database, and does the conversion a chunk at a 
time with manual transaction control. 

I think that my solution may be of benefit to others who might be able to 
adapt my code to their own situation -- but I am not sure how to publish 
it.  It is not a module, so publication on PyPi or such would be wrong.  
What would be effective?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/58bbdd64-2b6d-491f-9429-3cba8a54d94f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to load an object created from a model outside the project ?

[Matthieu]

These solutions seem interesting :) Thanks !

Le jeudi 14 mai 2015 13:44:15 UTC+2, Gergely Polonkai a écrit :
>
> Hello,
>
> depends on what exactly you want to do. You can take a look at Django’s 
> own serialization solution[1], or, if you want to build a REST framework 
> based on this, you may want to look at [2] or [3].
>
> Best,
> Gergely
>
> [1] https://docs.djangoproject.com/en/1.8/topics/serialization/
> [2] https://django-restless.readthedocs.org/en/latest/
> [3] http://www.django-rest-framework.org/
>
> 2015-05-14 12:06 GMT+02:00 Matthieu  >:
>
>> Hi Gergely,
>>
>> Thanks for you answer. 
>> I don't need to modify the object. I also tried with "dill" package but 
>> it's not working either. How can I use serialization with my example ?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/b260b072-3a28-4ef3-97a7-d419e401ca80%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1b3c26e3-fe17-4e21-92f8-148f03dcb309%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improve Performance in Admin ManyToMany

[Timothy W. Cook]

That is exactly the problem Simon.  Everyone of those models reference a
model called Project.  I did this so that when the items are displayed in
the selects, the user knows which project it is from.  In the interim I
guess I'll remove the call to Project from the __str__

I wonder if there is another approach that I can use to solve this?

Thanks,
Tim

On Thu, May 14, 2015 at 12:16 PM, Simon Charette 
wrote:

> I meant
>
> if db_field.name == 'bars':
>
> Sorry for noise...
>
>
> Le jeudi 14 mai 2015 11:15:49 UTC-4, Simon Charette a écrit :
>>
>> The last example should read:
>>
>> if db_field.name == 'baz':
>>
>>
>>
>> Le jeudi 14 mai 2015 11:14:24 UTC-4, Simon Charette a écrit :
>>>
>>> It's a bit hard to tell without your model definition and the executed
>>> queries but is it possible that the string representation method (__str__,
>>> __unicode__) of one the models referenced by a M2M access another related
>>> model during construction?
>>>
>>> e.g.
>>>
>>> from django.db import models
>>>
>>> class Foo(models.Model):
>>> pass
>>>
>>> class Bar(models.Model):
>>> foo = models.ForeignKey(Foo)
>>>
>>> def __str__(self):
>>> return "Foo %s" % self.foo
>>>
>>> class Baz(models.Model):
>>> bars = models.ManyToManyField(Bar)
>>>
>>> Here the Bar model uses the Foo related model for string representation
>>> construction and will result in N + 1 queries (N equals the number of Bar
>>> objects) if the Baz.bars fields is displayed in the admin. To make sure
>>> only one query is executed you must make sure Baz.bars are selected with
>>> their related Foo objects:
>>>
>>> from django.contrib import admin
>>>
>>> class BazAdmin(admin.ModelAdmin):
>>> def formfield_for_many_to_many(self, db_field, *args, **kwargs):
>>> formfield = super(BazAdmin, self).formfield_for_many_to_many(
>>> db_field, *args, **kwargs)
>>> if db_field.name == 'baz':
>>> formfield.queryset = formfield.queryset.select_related('foo'
>>> )
>>> return formfield
>>>
>>> Simon
>>>
>>> Le jeudi 14 mai 2015 07:29:36 UTC-4, Timothy W. Cook a écrit :

 It isn't that the individual queries are very slow. But that there are
 so many of them. Attached is a screenshot of DjDT.
 I see that I am not using a cache at all.  This was after doing a
 reload of the same standard Django admin/change_form.html.


 On Wed, May 13, 2015 at 1:58 PM, Tim Graham  wrote:

> Are you sure it's the query that's slow and not the template rendering
> and/or JavaScript performance?
>
>
> On Wednesday, May 13, 2015 at 12:32:50 PM UTC-4, Timothy W. Cook wrote:
>>
>> I have a model with 13 M2M relations and some of those have a few
>> thousand instances.
>> This renders rather slowly in the Admin.
>>
>> Thinking about improvements I wonder if it will help to setup
>> prefetch_related queries
>>
>> https://docs.djangoproject.com/en/1.8/ref/models/querysets/#prefetch-related
>>
>> inside a  formfield_for_manytomany method?
>>
>>
>> https://docs.djangoproject.com/en/1.8/ref/contrib/admin/#django.contrib.admin.ModelAdmin.formfield_for_manytomany
>>
>> ​I haven't tried it yet and am not even sure how to go about it.  But
>> if experienced developers think it will work, I'll give it a shot.
>>
>> Thoughts? ​
>>
>>
>> 
>> Timothy Cook
>> LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
>> MLHIM http://www.mlhim.org
>>
>>   --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/58f721ad-2ee4-4016-ac6f-b48661c4ce5b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



 --

 
 Timothy Cook
 LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
 MLHIM http://www.mlhim.org

   --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on 

Possible bug in QuerySet API when chaining filter() and update() methods?

[Alejandro Treviño]

Hello everyone, first-time poster here!

I ran into an interesting scenario earlier today that I thought was worth 
sharing:

Given this update statement using the django ORM:

>>> pks = MyModel.objects.all().values_list('pk', flat=True)
>>> pks
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, '...(remaining elements truncated)...']
>>> MyModel.objects.filter(pk__in=pks).update(foo_field=True)

When pks is a "reasonably small" size, the django ORM generates this 
(valid) SQL statement:

UPDATE `djangoapp_mymodel` SET `foo_field` = 1 WHERE 
`djangoapp_mymodel`.`id` IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

However, when pks is very large (300k records on my data set), it generates 
this instead:

UPDATE `djangoapp_mymodel` SET `foo_field` = 1 WHERE 
`djangoapp_mymodel`.`id` IN (SELECT U0.`id` FROM `djangoapp_mymodel` U0)

Which is not allowed in MySQL:

django.db.utils.OperationalError: (1093, "You can't specify target 
table 'djangoapp_mymodel' for update in FROM clause")


I'm wondering if this classifies as a bug, or if this is just a known 
limitation?  My workaround is to just do the updates in smaller batches. 
 An update on 100k records still generated valid SQL for me, but I haven't 
done enough testing to figure out what cut-off point is.

*Environment:*
Python 3.4.2
Django 1.8.1

mysql  Ver 14.14 Distrib 5.5.43, for osx10.8 (i386) using readline 5.1


Thanks!

Alex

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/10d0d83b-733a-47d2-b5d1-6f9c904269a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How do I display a map with GeoDjango models?

[Thomas Levine]

Indeed I can! It looks like I'm going to use Polymaps instead, but the
magic I was looking for was djgeojson.views.TiledGeoJSONLayerView.
I think. It's quite slow, so I might wind up generating them in batch.
Maybe that's why there wasn't such an obvious answer

On 14 May 06:52, Derek wrote:
> Could you not add them as multiple Leaflet layers; each layer pointing to a 
> view for the model in question.  There is a simple overview here:
> 
> http://www.slideshare.net/makinacorpus/team-up-django-and-web-mapping-djangocon-europe-2014
> 
> There is much more detail on the django-leaflet site 
> (e.g. https://github.com/makinacorpus/django-leaflet#default-tiles-layer )
> 
> 
> On Wednesday, 13 May 2015 21:36:13 UTC+2, Thomas Levine wrote:
> >
> > I have multiple GeoDjango models (http://dada.pink/scott2/map/models.py) 
> > and want them to be layers on the same map. 
> >
> > Surely there's a magic way to specify this. What do I do? 
> >

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20150514175803.GA11420%40d1stkfactory.
For more options, visit https://groups.google.com/d/optout.

Re: Improve Performance in Admin ManyToMany

[Simon Charette]

The last example should read:

if db_field.name == 'baz':



Le jeudi 14 mai 2015 11:14:24 UTC-4, Simon Charette a écrit :
>
> It's a bit hard to tell without your model definition and the executed 
> queries but is it possible that the string representation method (__str__, 
> __unicode__) of one the models referenced by a M2M access another related 
> model during construction?
>
> e.g.
>
> from django.db import models
>
> class Foo(models.Model):
> pass
>
> class Bar(models.Model):
> foo = models.ForeignKey(Foo)
>
> def __str__(self):
> return "Foo %s" % self.foo
>
> class Baz(models.Model):
> bars = models.ManyToManyField(Bar)
>
> Here the Bar model uses the Foo related model for string representation 
> construction and will result in N + 1 queries (N equals the number of Bar 
> objects) if the Baz.bars fields is displayed in the admin. To make sure 
> only one query is executed you must make sure Baz.bars are selected with 
> their related Foo objects:
>
> from django.contrib import admin
>
> class BazAdmin(admin.ModelAdmin):
> def formfield_for_many_to_many(self, db_field, *args, **kwargs):
> formfield = super(BazAdmin, self).formfield_for_many_to_many(
> db_field, *args, **kwargs)
> if db_field.name == 'baz':
> formfield.queryset = formfield.queryset.select_related('foo')
> return formfield
>
> Simon
>
> Le jeudi 14 mai 2015 07:29:36 UTC-4, Timothy W. Cook a écrit :
>>
>> It isn't that the individual queries are very slow. But that there are so 
>> many of them. Attached is a screenshot of DjDT.
>> I see that I am not using a cache at all.  This was after doing a reload 
>> of the same standard Django admin/change_form.html. 
>>
>>
>> On Wed, May 13, 2015 at 1:58 PM, Tim Graham  wrote:
>>
>>> Are you sure it's the query that's slow and not the template rendering 
>>> and/or JavaScript performance?
>>>
>>>
>>> On Wednesday, May 13, 2015 at 12:32:50 PM UTC-4, Timothy W. Cook wrote:

 I have a model with 13 M2M relations and some of those have a few 
 thousand instances. 
 This renders rather slowly in the Admin.

 Thinking about improvements I wonder if it will help to setup 
 prefetch_related queries

 https://docs.djangoproject.com/en/1.8/ref/models/querysets/#prefetch-related

 inside a  formfield_for_manytomany method?


 https://docs.djangoproject.com/en/1.8/ref/contrib/admin/#django.contrib.admin.ModelAdmin.formfield_for_manytomany

 ​I haven't tried it yet and am not even sure how to go about it.  But 
 if experienced developers think it will work, I'll give it a shot.

 Thoughts? ​


 
 Timothy Cook
 LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
 MLHIM http://www.mlhim.org

   -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-users...@googlegroups.com.
>>> To post to this group, send email to django...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-users.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-users/58f721ad-2ee4-4016-ac6f-b48661c4ce5b%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> -- 
>>
>> 
>> Timothy Cook
>> LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
>> MLHIM http://www.mlhim.org
>>
>> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/a98faea7-3973-44c3-bebb-d210a2e15cd9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What is the ideal web server to use with Django?

[Marc Aymerich]

On Thu, May 14, 2015 at 11:26 AM, Tom Evans  wrote:
>
> On Thu, May 14, 2015 at 4:36 AM, reduxionist  
> wrote:
> > The question you asked Tom was "Doesn't Apache create new process for each
> > request [thus eating memory when serving large amounts of static files
> > during traffic peaks]?", and the reason that Tom correctly answers "No" is
> > that as far as "serving large amounts of static files" goes you should be
> > using mpm-worker (multi-threaded Apache) which most definitely does not
> > spawn a new process for each request.
> >
> > The reason for those search results is that mpm-prefork does, however, spawn
> > a process per request,
>
> No, really, it does not. It only spawns a new process when there are
> no available workers to process an incoming request, and you have not
> reached the maximum number of workers that you have configured it to
> start. You can configure it to start all the worker processes you want
> when it starts up, and never to kill them off, and it will never spawn
> a new process.
>
> Apache processes are small, unless you do daft things like embed your
> web application in each worker process (mod_php style). This is the
> main complaint "Apache is eating all my memory" - it isn't, your web
> application you've embedded into Apache is eating all your memory.
>
> All of this is irrelevant for django, because with Apache you should
> use mod_wsgi in daemon mode, which separates out your web application
> processes from the web server.
>
> > but it is only needed for non-thread-safe
> > environments (most notoriously mod_php) and you shouldn't have to use it as
> > long as you've been a good coder and avoided global state in your Django app
> > (e.g. keep request-specific shared-state thread-local).
> >
> > I think the reason a lot of people seem to run mpm-prefork is just that it's
> > the default multi-processing module for Apache on most (all?) *nix platforms
> > and they don't know any better.
>
> Quite. We run a pair of Apache 2.4 reverse proxies in front of all of
> our (400+) domains, serving around 40 million requests per day,
> providing SSL termination and static file serving. We use event MPM
> and we have it scaled to support a peak of 2048 simultaneous
> connections. Load on the server never goes above 0.2, memory usage
> never goes above 1GB for the entire OS + applications, the rest of the
> RAM is used by the OS to cache the aforementioned static files.
>
> On our app servers we typically use Apache with worker MPM and
> mod_wsgi, although we have a few nginx+uwsgi sites, and I would dearly
> love some time to play around with a circusd + chausette + celery
> setup.


Hi Tom,
never heard about circusd and chaussette, but it sounds interesting.

I believe the advantage of this setup pays when you have several
independent low-traffic applications, because you don't want all of
them to have preforked wsgi worker processes, not even the master
process. I think uwsgi can do that (emperor mode, cheaper subsystem),
but circusd will allow you to do the same which celery (celery needs
at least one master per pool). Is this the main point? I'm asking
because I can only find just a couple of blogposts and the
documentation is concise, neither mention the real tangible advantage
over traditional deployments.


>
> The choice of web server is, these days, irrelevant. If it uses too
> much memory or can't handle enough users, it is never the fault of the
> web server, but instead of your application and/or configuration.
> Which is why I return to my original advice:
>
> > I am new to Django. I am building a app which will have to handle several
> > concurrent requests. Which web server is suitable for this?
>
> Any and all.
>
> Leave the fanboyism to the phone guys.
>
> Cheers
>
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/CAFHbX1KEVRM6WU7OCcLRSkJhpMS%2BfHpd7%2BWo7LO8XcEt8_f0Nw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.




-- 
Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2BDCN_uTrQqH%2BZUX1ibc4KF1unE0wiAEzZ814eiLOpJ-5hLgvA%40mail.gmail.com.
For more 

Re: Improve Performance in Admin ManyToMany

[Simon Charette]

I meant

if db_field.name == 'bars':

Sorry for noise...

Le jeudi 14 mai 2015 11:15:49 UTC-4, Simon Charette a écrit :
>
> The last example should read:
>
> if db_field.name == 'baz':
>
>
>
> Le jeudi 14 mai 2015 11:14:24 UTC-4, Simon Charette a écrit :
>>
>> It's a bit hard to tell without your model definition and the executed 
>> queries but is it possible that the string representation method (__str__, 
>> __unicode__) of one the models referenced by a M2M access another related 
>> model during construction?
>>
>> e.g.
>>
>> from django.db import models
>>
>> class Foo(models.Model):
>> pass
>>
>> class Bar(models.Model):
>> foo = models.ForeignKey(Foo)
>>
>> def __str__(self):
>> return "Foo %s" % self.foo
>>
>> class Baz(models.Model):
>> bars = models.ManyToManyField(Bar)
>>
>> Here the Bar model uses the Foo related model for string representation 
>> construction and will result in N + 1 queries (N equals the number of Bar 
>> objects) if the Baz.bars fields is displayed in the admin. To make sure 
>> only one query is executed you must make sure Baz.bars are selected with 
>> their related Foo objects:
>>
>> from django.contrib import admin
>>
>> class BazAdmin(admin.ModelAdmin):
>> def formfield_for_many_to_many(self, db_field, *args, **kwargs):
>> formfield = super(BazAdmin, self).formfield_for_many_to_many(
>> db_field, *args, **kwargs)
>> if db_field.name == 'baz':
>> formfield.queryset = formfield.queryset.select_related('foo')
>> return formfield
>>
>> Simon
>>
>> Le jeudi 14 mai 2015 07:29:36 UTC-4, Timothy W. Cook a écrit :
>>>
>>> It isn't that the individual queries are very slow. But that there are 
>>> so many of them. Attached is a screenshot of DjDT.
>>> I see that I am not using a cache at all.  This was after doing a reload 
>>> of the same standard Django admin/change_form.html. 
>>>
>>>
>>> On Wed, May 13, 2015 at 1:58 PM, Tim Graham  wrote:
>>>
 Are you sure it's the query that's slow and not the template rendering 
 and/or JavaScript performance?


 On Wednesday, May 13, 2015 at 12:32:50 PM UTC-4, Timothy W. Cook wrote:
>
> I have a model with 13 M2M relations and some of those have a few 
> thousand instances. 
> This renders rather slowly in the Admin.
>
> Thinking about improvements I wonder if it will help to setup 
> prefetch_related queries
>
> https://docs.djangoproject.com/en/1.8/ref/models/querysets/#prefetch-related
>
> inside a  formfield_for_manytomany method?
>
>
> https://docs.djangoproject.com/en/1.8/ref/contrib/admin/#django.contrib.admin.ModelAdmin.formfield_for_manytomany
>
> ​I haven't tried it yet and am not even sure how to go about it.  But 
> if experienced developers think it will work, I'll give it a shot.
>
> Thoughts? ​
>
>
> 
> Timothy Cook
> LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
> MLHIM http://www.mlhim.org
>
>   -- 
 You received this message because you are subscribed to the Google 
 Groups "Django users" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to django-users...@googlegroups.com.
 To post to this group, send email to django...@googlegroups.com.
 Visit this group at http://groups.google.com/group/django-users.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/django-users/58f721ad-2ee4-4016-ac6f-b48661c4ce5b%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> -- 
>>>
>>> 
>>> Timothy Cook
>>> LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
>>> MLHIM http://www.mlhim.org
>>>
>>> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b769eae9-7b78-43e8-bfaa-eae864bcd386%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improve Performance in Admin ManyToMany

[Simon Charette]

It's a bit hard to tell without your model definition and the executed 
queries but is it possible that the string representation method (__str__, 
__unicode__) of one the models referenced by a M2M access another related 
model during construction?

e.g.

from django.db import models

class Foo(models.Model):
pass

class Bar(models.Model):
foo = models.ForeignKey(Foo)

def __str__(self):
return "Foo %s" % self.foo

class Baz(models.Model):
bars = models.ManyToManyField(Bar)

Here the Bar model uses the Foo related model for string representation 
construction and will result in N + 1 queries (N equals the number of Bar 
objects) if the Baz.bars fields is displayed in the admin. To make sure 
only one query is executed you must make sure Baz.bars are selected with 
their related Foo objects:

from django.contrib import admin

class BazAdmin(admin.ModelAdmin):
def formfield_for_many_to_many(self, db_field, *args, **kwargs):
formfield = super(BazAdmin, self).formfield_for_many_to_many(
db_field, *args, **kwargs)
if db_field.name == 'baz':
formfield.queryset = formfield.queryset.select_related('foo')
return formfield

Simon

Le jeudi 14 mai 2015 07:29:36 UTC-4, Timothy W. Cook a écrit :
>
> It isn't that the individual queries are very slow. But that there are so 
> many of them. Attached is a screenshot of DjDT.
> I see that I am not using a cache at all.  This was after doing a reload 
> of the same standard Django admin/change_form.html. 
>
>
> On Wed, May 13, 2015 at 1:58 PM, Tim Graham  > wrote:
>
>> Are you sure it's the query that's slow and not the template rendering 
>> and/or JavaScript performance?
>>
>>
>> On Wednesday, May 13, 2015 at 12:32:50 PM UTC-4, Timothy W. Cook wrote:
>>>
>>> I have a model with 13 M2M relations and some of those have a few 
>>> thousand instances. 
>>> This renders rather slowly in the Admin.
>>>
>>> Thinking about improvements I wonder if it will help to setup 
>>> prefetch_related queries
>>>
>>> https://docs.djangoproject.com/en/1.8/ref/models/querysets/#prefetch-related
>>>
>>> inside a  formfield_for_manytomany method?
>>>
>>>
>>> https://docs.djangoproject.com/en/1.8/ref/contrib/admin/#django.contrib.admin.ModelAdmin.formfield_for_manytomany
>>>
>>> ​I haven't tried it yet and am not even sure how to go about it.  But if 
>>> experienced developers think it will work, I'll give it a shot.
>>>
>>> Thoughts? ​
>>>
>>>
>>> 
>>> Timothy Cook
>>> LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
>>> MLHIM http://www.mlhim.org
>>>
>>>   -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/58f721ad-2ee4-4016-ac6f-b48661c4ce5b%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
>
> 
> Timothy Cook
> LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
> MLHIM http://www.mlhim.org
>
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/fa817fe2-315d-4e41-bf27-8d95bac7ca83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

User defined API for Django Rest-Framework

[SUBHABRATA BANERJEE]

Dear Group,

I am trying to design API on Rest framework on Django. I could make one 
Browsable API as given 
in http://www.django-rest-framework.org/tutorial/quickstart/.

I am looking to change the views, to user defined views. 

If anyone may kindly suggest how may I do it? 

I am pretty new in Django and working on Python2.7+ on Windows 7 
Professional.
Thanks in advance,

Subhabrata Banerjee. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4323b1de-feee-42e5-bc74-2dfe77ae5e5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What is the ideal web server to use with Django?

[Jonathan Barratt]

> On 14 พ.ค. 2558, at 05:26, Tom Evans  wrote:
> 
>> On Thu, May 14, 2015 at 4:36 AM, reduxionist  
>> wrote:

>> The reason for those search results is that mpm-prefork does, however, spawn
>> a process per request,
> 
> No, really, it does not. It only spawns a new process when there are
> no available workers to process an incoming request, and you have not
> reached the maximum number of workers that you have configured it to
> start.

Oops, sorry, I was totally wrong about that.

> You can configure it to start all the worker processes you want
> when it starts up, and never to kill them off, and it will never spawn
> a new process.
> 
> Apache processes are small, unless you do daft things like embed your
> web application in each worker process (mod_php style). This is the
> main complaint "Apache is eating all my memory" - it isn't, your web
> application you've embedded into Apache is eating all your memory.
> 
> All of this is irrelevant for django, because with Apache you should
> use mod_wsgi in daemon mode, which separates out your web application
> processes from the web server.

>> I think the reason a lot of people seem to run mpm-prefork is just that it's
>> the default multi-processing module for Apache on most (all?) *nix platforms
>> and they don't know any better.
> 
> Quite. We run a pair of Apache 2.4 reverse proxies in front of all of
> our (400+) domains, serving around 40 million requests per day,
> providing SSL termination and static file serving. We use event MPM
> and we have it scaled to support a peak of 2048 simultaneous
> connections. Load on the server never goes above 0.2, memory usage
> never goes above 1GB for the entire OS + applications, the rest of the
> RAM is used by the OS to cache the aforementioned static files.
> 
> On our app servers we typically use Apache with worker MPM and
> mod_wsgi, although we have a few nginx+uwsgi sites, and I would dearly
> love some time to play around with a circusd + chausette + celery
> setup.
> 
> The choice of web server is, these days, irrelevant. If it uses too
> much memory or can't handle enough users, it is never the fault of the
> web server, but instead of your application and/or configuration.

Thanks for the awesome write-up Tom and for correcting my error, my apologies 
to all for contributing to any FUD!

Yours gratefully,
Jonathan

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/46CB380A-BBA2-4F78-8F7F-8DFC7A994859%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How do I display a map with GeoDjango models?

[Derek]

Could you not add them as multiple Leaflet layers; each layer pointing to a 
view for the model in question.  There is a simple overview here:

http://www.slideshare.net/makinacorpus/team-up-django-and-web-mapping-djangocon-europe-2014

There is much more detail on the django-leaflet site 
(e.g. https://github.com/makinacorpus/django-leaflet#default-tiles-layer )


On Wednesday, 13 May 2015 21:36:13 UTC+2, Thomas Levine wrote:
>
> I have multiple GeoDjango models (http://dada.pink/scott2/map/models.py) 
> and want them to be layers on the same map. 
>
> Surely there's a magic way to specify this. What do I do? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/f998233c-0f9a-4849-bfee-102cf2b0495c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django 1.8 tutorial-Chapter4: ValueError at /polls/1/vote/

[Charito Romeo]

Hi Alasdair,

Thanks a lot for the help. It's working now. Cheers :)

Charito

On Thu, May 14, 2015 at 7:57 PM, Alasdair Nicol  wrote:

> On 14/05/15 02:50, charito.romeo wrote:
>
>> |
>>  > value="{{
>>  choice.id }}"/>
>> |
>>
>
> The problem is that there is a new line in the middle of
>
> "{{ choice.id }}".
>
> Change it to:
>
>  value="{{ choice.id }}"/>
>
> Cheers,
> Alasdair
>
> --
> Alasdair Nicol
> Developer, MEMSET
>
> mail: alasd...@memset.com
>  web: http://www.memset.com/
>
> Memset Ltd., registration number 4504980.
> Building 87, Dunsfold Park, Stovolds Hill, Cranleigh, Surrey, GU6 8TB, UK
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/55548D93.3000701%40memset.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAM9Y-vv9jZi%3DqwXuh5KkF3sEVyh8H4DvvXq5iO%3D6rht6Phs6Cg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django-allauth without username, login with facebook doesn't work

[Fellipe Henrique]

Hello Mario, thanks for help me...

Yesterday I found the problem, here is: First i used python-social-auth
after I move to django-allauth... for some reason, when I uninstalled the
python-social-auth, the folder still in site-package, and when I call my
template, django use the python-social-auth javascript!!!

I manually remove the folder, and reinstall django-allauth and work fine
now!

Regards,

T.·.F.·.A.·. S+F
*Fellipe Henrique P. Soares*

e-mail: > echo "lkrrovknFmsgor4ius" | perl -pe \ 's/(.)/chr(ord($1)-2*3)/ge'
*Blog: http://fhbash.wordpress.com/ *
*GitHub: https://github.com/fellipeh *
*Twitter: @fh_bash*

On Thu, May 14, 2015 at 2:43 AM, Mario Gudelj 
wrote:

> Sorry man. My mistake.
>
> So are you saying that you have a link with onclick or something? Do you
> get the JS error? Is it logged inside your browser's console? It's a bit
> hard to tell without more code or the URL...
>
> On 14 May 2015 at 03:12, Fellipe Henrique  wrote:
>
>> Hi Mario,
>>
>> I`m not using python-social-auth... I choose to use all-auth[1] because
>> it's given to me more the social login, like: password reset, email
>> validation etc.
>>
>> My problem is: in template when I click on a button, nothing happen.. the
>> link (javascript) is correct, but nothing happen...
>>
>>
>> [1] - http://www.intenct.nl/projects/django-allauth/
>>
>> T.·.F.·.A.·. S+F
>> *Fellipe Henrique P. Soares*
>>
>> e-mail: > echo "lkrrovknFmsgor4ius" | perl -pe \
>> 's/(.)/chr(ord($1)-2*3)/ge'
>> *Blog: http://fhbash.wordpress.com/ *
>> *GitHub: https://github.com/fellipeh *
>> *Twitter: @fh_bash*
>>
>> On Wed, May 13, 2015 at 12:26 AM, Mario Gudelj 
>> wrote:
>>
>>> What do you mean by nothing happens? Do you get taken to FB?
>>>
>>> Here's what I have in my setup which may help:
>>>
>>> The template link looks like this:
>>>
>>> Facebook
>>>
>>> Make sure you have the following line in your urls.py:
>>>
>>> url('', include('social.apps.django_app.urls', namespace='social'))
>>>
>>> settings.py:
>>>
>>> SOCIAL_AUTH_USER_FIELDS = ['email']
>>> SOCIAL_AUTH_USERNAME_IS_FULL_EMAIL = True
>>> SOCIAL_AUTH_FACEBOOK_KEY = 'your_key'
>>> SOCIAL_AUTH_FACEBOOK_SECRET = 'your_secret'
>>> SOCIAL_AUTH_FACEBOOK_SCOPE = ['email']
>>> SOCIAL_AUTH_ENABLED_BACKENDS=('facebook', 'google')
>>> SOCIAL_AUTH_PROTECTED_FIELDS = ['email']
>>> SOCIAL_AUTH_ADMIN_USER_SEARCH_FIELDS = ['first_name', 'last_name',
>>> 'email']
>>>
>>> AUTHENTICATION_BACKENDS = (
>>> 'social.backends.facebook.FacebookOAuth2',
>>> 'social.backends.google.GoogleOAuth2',
>>> 'social.backends.twitter.TwitterOAuth',
>>> 'django.contrib.auth.backends.ModelBackend',
>>> )
>>>
>>> Add 'social.apps.django_app.default', to INSTALLED_APPS
>>>
>>> Use https://github.com/omab/python-social-auth and follow installation
>>> instructions from it. You'll need to setup your FB app as well.
>>>
>>> I think that's all you need.
>>>
>>> Good luck man.
>>>
>>>
>>>
>>> On 13 May 2015 at 04:13, Fellipe Henrique  wrote:
>>>
 Just  to complement the previous mail,  normal user (django user) like
 super-user create using syncdb, works fine!

 I'm using django 1.8.

 T.·.F.·.A.·. S+F
 *Fellipe Henrique P. Soares*

 e-mail: > echo "lkrrovknFmsgor4ius" | perl -pe \
 's/(.)/chr(ord($1)-2*3)/ge'
 *Blog: http://fhbash.wordpress.com/ *
 *GitHub: https://github.com/fellipeh *
 *Twitter: @fh_bash*

 On Tue, May 12, 2015 at 3:09 PM, Fellipe Henrique 
 wrote:

> Hello,
>
> I'm trying to use Django-AllAuth social login...
>
> I have a custom user profile and custom usermanage [1]
>
> But, when I click on html template facebook link, anything happen!
>  with username, works fine.
>
> Here is my settings [2]
>
> My settings is wrong? any idea why this problem happens?
>
> Thanks,
> regards
>
>
> [1] - https://gist.github.com/fellipeh/fe2deae4854072b0cb3d
> [2] - https://gist.github.com/fellipeh/033fcde17c9771dfb1b2
>
> T.·.F.·.A.·. S+F
> *Fellipe Henrique P. Soares*
>
> e-mail: > echo "lkrrovknFmsgor4ius" | perl -pe \
> 's/(.)/chr(ord($1)-2*3)/ge'
> *Blog: http://fhbash.wordpress.com/ *
> *GitHub: https://github.com/fellipeh *
> *Twitter: @fh_bash*
>

  --
 You received this message because you are subscribed to the Google
 Groups "Django users" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-users+unsubscr...@googlegroups.com.
 To post to this group, send email to django-users@googlegroups.com.
 

Re: Django 1.8 tutorial-Chapter4: ValueError at /polls/1/vote/

[Alasdair Nicol]

On 14/05/15 02:50, charito.romeo wrote:

|
 
|


The problem is that there is a new line in the middle of

"{{ choice.id }}".

Change it to:

value="{{ choice.id }}"/>


Cheers,
Alasdair

--
Alasdair Nicol
Developer, MEMSET

mail: alasd...@memset.com
 web: http://www.memset.com/

Memset Ltd., registration number 4504980.
Building 87, Dunsfold Park, Stovolds Hill, Cranleigh, Surrey, GU6 8TB, UK

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55548D93.3000701%40memset.com.
For more options, visit https://groups.google.com/d/optout.

Re: form_invalid and missing ob

[David]

Fixed. Was missing:

self.form = self.get_form(self.form_class)
self.object_list = self.get_queryset()

from def form_invalid(self, form):

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2cd7a35a-1150-412c-a551-9cdbb9ac4de3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to load an object created from a model outside the project ?

[Gergely Polonkai]

Hello,

depends on what exactly you want to do. You can take a look at Django’s own
serialization solution[1], or, if you want to build a REST framework based
on this, you may want to look at [2] or [3].

Best,
Gergely

[1] https://docs.djangoproject.com/en/1.8/topics/serialization/
[2] https://django-restless.readthedocs.org/en/latest/
[3] http://www.django-rest-framework.org/

2015-05-14 12:06 GMT+02:00 Matthieu :

> Hi Gergely,
>
> Thanks for you answer.
> I don't need to modify the object. I also tried with "dill" package but
> it's not working either. How can I use serialization with my example ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/b260b072-3a28-4ef3-97a7-d419e401ca80%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CACczBUJSJ_tZBZ04N_O8_8h0G0r3MHk9A6uNNf9GJGm8eP4eqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improve Performance in Admin ManyToMany

[Timothy W. Cook]

It isn't that the individual queries are very slow. But that there are so
many of them. Attached is a screenshot of DjDT.
I see that I am not using a cache at all.  This was after doing a reload of
the same standard Django admin/change_form.html.


On Wed, May 13, 2015 at 1:58 PM, Tim Graham  wrote:

> Are you sure it's the query that's slow and not the template rendering
> and/or JavaScript performance?
>
>
> On Wednesday, May 13, 2015 at 12:32:50 PM UTC-4, Timothy W. Cook wrote:
>>
>> I have a model with 13 M2M relations and some of those have a few
>> thousand instances.
>> This renders rather slowly in the Admin.
>>
>> Thinking about improvements I wonder if it will help to setup
>> prefetch_related queries
>>
>> https://docs.djangoproject.com/en/1.8/ref/models/querysets/#prefetch-related
>>
>> inside a  formfield_for_manytomany method?
>>
>>
>> https://docs.djangoproject.com/en/1.8/ref/contrib/admin/#django.contrib.admin.ModelAdmin.formfield_for_manytomany
>>
>> ​I haven't tried it yet and am not even sure how to go about it.  But if
>> experienced developers think it will work, I'll give it a shot.
>>
>> Thoughts? ​
>>
>>
>> 
>> Timothy Cook
>> LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
>> MLHIM http://www.mlhim.org
>>
>>   --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/58f721ad-2ee4-4016-ac6f-b48661c4ce5b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 


Timothy Cook
LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook
MLHIM http://www.mlhim.org

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2B%3DOU3XABUDKS0AVsi4Y%3Dz%2BqBiB5veYkgM56eoeEP_3DBS7DOw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: coffin.shortcuts error

[Stephen J. Butler]

The developer of coffin released v2.0 which is pretty much a simple stub.
He recommends you switch to django-jinja.

You probably want to limit your requirements.txt to coffin<2.0.

On Thu, May 14, 2015 at 1:28 AM,  wrote:

> Hi, first of all i'm sorry for my bad english.
>
>
> I am installing a django app for continuing its develop, but when I try to
> start the app I get this error:
>
> ImportError at /
>
> No module named shortcuts
>
>  Request Method: GET  Request URL: http://127.0.0.1:8000/  Django Version:
> 1.5  Exception Type: ImportError  Exception Value:
>
> No module named shortcuts
>
>  Exception Location: 
> /home/eneas/workspace/python/local/app-name/module-name/views/helpers.py
> in , line 4  Python Executable:
> /home/eneas/workspace/python/bin/python  Python Version: 2.7.6  Python
> Path:
>
> ['/home/eneas/workspace/python/local/app-name',
>  '/home/eneas/workspace/python/lib/python2.7',
>  '/home/eneas/workspace/python/lib/python2.7/plat-x86_64-linux-gnu',
>  '/home/eneas/workspace/python/lib/python2.7/lib-tk',
>  '/home/eneas/workspace/python/lib/python2.7/lib-old',
>  '/home/eneas/workspace/python/lib/python2.7/lib-dynload',
>  '/usr/lib/python2.7',
>  '/usr/lib/python2.7/plat-x86_64-linux-gnu',
>  '/usr/lib/python2.7/lib-tk',
>  '/home/eneas/workspace/python/local/lib/python2.7/site-packages',
>  '/home/eneas/workspace/python/lib/python2.7/site-packages']
>
>
>
>
>
>
> It looks like coffin is bad installed, but I don't know how I can fix it.
>
>
> I work in a virtualenv and I installed coffin inside this virtualenv like 
> this:
>
>
> pip install Coffin
>
> I am desperate and if I get some help I will be very grateful.
>
>
>
> Thanks in advance.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/3930f941-0157-4603-852e-e6cb9d85c23f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAD4ANxWLYutgMfokZKqwrTiXYN4o8KNTiBO08_ie_r-so7AGNw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Django 1.8 tutorial-Chapter4: ValueError at /polls/1/vote/

[charito.romeo]

I am following the django 1.8 tutorial and I currently finished Part 4. My 
problem is when I tried to click on one of the radio buttons to vote, it 
gave me a ValueError below:

ValueError at /polls/1/vote/

invalid literal for int() with base 10: '{{  choice.id }}'

Request Method:POSTRequest URL:http://127.0.0.1:8000/polls/1/vote/Django 
Version:1.8.1Exception Type:ValueErrorException Value:

invalid literal for int() with base 10: '{{  choice.id }}'

Exception 
Location:/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/fields/__init__.py
 
in get_prep_value, line 985Python Executable:
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/PythonPython
 
Version:2.7.9Python Path:

['/Users/cellbio1/Documents/django projects/mysite',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
 '/Library/Python/2.7/site-packages']

Server time:Thu, 14 May 2015 01:32:18 +
Traceback Switch to copy-and-paste view 


   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/core/handlers/base.py in get_response
   1. 
  
  response = wrapped_callback(request, 
*callback_args, **callback_kwargs)
  
  ...
   ▶ Local vars 
   - /Users/cellbio1/Documents/django projects/mysite/polls/views.py in vote
   1. 
  
  selected_choice = 
p.choice_set.get(pk=request.POST['choice'])
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/manager.py in manager_method
   1. 
  
  return getattr(self.get_queryset(), 
name)(*args, **kwargs)
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/query.py in get
   1. 
  
  clone = self.filter(*args, **kwargs)
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/query.py in filter
   1. 
  
  return self._filter_or_exclude(False, *args, **kwargs)
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/query.py in _filter_or_exclude
   1. 
  
  clone.query.add_q(Q(*args, **kwargs))
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/sql/query.py in add_q
   1. 
  
  clause, require_inner = self._add_q(where_part, 
self.used_aliases)
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/sql/query.py in _add_q
   1. 
  
  current_negated=current_negated, 
connector=connector, allow_joins=allow_joins)
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/sql/query.py in build_filter
   1. 
  
  condition = self.build_lookup(lookups, col, value)
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/sql/query.py in build_lookup
   1. 
  
  return final_lookup(lhs, rhs)
  
  ...
   ▶ Local vars 
   - /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
   packages/django/db/models/lookups.py in __init__
   1. 
  
  self.rhs = self.get_prep_lookup()
  
  ...
   ▶ Local vars 
   - 

coffin.shortcuts error

[eneasgallego]

Hi, first of all i'm sorry for my bad english.


I am installing a django app for continuing its develop, but when I try to 
start the app I get this error:

ImportError at / 

No module named shortcuts

 Request Method: GET  Request URL: http://127.0.0.1:8000/  Django Version: 
1.5  Exception Type: ImportError  Exception Value: 

No module named shortcuts

 Exception Location: 
/home/eneas/workspace/python/local/app-name/module-name/views/helpers.py 
in , line 4  Python Executable: 
/home/eneas/workspace/python/bin/python  Python Version: 2.7.6  Python Path: 

['/home/eneas/workspace/python/local/app-name',
 '/home/eneas/workspace/python/lib/python2.7',
 '/home/eneas/workspace/python/lib/python2.7/plat-x86_64-linux-gnu',
 '/home/eneas/workspace/python/lib/python2.7/lib-tk',
 '/home/eneas/workspace/python/lib/python2.7/lib-old',
 '/home/eneas/workspace/python/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/home/eneas/workspace/python/local/lib/python2.7/site-packages',
 '/home/eneas/workspace/python/lib/python2.7/site-packages']






It looks like coffin is bad installed, but I don't know how I can fix it.


I work in a virtualenv and I installed coffin inside this virtualenv like this:


pip install Coffin 

I am desperate and if I get some help I will be very grateful.



Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3930f941-0157-4603-852e-e6cb9d85c23f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

choice field with a twist?

[MikeKJ]

I may have to radically rethink this from model up but there may be a 
solution?

I have a model called ZoneCost which contains the data
A : 0.8
B: 0.9
C: 1.0
D: 1.2
E: 1.4 

and another model Card that among other things contains a zone identifier

What I want to do is in a view 
1. Get the zone applicable 
serial = form.cleaned_data[card_serial]
this = Card.objects.get(card_serial=serial)
zone = this.zone
2. Modify the choices 
trips = forms.ChoiceField(choices=trip_choices)
trip_choices = ( ("10", "10), etc

so instead of outputting as 10, 20 etc in a dropdown it outputs as
10 - value
20 - value

where value is choice option * the multiplier defined by the zone
ie for Zone D
10 - 12
20 - 24

and pass it to the form

any ideas please?


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ea4d7861-4473-4183-8d15-70f8786c559e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: form_invalid and missing ob

[David]

Doh, fixed.

Fix below for anyone else with the same problem:

def post(self, request, *args, **kwargs):
form = self.get_form()
if form.is_valid():
return self.form_valid(form)
else:
return self.form_invalid(form, *args, **kwargs)

def form_invalid(self, form, *args, **kwargs):
messages.error(self.request, '{0} could not be 
saved.'.format(self.model.__name__))
if self.request.is_ajax():
return JsonResponse(form.errors, status=400)
return super(ShowForumPostsView, self).form_invalid(form)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/f53fca4a-dd06-4dbc-82c4-21d7483e8f23%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

form_invalid and missing ob

[David]

Hi

When I submit my form with no errors it correctly processes and returns 
after completing form_valid.

However, when I submit my form with errors I get a django dump stating that 
'ShowForumPostsView' 
object has no attribute 'object_list'.

I am struggling to make it return to the same page but showing the errors.

Any help would be appreciate please.

Thank you


My code:

class ShowForumPostsView(GroupRequiredMixin, FormMixin, ListView):
model = Post
group_required = 'clients'
paginate_by = 4
form_class = PostForm




def post(self, request, *args, **kwargs):
form = self.get_form()
if form.is_valid():
return self.form_valid(form)
else:
return self.form_invalid(form)
#return self.get(request, *args, **kwargs)

def form_valid(self, form):
data = form.save(commit=False)
data.creator = self.request.user
data.thread_id = Thread.objects.get(slug=self.kwargs['threadslug']).pk
data.save()
messages.success(self.request, '{0} added.'.format(self.model.__name__))

if self.request.is_ajax():
return JsonResponse({'reset': True})
return super(ShowForumPostsView, self).form_valid(form)

def form_invalid(self, form):
messages.error(self.request, '{0} could not be 
saved.'.format(self.model.__name__))
if self.request.is_ajax():
return JsonResponse(form.errors, status=400)
#return super(ShowForumPostsView, self).form_invalid(form)
return self.render_to_response(self.get_context_data(form=form))

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b7322f53-a0dc-4146-b087-87c188130a69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to load an object created from a model outside the project ?

[Matthieu]

Hi Gergely,

Thanks for you answer. 
I don't need to modify the object. I also tried with "dill" package but 
it's not working either. How can I use serialization with my example ?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b260b072-3a28-4ef3-97a7-d419e401ca80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Scaling/Parallel patterns for a View executing complex database transaction

[Me Sulphur]

Stack: Django 1.7 + Postgres 9.3 + Linux (No caching)

Our application has a view which is called/executed very frequently. The
view receives, parses and responds a JSON.

In between request and response, there are about 3-5 inserts and around
1200-5000 look ups depending upon some if..else business logic. At around
2-4 seconds the view is very slow.

However, a lot of the look ups (which are the bottlenecks) can be
parallelized. But I do not know how can I do the same within a
request-response cycle.

If it was a web UI, I could use celery+polling, since it a machine-machine
API call, the parallelisation has to be possible within a View's life cycle.

If parallelisation is not possible, what alternatives do I have for scaling
and reducing response time.


Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CABSvzZCGjY4DH2D2VtbsVDDHTAk-m%2BReJJ_BwmK_jn0wCDp%3DMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: What is the ideal web server to use with Django?

[Avraham Serour]

My main reason for recommending nginx is the config file, they are simpler
than apache,
when developing an application you shouldn't spend much time configuring
the web server,
only after you reached so much traffic that it would make sense to spend
time with it
On May 14, 2015 12:26 PM, "Tom Evans"  wrote:

> On Thu, May 14, 2015 at 4:36 AM, reduxionist 
> wrote:
> > The question you asked Tom was "Doesn't Apache create new process for
> each
> > request [thus eating memory when serving large amounts of static files
> > during traffic peaks]?", and the reason that Tom correctly answers "No"
> is
> > that as far as "serving large amounts of static files" goes you should be
> > using mpm-worker (multi-threaded Apache) which most definitely does not
> > spawn a new process for each request.
> >
> > The reason for those search results is that mpm-prefork does, however,
> spawn
> > a process per request,
>
> No, really, it does not. It only spawns a new process when there are
> no available workers to process an incoming request, and you have not
> reached the maximum number of workers that you have configured it to
> start. You can configure it to start all the worker processes you want
> when it starts up, and never to kill them off, and it will never spawn
> a new process.
>
> Apache processes are small, unless you do daft things like embed your
> web application in each worker process (mod_php style). This is the
> main complaint "Apache is eating all my memory" - it isn't, your web
> application you've embedded into Apache is eating all your memory.
>
> All of this is irrelevant for django, because with Apache you should
> use mod_wsgi in daemon mode, which separates out your web application
> processes from the web server.
>
> > but it is only needed for non-thread-safe
> > environments (most notoriously mod_php) and you shouldn't have to use it
> as
> > long as you've been a good coder and avoided global state in your Django
> app
> > (e.g. keep request-specific shared-state thread-local).
> >
> > I think the reason a lot of people seem to run mpm-prefork is just that
> it's
> > the default multi-processing module for Apache on most (all?) *nix
> platforms
> > and they don't know any better.
>
> Quite. We run a pair of Apache 2.4 reverse proxies in front of all of
> our (400+) domains, serving around 40 million requests per day,
> providing SSL termination and static file serving. We use event MPM
> and we have it scaled to support a peak of 2048 simultaneous
> connections. Load on the server never goes above 0.2, memory usage
> never goes above 1GB for the entire OS + applications, the rest of the
> RAM is used by the OS to cache the aforementioned static files.
>
> On our app servers we typically use Apache with worker MPM and
> mod_wsgi, although we have a few nginx+uwsgi sites, and I would dearly
> love some time to play around with a circusd + chausette + celery
> setup.
>
> The choice of web server is, these days, irrelevant. If it uses too
> much memory or can't handle enough users, it is never the fault of the
> web server, but instead of your application and/or configuration.
> Which is why I return to my original advice:
>
> > I am new to Django. I am building a app which will have to handle several
> > concurrent requests. Which web server is suitable for this?
>
> Any and all.
>
> Leave the fanboyism to the phone guys.
>
> Cheers
>
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAFHbX1KEVRM6WU7OCcLRSkJhpMS%2BfHpd7%2BWo7LO8XcEt8_f0Nw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFWa6tKVcxdXkZOOr2mu1xvLkJKz2XcS4HTZKRo8vwTTh0qo%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: What is the ideal web server to use with Django?

[Tom Evans]

On Thu, May 14, 2015 at 4:36 AM, reduxionist  wrote:
> The question you asked Tom was "Doesn't Apache create new process for each
> request [thus eating memory when serving large amounts of static files
> during traffic peaks]?", and the reason that Tom correctly answers "No" is
> that as far as "serving large amounts of static files" goes you should be
> using mpm-worker (multi-threaded Apache) which most definitely does not
> spawn a new process for each request.
>
> The reason for those search results is that mpm-prefork does, however, spawn
> a process per request,

No, really, it does not. It only spawns a new process when there are
no available workers to process an incoming request, and you have not
reached the maximum number of workers that you have configured it to
start. You can configure it to start all the worker processes you want
when it starts up, and never to kill them off, and it will never spawn
a new process.

Apache processes are small, unless you do daft things like embed your
web application in each worker process (mod_php style). This is the
main complaint "Apache is eating all my memory" - it isn't, your web
application you've embedded into Apache is eating all your memory.

All of this is irrelevant for django, because with Apache you should
use mod_wsgi in daemon mode, which separates out your web application
processes from the web server.

> but it is only needed for non-thread-safe
> environments (most notoriously mod_php) and you shouldn't have to use it as
> long as you've been a good coder and avoided global state in your Django app
> (e.g. keep request-specific shared-state thread-local).
>
> I think the reason a lot of people seem to run mpm-prefork is just that it's
> the default multi-processing module for Apache on most (all?) *nix platforms
> and they don't know any better.

Quite. We run a pair of Apache 2.4 reverse proxies in front of all of
our (400+) domains, serving around 40 million requests per day,
providing SSL termination and static file serving. We use event MPM
and we have it scaled to support a peak of 2048 simultaneous
connections. Load on the server never goes above 0.2, memory usage
never goes above 1GB for the entire OS + applications, the rest of the
RAM is used by the OS to cache the aforementioned static files.

On our app servers we typically use Apache with worker MPM and
mod_wsgi, although we have a few nginx+uwsgi sites, and I would dearly
love some time to play around with a circusd + chausette + celery
setup.

The choice of web server is, these days, irrelevant. If it uses too
much memory or can't handle enough users, it is never the fault of the
web server, but instead of your application and/or configuration.
Which is why I return to my original advice:

> I am new to Django. I am building a app which will have to handle several
> concurrent requests. Which web server is suitable for this?

Any and all.

Leave the fanboyism to the phone guys.

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFHbX1KEVRM6WU7OCcLRSkJhpMS%2BfHpd7%2BWo7LO8XcEt8_f0Nw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.