Optimizing DB query involving annotate and aggregate

[ST]

Hi,

I'm trying to optimize the run-time of getting total credit and debt values 
out of our database. Ideally I'd like to formulate it as a Django query. 
This is the raw SQL query I have, which produces the right answer and is 
very fast (milliseconds):

SELECT sum(tg.total) FROM
  (
  SELECT sum(t.amount) AS total, t.member_id AS member_id
  FROM club_transaction AS t
  WHERE t.member_id IS NOT NULL
  GROUP BY t.member_id
  ) AS tg
WHERE tg.total < 0

(plus a second query for > 0)

My Django implementation was:

m = Member.objects.annotate(balance=Sum('transaction__amount'))
m_debt = m.filter(balance__lt=0).aggregate(total=Sum('balance'))
m_credit = m.filter(balance__gt=0).aggregate(total=Sum('balance'))

which looks a lot nicer, is easier to understand and maintain.

However, it results in the following SQL query (slightly redacted):

SELECT SUM(balance) FROM
  (
  SELECT "club_member"."id" AS "id", {all the other fields}, 
SUM("club_transaction"."amount") AS "balance"
  FROM "club_member"
  LEFT OUTER JOIN "auth_user" ON ("club_member"."user_id" = 
"auth_user"."id")
  LEFT OUTER JOIN "club_transaction" ON ("club_member"."id" = 
"club_transaction"."member_id")
  GROUP BY "club_member"."id", {all the other fields}, 
"auth_user"."last_name", "auth_user"."first_name"
  HAVING SUM("club_transaction"."amount") < 0
  ORDER BY "auth_user"."last_name" ASC, "auth_user"."first_name" ASC
  ) subquery

(again, plus another one for > 0)
which is very slow (almost 1.5 seconds).

How can I construct a Django query which doesn't request (and group by) all 
the unnecessary other fields ?
I already tried playing around with only() and values() but never got it to 
work.

Looking forward to your responses!

best regards,
ST

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/367c1f59-870c-40d1-9a53-4feb08890f29%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Optimizing DB query involving annotate and aggregate

[ST]

On Wednesday, February 5, 2014 8:01:53 PM UTC, Anssi Kääriäinen wrote:
>
> Something like this might work:
> 
> Transaction.objects.values_list('member_id').annotate(total=Sum('amount')).filter(total__gt=0).aggregate(Sum('total'))
>

This didn't work - it produced a "SELECT  FROM" query, which obviously 
didn't work - tried adding 'amount' to the values_list, but that didn't 
help either. Eventually got it to work by using .only('member', 'amount') 
instead, and it *was* fast, but it didn't quite do what we need: we only 
want the *outstanding* credit/debt, i.e. some member might have many 
transactions, but they all add up to zero - so instead of increasing both 
total_debt and total_credit by that value, it shouldn't affect the totals. 
Which is why I wanted to total up by members first... any idea how to get 
that to work ?

 

>
> That is, don't start from Member, Django isn't smart enough to get rid of 
> the non-necessary joins. Instead go directly for the query you wrote in 
> SQL. In the ORM .values() is the way to control GROUP BY clause. Getting 
> the order of .values() and .annotate() right is important, and it is 
> sometimes hard to figure out how to do it correctly. Also, if you have some 
> extra ordering going on in the query just issue an empty .order_by() call - 
> that will get rid of all ordering.
>
>  - Anssi
>
> On Wednesday, February 5, 2014 8:11:29 PM UTC+2, ST wrote:
>>
>> Hi,
>>
>> I'm trying to optimize the run-time of getting total credit and debt 
>> values out of our database. Ideally I'd like to formulate it as a Django 
>> query. This is the raw SQL query I have, which produces the right answer 
>> and is very fast (milliseconds):
>>
>> SELECT sum(tg.total) FROM
>>   (
>>   SELECT sum(t.amount) AS total, t.member_id AS member_id
>>   FROM club_transaction AS t
>>   WHERE t.member_id IS NOT NULL
>>   GROUP BY t.member_id
>>   ) AS tg
>> WHERE tg.total < 0
>>
>> (plus a second query for > 0)
>>
>> My Django implementation was:
>>
>> m = Member.objects.annotate(balance=Sum('transaction__amount'))
>> m_debt = m.filter(balance__lt=0).aggregate(total=Sum('balance'))
>> m_credit = m.filter(balance__gt=0).aggregate(total=Sum('balance'))
>>
>> which looks a lot nicer, is easier to understand and maintain.
>>
>> However, it results in the following SQL query (slightly redacted):
>>
>> SELECT SUM(balance) FROM
>>   (
>>   SELECT "club_member"."id" AS "id", {all the other fields}, 
>> SUM("club_transaction"."amount") AS "balance"
>>   FROM "club_member"
>>   LEFT OUTER JOIN "auth_user" ON ("club_member"."user_id" = 
>> "auth_user"."id")
>>   LEFT OUTER JOIN "club_transaction" ON ("club_member"."id" = 
>> "club_transaction"."member_id")
>>   GROUP BY "club_member"."id", {all the other fields}, 
>> "auth_user"."last_name", "auth_user"."first_name"
>>   HAVING SUM("club_transaction"."amount") < 0
>>   ORDER BY "auth_user"."last_name" ASC, "auth_user"."first_name" ASC
>>   ) subquery
>>
>> (again, plus another one for > 0)
>> which is very slow (almost 1.5 seconds).
>>
>> How can I construct a Django query which doesn't request (and group by) 
>> all the unnecessary other fields ?
>> I already tried playing around with only() and values() but never got it 
>> to work.
>>
>> Looking forward to your responses!
>>
>> best regards,
>> ST
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3e239afe-037c-4ace-b9eb-ac2404f75981%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Optimizing DB query involving annotate and aggregate

[ST]

Managed it in the end, so for reference here is what I ended up with:

def total_credit_debt(self):
transaction_totals = 
Transaction.objects.filter(member__isnull=False).values('member').annotate(total=Sum('amount')).order_by()
creditors = transaction_totals.filter(total__gt=0)
debtors = transaction_totals.filter(total__lt=0)
total_credit = sum(row['total'] for row in creditors)
total_debt = sum(row['total'] for row in debtors)
return total_credit, total_debt

(@classmethod on TransactionManager)

Unfortunately I didn't manage to get it to do the second sum inside the 
database as well, but this is fast enough - there's only a few hundred 
members with non-zero balances. If you think there's any way this could be 
improved / cleaned up, I'd be happy to hear:)

Thanks!


On Thursday, February 6, 2014 7:59:37 PM UTC, arnonym wrote:
>
> On Wed, 5 Feb 2014 10:11:29 -0800 (PST) ST 
> > 
>
> wrote: 
> > Hi, 
> > 
> > I'm trying to optimize the run-time of getting total credit and debt 
> > values out of our database. Ideally I'd like to formulate it as a 
> > Django query. This is the raw SQL query I have, which produces the 
> > right answer and is very fast (milliseconds): 
> > 
> > SELECT sum(tg.total) FROM 
> >   ( 
> >   SELECT sum(t.amount) AS total, t.member_id AS member_id 
> >   FROM club_transaction AS t 
> >   WHERE t.member_id IS NOT NULL 
> >   GROUP BY t.member_id 
> >   ) AS tg 
> > WHERE tg.total < 0 
> > 
> > (plus a second query for > 0) 
> > 
> > My Django implementation was: 
> > 
> > m = 
> > Member.objects.annotate(balance=Sum('transaction__amount')) m_debt = 
> > m.filter(balance__lt=0).aggregate(total=Sum('balance')) m_credit = 
> > m.filter(balance__gt=0).aggregate(total=Sum('balance')) 
> > 
> > which looks a lot nicer, is easier to understand and maintain. 
> > 
> > However, it results in the following SQL query (slightly redacted): 
> > 
> > SELECT SUM(balance) FROM 
> >   ( 
> >   SELECT "club_member"."id" AS "id", {all the other fields}, 
> > SUM("club_transaction"."amount") AS "balance" 
> >   FROM "club_member" 
> >   LEFT OUTER JOIN "auth_user" ON ("club_member"."user_id" = 
> > "auth_user"."id") 
> >   LEFT OUTER JOIN "club_transaction" ON ("club_member"."id" = 
> > "club_transaction"."member_id") 
> >   GROUP BY "club_member"."id", {all the other fields}, 
> > "auth_user"."last_name", "auth_user"."first_name" 
> >   HAVING SUM("club_transaction"."amount") < 0 
> >   ORDER BY "auth_user"."last_name" ASC, "auth_user"."first_name" ASC 
> >   ) subquery 
> > 
> > (again, plus another one for > 0) 
> > which is very slow (almost 1.5 seconds). 
> > 
> > How can I construct a Django query which doesn't request (and group 
> > by) all the unnecessary other fields ? 
> > I already tried playing around with only() and values() but never got 
> > it to work. 
>
> I did something similar a few years back. Its next to impossible with 
> django-orm, just do it in raw sql. 
> The django-orm can't really do any advanced group_by clauses. And you 
> can't give them with extra(). Well, you can give them but they won't be 
> used;-) 
>
> Doing two (or three) orm-queries and then joining the data in python 
> will actually be slower then doing it all in hand-crafted sql. 
>
> So just do the sql by hand. And then optimize by having three columns, 
> one with the SUM(amount) if amount>0, one with the SUM(amount) if 
> amount<0 and one with the SUM(amount). Total credits, total depts and 
> balance all in one query (if possible)... 
>
> - Arnold 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/73bd4ddb-d6ee-461a-bed3-a3ee915b8859%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Django model problem

[St@n]

Hi there,

I am trying to create a RESTful API with a django website i have
running. The project folder is called rosebud and inside the folder r2
lies my application.

I have tried to retrieve objects by calling the Users table in
django's default authentication model.

But i have problems retrieving objects from other tables in my
database. Is it something to do with how i defined my model.py?

I have this table called timeline in my database but each time i run
the server and query it, it comes back with an error saying
r2_timeline table does not exist.

the error msg is sound because r2_timeline really does not exist. What
exists is the table TIMELINE.

This is what i have in my models.py:

from tastypie.utils import now
from django.contrib.auth.models import User
from django.db import models
from django.template.defaultfilters import slugify

class timeline(models.Model):
tweet_id = models.IntegerField(max_length=200)
keyword_id = models.IntegerField(max_length=200, blank=True)

def __unicode__(self):
return self


This is what i have in my api.py file

from django.contrib.auth.models import User
from tastypie.authentication import BasicAuthentication
from tastypie.authorization import DjangoAuthorization
from tastypie import fields
from tastypie.resources import ModelResource
from r2.models import timeline

class timelineResource(ModelResource):
class Meta:
queryset = timeline.objects.all()
resource_name = 'timeline'
excludes = ['id']
include_resource_uri = False

I would appreciate any help i can get to help me get a better idea as
i'm pulling my hair out already. Thank you so much!

Stanwin



-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Django models

[St@n]

Hello,

How do i ask django to query for a table that is already created in
the database but not part of the models.py?

the models.py is used to described the database and to create tables
with that models.py. However i already have my tables created prior to
creating the django app.

How then do i get django to query those tables created before?

Any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Creating a hierarchy of objects in my api.py

[St@n]

Hello,

I am playing around with tastypie and i want to create a hierarchy of
data.

What i currently have is this:


class keywordResource(ModelResource):
tweets = fields.ToManyField(timelineResource, 'tweets', full=True)
class Meta:
queryset = Keyword.objects.all()
resource_name = 'keyword'
excludes = ['id', 'keyword_id']
include_resource_uri = False

def alter_list_data_to_serialize(self, request, data_dict):
if isinstance(data_dict, dict):
if 'meta' in data_dict:
# Get rid of the "meta".
del(data_dict['meta'])
# Rename the objects.
data_dict['keyword'] = copy.copy(data_dict['objects'])
del(data_dict['objects'])
return data_dict



Can someone explain the relationship to me in creating such a
hierarchy?

in the line tweets = fields.ToManyField(timelineResource, 'tweets',
full=True)

it means that timelineResource is a child of Keyword right? and
"tweets" would be the column name in timeline table or just a generic
name?

or must i map a matching column that appears in both tables (keyword,
timeline)?


Thank you

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Writing some tests for new feature

[ST LEON]

I want to contribute one new feature to Django (doing this first time).

I reading this  and 
have one question.

First, write tests. I want to improve work of ping_google() 

 and 
add some new behavior. 

But I can't find any tests that cover this function. 

I think, that it might be here 
, but I 
can't find tests for *ping_google()*.

So, what should I do? That test needs me to see, how they works, change 
some of them and write new ones.

My changes aren't be so serious: just a very simple 2-3 new lines.

Please, help. 

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1a427e10-c658-4382-88f7-75a96961e0b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: Complex query reduction

[Robin St . Clair]

Hi
You are running into typical production Django problems - works fine in 
testing, terrible in production.
Firstly
If you use attributed many-to-manys, create the table outside of Django and use 
raw SQL.Create a primary key on the two things being related (usually larger 
table first).
Multi-column indicesTwo choicesA single index that includes both (do this 
outside Django)Two indices, one for each columnThe size will be roughly 
equivalent between the two
Single composite indexsmaller than two indexes.faster to update.accelerates a 
more restricted range of queries.An index on (A, B) does nothing for queries 
just using BBut on the queries it does accelerate, it will be faster.
Two indiceslargerslower to updatewill help on queries on both columns, but not 
as much as a single composite indexwill help accelerate queries on either of 
the columnscan be expressed directly in Django
Helpful Toolpg_stat_activity shows a wealth of useful information - including
which indexes are actually being used
If an index is not being used (or not being used very often) drop itif you are 
surprised that it is not being used, find out why
Basic Performance Rules1. Do not iterate over QuerySets2. If you think you have 
to iterate over a QuerySet, see rule #13. If you are absolutely, certainly, 
100% positive that the only possible solution to your problem is iterating over 
a QuerySet, see rule #3
Iteration - A Health Warning• Ignores Django’s lazy-evaluation mechanism and 
copies everything into local memory• Copies data from the database just to 
process it locally• Does filtration or summation in the application that is 
processed more efficiently in the database.• Databases are good at this sort of 
stuff let the DB do it
Alternatives to Iteration• QuerySet.update()• cursor.execute(“UPDATE 
reader_hours ...”)• Stored procedures
Take in the code below
How much many objects are in memory at point A?
qs = Orders.objects.all() # There are about 2,500,000 rows in “orders”
for order in qs:order.age_in_days += 1 # POINT A order.save()
Answer  - 2,500,000
Why Is This?
Django does lazy evaluation… (everyone tells me so!)The Django code carefully 
asks for a slice of 100 objectswhich trickles down through lots of really 
convoluted Python to psycopg2which dutifully asks for 100 rows from 
Postgreswhich sends all 2,500,000 over the wire
Solution - Named CursorsThe protocol between the Postgres client and server 
only does partial sends when using named cursorspsycopg2 fully supports named 
cursorsDjango doesn’t use named cursorsTherefore, the first time you ask for any 
object in a QuerySet, you get all of themThis is a very good reason not to ask 
for large result sets
INif using Django avoid the IN operation at all costs
If there are potentially more than 15 items in the list, rework the IN as a 
JOIN against whatever the source of the keys is

At some stage you are going to have to get to grips with SQL, might as well be 
sooner than later
R+C
PS some time ago one of my clients was only able to process 4 transactons a 
second against their very complicated DB. They needed to process 70+ 
transactions per second. After 3 days of code analysis we had the transaction 
rate up to 240 per second and so could get rid of the 6 week backlog that had 
developed. Front end coders didn't consider the DBMS to be importantDate: Sun, 
3 Nov 2013 02:37:09 -0800
From: akaar...@gmail.com
To: django-users@googlegroups.com
Subject: Re: Complex query reduction

You should rewrite the query into a form that doesn't require distinct. In 
general, when you see a query that has joins and DISTINCT, that should be an 
alarm bell that something isn't written correctly in the query. Unfortunately 
Django's ORM generates such queries, and that isn't easy to fix as there are 
databases that like DISTINCT more than rewriting the query to use subqueries.

In any case, you should check how to write the query without the need of 
distinct. Something like this should work:

self.items = BibliographicRecord.objects.listable_objects().filter(

authored__researcher__in=researchers,

)

self.items = BibliographicRecord.objects.listable_objects().filter(

pk__in=self.items.values_list('pk')

)

But maybe you can push the __in to deeper into the authored__researches 
lookup...

 - Anssi

On Saturday, November 2, 2013 4:50:42 PM UTC+2, Daniele Procida wrote:On Fri, 
Nov 1, 2013, Javier Guerra Giraldez  wrote:



>have you tried eliminating the second IN relationship?  something like

>

>entities = entity.get_descendants()

>

>items = BibliographicRecord.objects.filter

>(authored__researcher__person__member_of__entity__in=entities).distinct()



Indeed I have, but in that form it takes around 1770ms, compared to around 
1540ms in the original form. What I actually do is:



# breaking apart the queries allows the use of values_lists

entities = self.entity.get_descendants(

include_self=True

).values_list('id', f

Re: Complex query reduction

[Robin St . Clair]

Anssi

The last time I checked the use of IN, all the records from the database 
in the query were brought back to the workstation, rather than being 
processed on the backend and only the results returned to the workstation.


Have there been changes that carry out the entire query on the backend? 
What has changed to cause you to prefer the use of the IN statement?


R+C
On 08/11/2013 05:55, akaariai wrote:

On Sunday, November 3, 2013 1:48:07 PM UTC+2, Robin St.Clair wrote:

*IN*

  * if using Django avoid the IN operation at all costs


If there are potentially more than 15 items in the list, rework
the IN as a JOIN against whatever the source of the keys is


I don't necessarily agree with everything else said in the post, but 
this one is just plain wrong. It is completely OK to use 
__in=queryset. In fact, it is recommended in cases where the alternate 
is using join + distinct.


 - Anssi
--
You received this message because you are subscribed to the Google 
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-users+unsubscr...@googlegroups.com.

To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/6b5f9e3a-2e41-47b5-b0d6-94473fe323b8%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/BLU0-SMTP357C2324CC6BEC3F975EB4AE2F20%40phx.gbl.
For more options, visit https://groups.google.com/groups/opt_out.

html validator middleware

[Brian St. Pierre]

I recently hacked together a small middleware that validates all
outgoing html and if it encounters a validation error, throws a 500
status code and error page with the validation error message(s) and
html source.

Similar in aim to Luke Plant's Validator App (http://lukeplant.me.uk/
resources/djangovalidator/), but lighter weight.

Requires python 2.4 (for the subprocess module).
Requires the offline "validate" utility for HTML validation from
HTMLHelp.com, or its equivalent.

Available here: http://bstpierre.org/Projects/HtmlValidatorMiddleware/

Comments & questions welcome.

-Brian St. Pierre
[EMAIL PROTECTED]


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

using recaptcha with django's contrib comments module

[Eric St-Jean]

Hi,

Short form of my question: i have a custom view which does some work and 
then calls django.contrib.comments.views.comments.post_free_comment; 
i've customized the templates/comments/free_preview.html, and i would 
like my custom view to somehow pass an additional variable to that 
template, without having to do the whole form validation and 
re-instantiation that post_free_comment does. How?

Very long form, skip if you understand the short form:
I integrated recaptcha[1] into the django.contrib.comments comment form 
of a little blog app that i wrote with django (based on the django 
website blog app, actually, but with tagging).

So i wanted to keep using most of the code from django.contrib.comments 
(view, validation, actual posting, ...) while also using recaptcha.

The hack i came up with to do so with minimal work, was to copy and 
customize templates/comments/free_preview.html, incorporating the 
recaptcha code into it.

The django.contrib.comments.views.comments.post_free_comment view does 
everything i need (except for the captcha) for both previewing and 
posting. It's in the preview that i show the captcha, using the 
aforementionned template (no unnecessary requests to the recaptcha 
servers that way). But then, if it's the actual comment post (not the 
for-previewing post), i need to actually check that the user solved it. 
No problem: i redirected the /comments/postfree/ view to a custom view. 
If it's the previewing step, or if it's the posting step and the user 
solved the captcha it just goes
return post_free_comment(request)
which does the rest of the validation, and posts the comment.

My small problem is that at that point, i need to pass a simple variable 
to free_preview.html, which is the error code that recaptcha sent us 
(such as "the user can't read warped text"), so that the captcha on the 
refreshed screen will have error text[2] the problem is that to 
instantiate that template with its form, FreeComment's post_free_comment 
view uses a ton of code, and i had to duplicate a lot of it into my own 
view:

 try:
 options, target, security_hash = 
request.POST['options'], request.POST['target'], request.POST['gonzo'] 
   except KeyError:
 raise Http404, _("One or more of the required fields 
wasn't submitted")
 content_type_id, object_id = target.split(':') # target is 
something like '52:5157'
 option_list = options.split(',')
 new_data = request.POST.copy()
 new_data['content_type_id'] = content_type_id
 new_data['object_id'] = object_id
 new_data['ip_address'] = request.META['REMOTE_ADDR']
 new_data['is_public'] = IS_PUBLIC in option_list
 manipulator = PublicFreeCommentManipulator()
 return render_to_response('comments/free_preview.html', {
 'comment': "",
 'comment_form': oldforms.FormWrapper(manipulator, 
new_data, None),
 'options': options,
 'target': target,
 'hash': security_hash,
 'recaptcha_error': recatpcha_resp_text[1],
 }, context_instance=RequestContext(request))

(see http://dpaste.com/hold/12801/ for nicely formatted view)

all of that only to pass an extra var ('recaptcha_error') to the 
template... the rest of that code is verbatim from the default view.

after all this rambling, my question is quite simple: after having 
validated the captcha, if there was an error with it, i'd prefer to just 
  call django.contrib.comments.views.comments.post_free_comment so that 
*it* bothers itself with refreshing the preview form, rendering my 
customized template, and so that i don't duplicate the above code. But 
how can i pass some value to the template when i'm not the one 
instantiating it? Should i add the error text to the session and have 
the template somehow retrieve that? Can a custom templatetag have access 
to session data?

It all works; i just want it Pretty, and i'm sure there's a stupid 
solution that i'm too newb to see.

thank you for your kind attention...

[1] http://recaptcha.net

[2] See http://recaptcha.net/apidocs/captcha Basically, a little js 
function goes to the recaptcha server, which returns a form that gets 
inserted into the page (so that happens on the client side). If there 
was an error in the previous recaptcha validation, the validation server 
will have returned an error code. You add this error code to the url 
that the client will use to fetch the captcha again, which will then 
contain a nice little message telling the user what he did wrong.

-- 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this gro

flatpages and i18n???

[Eric St-Jean]

Hi,
Is there a way to specify different flatpages for the same url, but for
a different language???
So it would default to the one page if there's only one defined, but if 
language is fr-fr and there's a page of the same url defined for tha 
language, it would return that version instead.

I'll haphazardly guess that no, there isn't. I'm definitely willing to 
give that a go, however: a simple change to the model to allow for a 
language column, and to the middleware, to put in the logic to pick 
which version to use, should work, but surely someone here would have an 
opinion on how best to approach this. Please share if that's the case.

Thank you!

-- 
______
Eric St-Jean[EMAIL PROTECTED]



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: flatpages and i18n???

[Eric St-Jean]

nevermind...
i should have looked at the flatpages module before posting: it's super 
simple. I'll just copy it to my apps and modify it....

Eric St-Jean a écrit :
> Hi,
> Is there a way to specify different flatpages for the same url, but for
> a different language???
> So it would default to the one page if there's only one defined, but if 
> language is fr-fr and there's a page of the same url defined for tha 
> language, it would return that version instead.
> 
> I'll haphazardly guess that no, there isn't. I'm definitely willing to 
> give that a go, however: a simple change to the model to allow for a 
> language column, and to the middleware, to put in the logic to pick 
> which version to use, should work, but surely someone here would have an 
> opinion on how best to approach this. Please share if that's the case.
> 
> Thank you!
> 


-- 
__
Eric St-Jean[EMAIL PROTECTED]



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: using recaptcha with django's contrib comments module

[Eric St-Jean]

Hi,
Aidas Bendoraitis a écrit :
> You can do that by writing a wrapper for the
> django.contrib.comments.views.comments.post_free_comment view. It
> should take the request as an argument, check the validity of the
> captcha, add the additional parameter to the request, call
> the post_free_comment passing the request to it, and return its response.

that, i had done...

> Then you will need the request context processor
> "django.core.context_processors.request" to be activated in your
> TEMPLATE_CONTEXT_PROCESSORS. You will be able to access the parameter
> passed like this: {{ request.custom_parameter }}.

but this is precisely the info i was looking for.

thanks a bunch!

> Maybe this is not the nicest solution in the world, but it works.

-- 
__
Eric St-Jean[EMAIL PROTECTED]



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

admin forms lose javascript with mod_python

[Eric St-Jean]

I have a weird problem where if i run django standalone, i see the 
calendar widget by a DateField entry in the admin form, but when run 
from mod_python, i lose it. (i lose the calendar date picking widget, 
and the "today" shortcut)
/media *is* working - the css gets loaded properly. I'm using the same 
browser in both instances. If i load firebug, it sees the javascript 
files - i do think they are being served by apache along with the css ( 
i can look at them from the browser as well). But the widget doesn't 
appear, and i have to enter the date by hand. I was using the debian 
testing django (some form of 0.96), but then also tried trunk, with the 
same results (i even removed the deb package to be absolutely sure it 
wasn't used).
This is django trunk (as of yesterday), with apache 2.2.3-4 and 
mod_python 3.2.10-4, python 2.4.4-2 on debian 4.0.

Any quick ideas???
Thanks

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: database error instead of 400

[Brian St. Pierre]

On Jun 25, 7:30 pm, Francis Lavoie <[EMAIL PROTECTED]> wrote:
> OperationalError: unable to open database file

An addition to the page that Ramiro pointed to: the webserver user
needs (at least) execute permission (d--x--x--x) on every directory
below the one that holds the database file. Otherwise apache can't
even see the file or the parent directory.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: admin forms lose javascript with mod_python

[Eric St-Jean]

Graham Dumpleton a écrit :
> Do what has been suggested with using something like live http
> headers, but in particular pay close attention to the Content-Type
> header being returned for static files served from your /media
> directory. When using Apache the content types for such files will be
> dictated by Apache configuration and/or associated mime type files. It
> is possible that Django in standalone mode returns something a little
> different if it is serving up the static files in that case.

thanks for both of your replies. I used wireshark (previously known as 
ethereal) to follow the HTTP stream (being a low-level type of guy, i 
tend to use low-level type of tools :) ). Saw that the js files were 
returning 404's, even though the css's weren't...

It turns out i made a mistake in the apache/mod_python config. Because i 
serve multiple top-level urls from django, i use LocationMatch instead 
of Location, like so:

 SetHandler python-program
 PythonHandler django.core.handlers.modpython
 PythonInterpreter wwd
 SetEnv DJANGO_SETTINGS_MODULE wwd.settings
 PythonDebug On
 PythonPath "['/var/www/django','/var/www/django-projects'] + sys.path"


well, the regex also catches, e.g., /media/js/admin/DateTimeShortcuts.js
even though media is normally served statically. It doesn't catch the 
css media, though, because it's not in admin. Changing the regex to this:


solves the problem nicely.

PS: i use apache+mod_python simply because i have a few other apps that 
are raw mod_python, so i need apache+mod_python anyways - i might a well 
use it for django, too.

-- 
__
Eric St-Jean[EMAIL PROTECTED]



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

a generic JSON serializer???

[Eric St-Jean]

Hi,
There does not seem to be an easy way to serialize anything to JSON:
serializers.serialize will handle a QuerySet
simplejson.dumps will handle normal python data structures

But what about a python dict which has a django model instance in it, 
for example??? Or a Field?

Sure, i can (and did) write a trivial recursive function (it was a hack, 
i should make it a proper serializer) which looks into the structure and 
loops into the elements.
upon finding a list or dict or tuple, it calls itself.
Finding a queryset it calls serializers.serialize
Finding a base python type it calls simplejson.dumps
Finding a model instance, i made a stupid quick hack to get a queryset 
which has that instance only, and then i call serializers.serialize on it.

But i can hardly believe i'm the first one with this problem; did anyone 
make a generic serializer, which will handle both python stuff and 
django stuff (not just QuerySets)???

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

ModelAdmin

[Mario St-Gelais]

Hello group, first post here.
Quick question.  Can capabilities of ModelAdmin such as list_filter or
search_fields be used outside of the admin interface?

I want a view that would allow non logged users to search a site through
checkboxes and some search fields possibly and obviously, the way this is
implemented in the admin view is rather good.

Alternatively, what approach can be used in django?

Thanks
Mario

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFmF6_vLzivA3Y-sskqmcZzsuLc27opP6xW3m3wvmV7nepu%3D-A%40mail.gmail.com.