Re: Is this what you would expect from this Q query?

[akaariai]

On Thursday, November 14, 2013 2:17:43 PM UTC+2, Phoebe Bright wrote:
>
> After a long time trying to create a simple version that would replicate 
> the error, and failing, I've tracked it through the original code.
>
> Line 1140 in django/db/models/sql/query.py in function build_filter
>
> if current_negated and (lookup_type != 'isnull' or value is False):
> self.promote_joins(join_list)
> if (lookup_type != 'isnull' and (
> self.is_nullable(targets[0]) or
> self.alias_map[join_list[-1]].join_type == 
> self.LOUTER)):
> # The condition added here will be SQL like this:
> # NOT (col IS NOT NULL), where the first NOT is added in
> # upper layers of code. The reason for addition is that if 
> col
> # is null, then col != someval will result in SQL "unknown"
> # which isn't the same as in Python. The Python None 
> handling
> # is wanted, and it can be gotten by
> # (col IS NULL OR col != someval)
> #   <=>
> # NOT (col IS NOT NULL AND col = someval).
> clause.add((Constraint(alias, targets[0].column, None), 
> 'isnull', False), AND)  < this is adding the spurious clause
>
> This is django version 1.6 and happens using both the original Q version 
> and using your suggested .exclude version.  
> Having found it I'm still not sure what it means or how to avoid it being 
> triggered!
>
> Any suggestions?
>

I believe the query is actually correct. The idea is that 
.exclude(condition) produces complement of .filter(condition). The simplest 
example I can generate is using raw SQL on PostgreSQL:
akaariai=# create temp table foo(id integer);
CREATE TABLE
akaariai=# insert into foo values(null);
INSERT 0 1
-- What .filter(id__in=[1, 2, 3]) will do:
akaariai=# select * from foo where id in (1, 2, 3);
 id 

(0 rows)

-- If the  "id is not null" condition isn't there
akaariai=# select * from foo where not (id in (1, 2, 3));
 id 

(0 rows)

-- Note: we get still zero rows while we should get the complement, that is 
one row.
-- Add in the id is not null condition
akaariai=# select * from foo where not (id in (1, 2, 3) and id is not null);
 id 

   
(1 row)
-- Now we got the complement of the .filter() query.

The problem comes from this feature in SQL:
  NULL in (1, 2, 3) => unknown
  NOT (NULL in (1, 2, 3)) => unknown
so, NOT (condition) doesn't produce complement of (condition) in SQL! To 
avoid that problem we have to add the AND id IS NOT NULL into the condition.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/133e9b0b-1e93-47fc-82ac-4a97d0cca021%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Complex query reduction

[akaariai]

On Friday, November 8, 2013 8:44:09 AM UTC+2, Robin St.Clair wrote:
>
>  Anssi
>
> The last time I checked the use of IN, all the records from the database 
> in the query were brought back to the workstation, rather than being 
> processed on the backend and only the results returned to the workstation.
>
> Have there been changes that carry out the entire query on the backend? 
> What has changed to cause you to prefer the use of the IN statement?
>

Django has had the ability to execute __in=qs in single query for a 
somewhat long time (from Django 1.2 maybe?).

It is true that __in lookup against a large list of values is often a bad 
choice even if you have the values at hand. Unfortunately this doesn't 
apply to all databases, for example older versions of MySQL do not handle 
subqueries well.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/f3428fda-e4be-4d75-8626-a95bd20f66c8%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Complex query reduction

[akaariai]

On Sunday, November 3, 2013 1:48:07 PM UTC+2, Robin St.Clair wrote:
>
> *IN*
>
>- if using Django avoid the IN operation at all costs
>
>
> If there are potentially more than 15 items in the list, rework the IN as 
> a JOIN against whatever the source of the keys is
>

I don't necessarily agree with everything else said in the post, but this 
one is just plain wrong. It is completely OK to use __in=queryset. In fact, 
it is recommended in cases where the alternate is using join + distinct.

 - Anssi 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/6b5f9e3a-2e41-47b5-b0d6-94473fe323b8%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django 1.6 transactions witn multiple databases

[akaariai]

On Thursday, November 7, 2013 3:36:17 PM UTC+2, Vaal wrote:
>
> Hi
> How to be in 1.6 with queries that involve multiple databases?
> Previously recommended to create custom TransactionMiddleware. It was 
> convenient. Now TransactionMiddleware is deprecated.
>

You can set ATOMIC_REQUESTS per database.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/404f3556-5d44-4325-a47f-ae36cfd5cd3c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Complex query reduction

[akaariai]

You should rewrite the query into a form that doesn't require distinct. In 
general, when you see a query that has joins and DISTINCT, that should be 
an alarm bell that something isn't written correctly in the query. 
Unfortunately Django's ORM generates such queries, and that isn't easy to 
fix as there are databases that like DISTINCT more than rewriting the query 
to use subqueries.

In any case, you should check how to write the query without the need of 
distinct. Something like this should work:

self.items = BibliographicRecord.objects.
listable_objects().filter( 
authored__researcher__in=researchers, 
)
self.items = BibliographicRecord.objects.
listable_objects().filter( 
pk__in=self.items.values_list('pk') 
)

But maybe you can push the __in to deeper into the authored__researches 
lookup...

 - Anssi

On Saturday, November 2, 2013 4:50:42 PM UTC+2, Daniele Procida wrote:
>
> On Fri, Nov 1, 2013, Javier Guerra Giraldez > 
> wrote: 
>
> >have you tried eliminating the second IN relationship?  something like 
> > 
> >entities = entity.get_descendants() 
> > 
> >items = BibliographicRecord.objects.filter 
> >(authored__researcher__person__member_of__entity__in=entities).distinct() 
>
> Indeed I have, but in that form it takes around 1770ms, compared to around 
> 1540ms in the original form. What I actually do is: 
>
> # breaking apart the queries allows the use of values_lists 
> entities = self.entity.get_descendants( 
> include_self=True 
> ).values_list('id', flat=True) 
>
> # and the set() here is about 230ms faster than putting a distinct() 
> on 
> # the first query 
> researchers = set(Researcher.objects.filter( 
> person__entities__in=entities 
> ).values_list('person', flat=True)) 
>
> self.items = BibliographicRecord.objects.listable_objects().filter( 
> authored__researcher__in=researchers, 
> ).distinct() 
>
> I think that's partly because this way the SELECT doesn't have to grab all 
> the fields of publications_bibliographicrecord. 
>
> But, the real killer is the combination of ordering (in the queryset or on 
> the model, it doesn't matter) with the distinct() - as soon as one is 
> removed from the equation, the execution time drops to around 250ms. 
>
> That's for 55000 BibliographicRecords created by that last operation 
> (before distinct() is applied; distinct() reduces them to 28000). 
>
> That seems excessive to me. 
>
> BibliographicRecord has a custom primary key, and its id fields look like 
> "d9ce7e2f-663e-4fc6-8448-b214c6915aed:web-of-science". Could that be 
> implicated in performance? 
>
> Daniele 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8208cc22-cc94-4fe6-9245-709bdc42647f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Complex query

[akaariai]

On Monday, September 9, 2013 12:13:09 AM UTC+3, Yegor Roganov wrote:
>
> I found out this is a so called "top n per group" problem and it's not 
> that easy to implement in django. Maybe someone could help me to find 
> another way (for example, alter the schema in some way)?
>

This is something that would be nice to support with prefetch_related(). 
This is exactly the type of query where ORM abstraction would be very 
welcome. Writing this in SQL isn't easy, and the most efficient way to 
write the query varies a lot depending on the used database backend.

Unfortunately I don't see other solutions than running multiple queries or 
using raw SQL.

 - Anssi


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: What's the new "field" parameter in the ManyToOneRel __init__ function?

[akaariai]

On Saturday, September 7, 2013 6:52:14 AM UTC+3, Mark Young wrote:
>
> For a little bit of context, here's an example of code that perviously 
> worked but now doesn't:
>
> class CategoryAdminForm(forms.ModelForm):
> """Form for Category's Admin"""
> parent = TreeNodeChoiceField(
> label=_('Parent category'),
> level_indicator='|--', required=False,
> empty_label=_('No parent category'),
> queryset=Category.objects.all())
>
> def __init__(self, *args, **kwargs):
> super(CategoryAdminForm, self).__init__(*args, **kwargs)
> rel = ManyToOneRel(Category, 'id')
> self.fields['parent'].widget = RelatedFieldWidgetWrapper(
> self.fields['parent'].widget, rel, self.admin_site)
>

Seems to be the other side of the relation. So, you need the foreign key 
pointing to Category.id, from the above code it isn't obvious what that 
field is. Maybe something like Category._meta.get_field('parent_id') is 
correct?

This might allow both 1.5 and 1.6 to work:
 if django's version is 1.6:
rel = ManyToOneRel(Category._meta.get_field('parent_id'), Category, 
'id')
else:
rel = ManyToOneRel(Category, 'id')

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: .filter() and .exclude() don't add up

[akaariai]

On Tuesday, July 30, 2013 6:26:47 PM UTC+3, Daniele Procida wrote:
>
> On Mon, Jul 29, 2013, akaariai > wrote: 
>
> >> I understood that part. But by "more general" I mean one that will work 
> >> for any case, without having to know where the Nulls might be. 
> >> 
> >> So given queryset A, and its subset queryset B, we can place B against 
> A 
> >> and obtain its complement. 
> >> 
> >> Or to put it another way: give me all the items in A that are not in B. 
> >> 
> > 
> >You can do this with a subquery in Django. non_red_things = 
> >queryset.exclude(pk__in=red_things). If this performs well is a different 
> >thing. 
>
> It seems to take about twice as long to execute, so no, it doesn't perform 
> very well. 
>
> >I think that in SQL one can use WHERE (original_condition) is not true; 
> >which will match both unknown (null comparison's result) and false in the 
> >original condition. 
>
> But this isn't available as a Django query, without using raw SQL? 
>
>  
No it isn't. Writing a patch that adds QuerySet.negate() operation would be 
fairly straightforward. If such an operation will be accepted to Django is 
a different question. In my opinion the main question is if queries written 
as "WHERE (original_condition) is not true" will perform well enough. If 
not, then adding the operation isn't a good idea, but if it generally 
performs well, then addition of it seems like a good idea to me.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: invalid reference to FROM-clause entry for table

[akaariai]

On Tuesday, July 30, 2013 6:40:27 AM UTC+3, hippomoe wrote:
>
> I'm using djorm-ext-pgfulltext extension to do full-text-search using 
> Postgres on a title field for one of my models (Bookmark).
>
> class Bookmark(TimeStampModel):
> title = models.CharField(max_length = 100)
> # other fields
>
> search_index = VectorField()
>
> objects = SearchManager(
> fields = ('title'),
> config = 'pg_catalog.english',
> search_field = 'search_index',
> auto_update_search_field = True
> )
>
> I have another model called SharedBookmark that is OneToOne related to the 
> Bookmark.
>
> class SharedBookmark(TimeStampedModel):
> bookmark = models.OneToOneField(Bookmark)
> # other fields
>
> I am able to do a search through my Bookmark instances using:
>
> Bookmark.objects.search(query)
>
> But when I try to do the following filter
>
> SharedBookmark.objects.filter(bookmark__in=Bookmark.objects.search(query))
>
> I receive the following error:
>
> invalid reference to FROM-clause entry for table "bookmarks_bookmark" LINE 
> 1: ...ECT U0."id" FROM "bookmarks_bookmark" U0 WHERE ( ("bookmarks... ^ 
> HINT: Perhaps you meant to reference the table alias "u0"
>
> This is the traceback:
>
> Traceback:
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/core/handlers/base.py"
>  
> in get_response
>   115. response = callback(request, 
> *callback_args, **callback_kwargs)
> File "/home/derek/Development/skillfare/skillfare/bookmarks/views.py" in 
> search
>   76. print shared_bookmarks
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/models/query.py"
>  
> in __repr__
>   77. data = list(self[:REPR_OUTPUT_SIZE + 1])
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/models/query.py"
>  
> in __len__
>   92. self._result_cache.extend(self._iter)
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/models/query.py"
>  
> in _safe_iterator
>   344. for item in iterator:
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/models/query.py"
>  
> in iterator
>   301. for row in compiler.results_iter():
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py"
>  
> in results_iter
>   775. for rows in self.execute_sql(MULTI):
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py"
>  
> in execute_sql
>   840. cursor.execute(sql, params)
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/backends/util.py"
>  
> in execute
>   41. return self.cursor.execute(sql, params)
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py"
>  
> in execute
>   58. six.reraise(utils.DatabaseError, 
> utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
> File 
> "/home/derek/.virtualenvs/skillfare/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py"
>  
> in execute
>   54. return self.cursor.execute(query, args)
>
> I can't make sense of the error. What causes this error and how should I 
> fix it? 
>
> Thanks for any advice!
> hippomoe
>

The reason for this error is that djorm-ext-pgfulltext doesn't support 
alias relabeling. The table aliases need to be relabeled when a query is 
used as a subquery (happen for bookmark__in=query). When this happens, all 
parts of the query that reference table aliases need to be relabeled so 
that the references continue to point to correct tables. A likely reason 
why this doesn't work in pgfulltext is that it uses .extra() somewhere, and 
.extra() doesn't support alias relabeling at all.

You can try to fix the bug in pgfulltext, or you could use 
bookmark__in=list(Bookmark.objects.search(...)). This will issue two 
separate queries.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: .filter() and .exclude() don't add up

[akaariai]

On Sunday, July 28, 2013 10:44:49 AM UTC+3, Daniele Procida wrote:
>
> On Sat, Jul 27, 2013, Steve McConville > 
> wrote: 
>
> >Perhaps I'm not sure exactly what you mean by "more general", but I 
> >was recommending something like 
> > 
> >red_things = queryset.filter(Q(color="red")) 
> >non_red_things = queryset.filter(~Q(color="red") | Q(color__isnull=True) 
> > 
> >This will produce SQL like 
> > 
> >SELECT * FROM queryset WHERE color IS 'red'; 
> >SELECT * FROM queryset WHERE color IS NOT 'red' OR color IS NULL; 
> > 
> >The set non_red_things will be the complement of red_things. 
>
> I understood that part. But by "more general" I mean one that will work 
> for any case, without having to know where the Nulls might be. 
>
> So given queryset A, and its subset queryset B, we can place B against A 
> and obtain its complement. 
>
> Or to put it another way: give me all the items in A that are not in B. 
>

You can do this with a subquery in Django. non_red_things = 
queryset.exclude(pk__in=red_things). If this performs well is a different 
thing.

I think that in SQL one can use WHERE (original_condition) is not true; 
which will match both unknown (null comparison's result) and false in the 
original condition.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Creating database views, functions, stored procedures etc

[akaariai]

On 20 kesä, 14:32, Mark Robson  wrote:
> Hi,
>
> I've got a Django application which uses a mixture of managed and unmanaged
> models.
>
> The managed models get their tables created in the usual way, the unmanaged
> ones via a number of .sql files in the application/sql directory.
>
> This works fine, but I also have some views and user-defined functions in
> the database. Until now I've been manually creating these after
> installation.
>
> However, I'm trying to add better unit-tests, some of which will need to
> use the views and functions.
>
> I've done a lot of research and apparently drawn a blank on this. I can
> hook the post_syncdb signal in my application, but this is executed after
> syncdb, but before custom SQL has been run.
>
> Or alternatively, I could "tack on" the views and functions to one of my
> models' .sql files, and hope that it works - but of course, the views are
> executing joins, so I need the tables they depend upon to be all created
> beforehand.
>
> (NB: I'm using Postgresql, if it makes any difference)
>
> Does anyone have a suggestion for how to handle this?
>
> Thanks
>
> Mark

You will likely want to use a custom test runner. Override its
setup_databases method and run your custom SQL there.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django foreing keys joins

[akaariai]

On 6 kesä, 02:05, Àlex Pérez  wrote:
> Can someone tell me why the or in that query:
> print Counter.on_medicos.filter(loc__isnull=True).only("id").query
>
> ""SELECT `web_counter`.`id` FROM `web_counter` LEFT OUTER JOIN
> `locations_localidad` ON (`web_counter`.`loc` = `locations_localidad`.`id`)
> WHERE (`web_counter`.`mode` = 1  AND `locations_localidad`.`id` IS NULL)"""
>
> The orm is making a JOIN?? i can't understant!

1.6 should not create a join in this case.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is it possible to add SQL comments to a query built with the ORM?

[akaariai]

There is no public API way. You can likely do this with .extra():
Object.objects.filter(Q(pub_date__lte=datetime.now)).extra(where=['1=1 /
* query no. 123 */'])

 - Anssi

On 30 touko, 12:42, Timo Josten  wrote:
> Hint: This is a cross-post 
> fromhttp://stackoverflow.com/questions/16831162/is-it-possible-to-add-sql...
>
> I am trying to identify slow querys in a large-scale Django 1.3 web
> application. As it is kind of difficult to match the raw sql query in the
> slow query log with the specific ORM statement in the code, I wondered if
> it is possible to add a SQL comment to the query constructed with the ORM,
> something like..
>
> > Object.objects.filter(Q(pub_date__lte=datetime.now)).comment('query no.
> > 123')
>
> Kind regards
> Timo

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ORM Prefetch related and only()

[akaariai]

There is no way to alter the behavior of prefetch_related. It fetches
the related objects with the query defined by the related manager and
that's it.

 - Anssi

On 29 touko, 19:40, Àlex Pérez  wrote:
> I want to know if there is a way that the query that is executed to get the
> prefetch related information can get only certains values, actually that
> only works for select_related,
>
> May be i have done something wrong but the prefetch_related query is
> getting all the fields of the model.
>
> Someone can help me please?
>
> --
> Alex Perez
> alex.pe...@bebabum.com
>
>  *bebabum* be successful
>
> c/ Còrsega 301-303, Àtic 2
> 08008 
> Barcelonahttp://www.bebabum.comhttp://www.facebook.com/bebabumhttp://twitter.com/bebabum
>
> This message is intended exclusively for its addressee and may contain
> information that is confidential and protected by professional privilege.
> If you are not the intended recipient you are hereby notified that any
> dissemination, copy or disclosure of this communication is strictly
> prohibited by law.
>
> Este mensaje se dirige exclusivamente a su destinatario y puede contener
> información privilegiada o confidencial. Si no es vd. el destinatario
> indicado,
> queda notificado que la utilización, divulgación y/o copia sin autorización
> está prohibida en virtud de la legislación vigente.
>
> Le informamos que los datos personales que facilite/ha facilitado pasarán a
> formar parte de un fichero responsabilidad de bebabum, S.L. y que tiene
> por finalidad gestionar las relaciones con usted.
> Tiene derecho al acceso, rectificación cancelación y oposición en nuestra
> oficina ubicada en c/ Còrsega 301-303, Àtic 2 de Barcelona o a la dirección
> de e-mail l...@bebabum.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Object Lookup after Save

[akaariai]

On 29 touko, 01:07, Chris Conover  wrote:
> Adding commit_unless_managed() before the get() seems to fix the problem.
> Looking at it locally, the Gearman worker starts a transactions and just
> calls commit when the processing is done, over and over, never starting a
> new transaction. That combined with REPEATABLE-READ seems to be the culprit.
>
> My next step was going to be to set the isolation level to READ-COMMITTED
> which would probably have fixed it as well.  Thought I'm not sure how that
> would have affected performance.
>
> Will the changes to transactions in 1.6 possibly address this issue so I
> can remove the commit_unless_managed() call? It looks like the major change
> is using the underlying database's autocommit mode rather than emulating it
> in the ORM. I suppose I'll have to test...

The thing is Django didn't even emulate the autocommit mode in pre 1.6
versions. The behavior in 1.5 is this:
  1. Any query will start a transaction (unless of course you are
already in transaction)
  2. An ORM write will commit after the write is done (but not
rollback on exception!)
  3. Any other write will not commit

This is changed to:
  1. Any query will happen in autocommit mode - that is there are no
user-visible transactions.
  2. Some ORM write operations are wrapped in transaction (unless
already in transaction). Again, this is not user visible. Reason is
that for example multitable save() should be atomic.

That is, in 1.6 the mode is autocommit with some extra protection for
certain ORM write operations.

Another significant change: In pre 1.6 you can enter transaction block
by @commit_on_success. All queries inside the block will be part of
the same transaction. The name should be treated literally. If you
nest commit_on_success decorators you might get surprising behavior:
with commit_on_success: # Starts a transaction
with commit_on_success: # Uses the same transaction
obj.save()
# Second commit_on_success commits on success. Note however that
the first commit_on_success started the transaction.
otherobj.save()
raise Exception("what happens?")

End situation is that obj.save() was committed, otherobj.save()
wasn't. A new transaction was started after the inner
commit_on_success block exited. In 99% of cases this isn't what was
wanted.

In 1.6 the code should be converted to:
with atomic: # Starts a transaction
with atomic: # Creates a savepoint
obj.save()
# The savepoint is released
otherobj.save()
raise Exception

Now, the exception will cause a rollback of the transaction. Both obj
and otherobj save are rolled back.

All in all 1.6 transactions should be a lot easier to understand
(there isn't one unless you explicitly asked for a transaction), and
the atomic decorator is more powerful than commit_on_success.

I am not sure if you will need some form of commit in 1.6 before
the .get(). It depends on how the code is structured, and what does
start the problematic transaction. If the code uses some form of
explicit transaction control (commit_on_success, commit_manually) then
the autocommit behavior will not be in effect, and you will need
manual commit. If autocommit mode is in effect, things should just
work.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Object Lookup after Save

[akaariai]

On 28 touko, 22:20, Chris Conover  wrote:
> Well, you can inspect the object and see it's primary key. Surely that
> means the INSERT is completed? So the workflow goes like this:
>
> foo = Foo()
> foo.save()
> foo.pk # new primary key is available
> submit_gearman_task(foo.pk)
>
> Then in the Gearman worker:
>
> foo = Foo.objects.get(pk=foo_pk) # this causes a Foo.DoesNotExist exception
>
> That's what I don't understand. The primary key is assigned and available
> in the application layer in one place but then the lookup using that
> primary key fails in another place. Maybe it has something to do with the
> fact that the database is set to REPEATABLE-READ isolation level versus
> READ-COMMITTED. Still investigating...

Could it be that the Gearman worker is in a transaction? When you
issue Foo.objects.get(), if the transaction Gearman is in has started
before the save() you will not see the new object (as MySQL uses
REPEATABLE READ transaction).

Note that any read query will start a transaction implicitly, and ORM
writes commit implicitly started transaction. You can also use
connection.commit_unless_managed() to commit implicitly started
transaction (not part of public API!). So, even if you haven't
explicitly started any transactions in Gearman worker an earlier read
query might have started one for you. (If you find this behavior
confusing this is one of the reasons why transaction handling will be
changed in Django 1.6).

In short: try if connection.commit_unless_managed() before the get()
changes anything. If so, investigate why there is an ongoing
transaction in the worker.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Testing with complex database setups

[akaariai]

On 23 touko, 15:09, Jani Tiainen  wrote:
> Hi,
>
> I've product that uses complex setup scripts - tens of SQL scripts that 
> creates tables, views, complex triggers, grants permissions and such and 
> finally populates database with some pre-data.
>
> So how I should proceed with unit testing with database like that, since some 
> operations rely heavily that there really exists all that trigger-function 
> mess in the database? So that I don't need everytime to start from the 
> scratch but from some known state of the db?

I have some code I run my tests by the doing the following:
  1. dump schema + some data from production
  2. create test database by loading the dump.
  3. Apply any migrations I have (these are plain SQL scripts, no
South involved here)
  4. run tests using SimpleTestCase or plain unittest TestCases.

So, I am not using Django's testing framework at all. There are two
reasons: 1. I haven't found a nice way to create the database with
custom scripts etc. loaded. 2. Django's test framework likes to flush
all data from the database. This doesn't fit my needs.

So, I suggest that you either hack a custom test runner for Django, or
just run tests without Django's testing framework. There might exists
something reusable in the ecosystem for this use case.

It would be nice to have something that allows easy testing this way
in Django. The problem is that once you enter complex database setups
domain it is hard to find a minimal useful feature set.

Unfortunately I don't have anything reusable for running tests this
way, my scripts for managing the above steps are too hacky to reuse.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django's cascade delete is executing the DELETE SQL twice

[akaariai]

On 22 touko, 18:57, Michael  wrote:
> I've got an odd situation here, any idea why the delete SQL for the
> `myapp_entry_tag` table is being performed twice?
>
> >>> from myapp.models import Tag
> >>> from django.db import connection
> >>> from pprint import pprint
> >>> Tag.objects.only('pk').get(pk=1).delete()
> >>> pprint(connection.queries)
>
> [{u'sql': u'SELECT `myapp_tag`.`id` FROM `myapp_tag` ''WHERE
> `myapp_tag`.`id` = 1 ',
>   u'time': u'0.000'},
>  {u'sql': u'DELETE FROM `myapp_entry_tag` WHERE `myapp_entry_tag`.`tag_id`
> IN (1)',
>   u'time': u'0.000'},
>  {u'sql': u'DELETE FROM `myapp_entry_tag` WHERE `myapp_entry_tag`.`tag_id`
> IN (1)',
>   u'time': u'0.000'},
>  {u'sql': u'DELETE FROM `myapp_tag` WHERE `id` IN (1)', u'time': u'0.000'}]
>
> models.py:
>
>     class Tag(models.Model):
>         ...
>
>     class EntryTag(models.Model):
>         ...
>         tag = models.ForeignKey(Tag)

The answer is: this should not happen, and I wasn't able to reproduce
this using Django master.

What version of Django is this? Could it be that you have some custom
code running (signal handlers for example)? Can you produce a small
self-contained Django project where this happens?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Pickling of dinamically created models.Model subclasses: "attribute lookup failed", django 1.5

[akaariai]

I have finally found the time to work on this issue. There is a patch
in the ticket (https://code.djangoproject.com/ticket/20289), and I
think the patch does solve the regression.

I didn't find a simple test case that works in 1.4 but fails in 1.5.
If anybody knows how to reproduce the original regression that would
be very useful in ensuring the regression is really solved.

 - Anssi


On 19 huhti, 02:06, akaariai  wrote:
> On 18 huhti, 22:07, Greg H  wrote:
>
> > Running into a similar issue on my own project, seems to be when you try an
> > cache a model which has a related model which in turn has a many to many
> > field. So for example, I have an instance of a Student, which has a
> > ForeignKey to Book, which in turn has a ManyToMany to Author. If I try and
> > cache my Student instance, I get that pickling error. In Django 1.4 it
> > worked fine, but not in 1.5.
>
> > Figure out a resolution to this?
>
> The problem is that for an instance of a class to be picklable the
> class must be available as a module level attribute somewhere. For
> dynamically created classes this is usually not true - they aren't
> available at module level. One set of such classes is automatically
> created through models for m2m fields. The automatic through classes
> could explain the ManyToMany problem.
>
> All in all this looks like a regression that should be fixed in 1.5. I
> created a ticket for this, see:https://code.djangoproject.com/ticket/20289
>
>  - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: How to check for a specific field change in a model?

[akaariai]

On 18 huhti, 20:34, deepankar bajpeyi  wrote:
> I have my models.py :
>
> class Hotel(models.Model):
>     name = models.CharField(max_length=20)
>     currency = models.ForeignKey(Currency)
>
> class Currency(models.Mode):
>     code = models.CharField(max_length=3)
>     name = models.CharField(max_length=10)
>
> Whenever the currency field in hotel is changing I need to be able to do
> something. So I have a function like :
>
> @receiver(pre_save,sender=Hotel)def update_something(sender,**kwargs)
>     obj = kwargs['instance']
>     old_object = Hotel.objects.get(pk=obj.pk)
>     '''
>      Now I can do anything here comparing the oldo  object with the instance
>     '''
>
> The thing is I don't want to make a query for this, since then the purpose
> of signals becomes stupid and I become a fool.
>
> So I should be able to do something like :
>
> updated = kwargs['update_fields']
> new_currency = updated['currency']
>
> Is their a way I can find out that change for *ONLY* one particular field
> say currency ,instead of doing a query like this. I want to get the changes
> related to the currency foreign key and update things before I save.
>
> Thanks :)

Unfortunately there isn't any good way to do this, just doing an extra
query on save seems like the best way forward.

Django doesn't currently store the DB fetched values anywhere. The
reason for this is that doing so is somewhat expensive. It would be
nice to offer doing this as an option, this would allow for some nice
features, like automatic update of only changed fields, and updatable
natural primary keys.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Pickling of dinamically created models.Model subclasses: "attribute lookup failed", django 1.5

[akaariai]

On 18 huhti, 22:07, Greg H  wrote:
> Running into a similar issue on my own project, seems to be when you try an
> cache a model which has a related model which in turn has a many to many
> field. So for example, I have an instance of a Student, which has a
> ForeignKey to Book, which in turn has a ManyToMany to Author. If I try and
> cache my Student instance, I get that pickling error. In Django 1.4 it
> worked fine, but not in 1.5.
>
> Figure out a resolution to this?

The problem is that for an instance of a class to be picklable the
class must be available as a module level attribute somewhere. For
dynamically created classes this is usually not true - they aren't
available at module level. One set of such classes is automatically
created through models for m2m fields. The automatic through classes
could explain the ManyToMany problem.

All in all this looks like a regression that should be fixed in 1.5. I
created a ticket for this, see: https://code.djangoproject.com/ticket/20289

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: field lookups problem

[akaariai]

On 22 maalis, 15:44, Roberto López López  wrote:
> Hi,
>
> I have a problem with my data model while doing field lookups. This is
> my models.py:
>
> from django.db import models, IntegrityError
>
> # Create your models here.
>
> class Model1(models.Model):
>     title = models.CharField(max_length=15)
>     models2 = models.ManyToManyField('Model2', through='ThroughModel')
>
>     def __unicode__(self):
>         return self.title
>
> class Model2(models.Model):
>     title = models.CharField(max_length=15)
>
>     def __unicode__(self):
>         return self.title
>
> class ThroughModel(models.Model):
>     model1 = models.ForeignKey(Model1)
>     model2 = models.ForeignKey(Model2)
>     lead = models.BooleanField(default=False)
>
>     def __unicode__(self):
>         return u'{0} - {1} - {2}'.format(self.model1, self.model2,
> self.lead)
>
> Testing it on the django shell:
>
> >>> m1 = Model1.objects.create(title='blabla')
> >>> m2 = Model2.objects.create(title='blabla2')
> >>> m1.__eq__(m2)
>
> False                                                 # OBVIOUSLY>>> t = 
> ThroughModel.objects.create(model1=m1, model2=m2)
> >>> ThroughModel.objects.filter(model1__exact=m1)
>
> []            # OK>>> 
> ThroughModel.objects.filter(model1__exact=m2)
>
> []            # NOT OK!!!
>
> Am I missing anything? Can anyone spot the problem?
>
> Thanks for your advice.
>
> Regards,
>
> Roberto

The problem here is that Django doesn't do any verification that the
given model values for lookups are of correct type.  My bet is that
m1.pk == m2.pk. So, Django will create a query using m2.pk in
model1__exact=m2, and this is the reason you get the results you get.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Django & Oracle connection problem

[akaariai]

On 25 tammi, 00:27, Dylan Klomparens 
wrote:
> I have a Django program that is connecting to an Oracle database. In my
> settings.py file I have this configuration:
>
> DATABASES = {
>   'default': {
>     'ENGINE': 'django.db.backends.oracle',
>     'NAME': 'xe',
>     'USER': 'MY_USER_NAME',
>     'PASSWORD': 'abcdefghijklmnopqrstuvwxyz',
>     'HOST': 'db_server.example.com',
>     'PORT': '1234',
>   }}
>
> I received a strange error when attempting to load the website:
>
> ORA-28547: connection to server failed, probable Oracle Net admin error
>
> After further investigation, I sniffed the TCP traffic between the
> webserver and the database server. I discovered this text in the network
> communication, which I reformatted for this post:
>
> (DESCRIPTION=
>     (ADDRESS=
>         (PROTOCOL=TCP)
>         (HOST=1.2.3.4)
>         (PORT=1234)
>     )
>     (CONNECT_DATA=
>         (SID=xe)
>         (CID=
>             (PROGRAM=httpd@webserver_hostname)
>             (HOST=webserver_hostname)
>             (USER=apache)
>         )
>     ))
>
> So my question is: why is Django attempting to connect to the Oracle
> database with different credentials than the ones I specified? Notably, it
> is attempting to use user 'apache' instead of 'MY_USER_NAME'. The database
> host IP, port, and SID are correct and what I specified. It just appears to
> be the user name that is different.
>
> (As a side note, I suppose the password is transmitted separately in a
> later portion of the log in process?)

My understanding is that the USER in there isn't the username for
Oracle login, it is the username of current user running the program.
If everything else is OK but you have wrong username you should get an
error telling you that username or password was incorrect.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Global filtering of the querystes

[akaariai]

On 22 tammi, 08:32, Jani Tiainen  wrote:
> Hi,
>
> I've several models that share same attribute and form doing ajax
> queries to/from form in a browser window.
>
> For example owner:
>
> class MyClass(models.Model):
>      title = models.TextField(max_length=100)
>      owner = models.TextField(max_length=100)
>
> class MyOtherClass(models.Model):
>      title = models.TextField(max_length=100)
>      value = models.IntegerField()
>      owner = models.TextField(max_length=100)
>
> And so on.
>
> Now I would like to user have option in browser to filter any query
> based on model that has "owner" property to be filtered by owner.
>
> What is the best way to doit?
>
> I only could come up with threadlocals and special manager that can be
> fed with global filtering rules.

I think your solution is fine (even if it is uqly).

You could also try to avoid threadlocals and pass the data in
parameters. Sill, if you want you can likely play with DB views and do
the filtering in there using some session parameters and views. Other
than that I can't think of any obvious way to achieve what you want.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Correct way to specifiy database-level column defaults

[akaariai]

On 10 tammi, 16:04, john.wolt...@tpiengineering.com wrote:
> What is the best way to specify a database level default value with
> Django?  If I want a timestamp column to default to the SQL function now(),
> is there an accepted way to make it happen?  Or to default a boolean column
> to True?  I notice that when I call *manage.py sqlall* I don't see any *
> DEFAULT* values specified in the generated queries for Postgres.

Support for this would be an useful addition. Personally I have need
for at least now() and txid_current(), but preferably we want any
default clause (maybe even any per column SQL, so that you could do
CHECK too).

There are three problems for this feature:
  1. How to get the SQL into the CREATE TABLE clauses?
  2. How to return the values back after insert?
  3. If the way for no.1 is hand edited SQL, how to support this in
testing?

Above, no.2 seems the hardest problem. Some databases support
RETURNING, but not all.

Currently, you can do DB defaults by hand editing the sqlall output +
using post_save signal() and fetch the DB generated values into the
saved instance in the signal. But, you will need some way to solve no.
3 in this case. South might be a good option here. Personally I use a
custom testing system where I load the database schema from production
(with some data, too), apply migrations (that is, load custom SQL) and
then run tests.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ORM, Oracle and UTF-8 encoding problem.

[akaariai]

I created the following test case into django's test suite modeltests/
basic/tests.py:
def test_unicode(self):
# Note: from __future__ import unicode_literals is in
effect...
a = Article.objects.create(headline='0
\u0442\u0435\u0441\u0442 test', pub_date=datetime.n  ow())
self.assertEqual(Article.objects.get(pk=a.pk).headline, '0
\u0442\u0435\u0441\u0442 test'   )

This does pass on Oracle when using Django's master branch, both with
Python 2.7 and 3.3.

Django's backend is doing all sorts of trickery behind the scenes to
get correct unicode handling. I am not sure where the problem is. What
Django version are you using?

 - Anssi

On 8 tammi, 17:34, Jani Tiainen  wrote:
> Hi,
>
> I've been trying to save UTF-8 characters to oracle database without
> success.
>
> I've verified that database is indeed UTF-8 capable.
>
> I can insert UTF-8 characters directly using cx_Oracle.
>
> But when I use ORM it will trash characters.
>
> Model I use:
>
> class MyTest(models.Model):
>      txt = CharField(max_length=128)
>
> s = u'0 \u0442\u0435\u0441\u0442 test'
>
> i = MyTest()
> i.txt = s
> i.save()
>
> i2 = MyTest.objects.get(id=i.id)
> print i2.txt
>
> u'0 \xbf\xbf\xbf\xbf test'
>
> So what happens here? It looks like Django trashes my unicode string at
> some (unknown point).
>
> Additional note:
>
> If I use cursor() from Django connection object strings get broken also.
> So it must be django Oracle backend doing something evil for me.
>
> --
> Jani Tiainen
>
> - Well planned is half done and a half done has been sufficient before...

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Get objects sorted by time of last comment

[akaariai]

On 5 tammi, 06:41, Vibhu Rishi  wrote:
> Hi Anssi,
>
> Thanks for the comment.
>
> I was thinking that if I have to do the last comment hackery for all the
> objects would it also work if I
> * add a last_comment_time field to the models I use with comments.
> * modify the comment form ( not sure which file to modify, still thinking
> it through) to add comment timestamp to the post model data.
>
> then the query could become much simpler in the sense that I only need to
> get post objects, and sort by last_comment_time field.

This is likely the best approach for you. If you have an index on the
last_comment_time field then db queries will be really fast. This is
not the case for the view approach. In addition the view approach is
complex and somewhat hard to maintain while the last_comment_time
field is fairly straightforward to implement.

I wouldn't do this on the form level. Instead I would do this in
comment.save() (or post_save signal), or alternatively with a
dedicated method for comment saving. If every comment is saved through
that method then you know the last_comment_time will be correctly
maintained. Of course, there are a lot of ways to do this, and they
have different tradeoffs. Experiment and see what fits your purposes.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Get objects sorted by time of last comment

[akaariai]

On 2 tammi, 08:50, Vibhu Rishi  wrote:
> Hi All,
>
> A very happy new year to you all !
>
> I am working on a website I am making as my hobby project. It is to do with
> motorcycle touring.
>
> I have done some initial work on it, and incrementally making changes as
> and when I can.
>
> I am trying to figure out the following issue :
> 1. I have a forum object where people can start threads.
> 2. the forum object uses the django comments module along with mptt. So far
> so good.
> 3. Now, I want to show the "latest commented on" posts. But I am not able
> to figure it out.
>
> For reference :http://bikenomads.herokuapp.com/
>
> On the box on the right, I want to show the posts based on the last comment
> time. However, all I can do right now is show the last post based on
> creation time (this is a field for the post object). I am not able to
> figure out how to sort based on comment time.
>
> Solutions :
> 1. Ideally there should be a way to sort object by comment time using the
> inbuilt comments module in django. Is this possible ?
> 2. Alternatively, I will need to update the post model to have another
> field for 'last_comment_time' and when someone posts a comment, I will need
> to update this field. I would rather not do this as I will need to make
> sure all the objects using comments will need to have this exact field.
>
> What would you suggest ?

The ORM doesn't offer a way to do the query you want. The query will
be something like this:
select * from post
  left join comment on comment.object_id = post.id and
comment.object_type = 'post'
and comment.timestamp = (select max(timestamp) from
comment
  where object_id = post.id and
comment.object_type = 'post'
 )
 order by comment.timestamp;
(Assuming unique comment timestamp per post).

There are two underlying problems. First, Django's ORM doesn't
currently offer any way to generate more than single column equality
comparison in the JOIN clause. We are working on removing this
limitation from 1.6 (actually, the underlying limitation is already
somewhat gone). Second, the SQL needed isn't exactly nice. There are
multiple ways to write the same SQL, and how they perform differ
somewhat depending on the used DB and the amount of data.

So, what can you do pre 1.6? One way is to use views and some hackery
to do what you want.

class LatestComment(models.Model):
post = models.OneToOneField(Post, primary_key=True,
on_delete=models.DO_NOTHING, related_name='latest_comment')
{{ duplicate the columns in comment model here - you don't need
content type id }}

class Meta:
managed = False
db_table = 'latest_post_comment_view'

Then, create a view like this in the DB:

create or replace view "latest_post_comment_view" as (
select object_id as post_id, ...
  from comment
 where object_type = 'post'
 group by post_id, ...
having max(timestamp) = timestamp
);

The SQL above is untested. In any case, you should now be able to do:
 
Post.objects.select_related('latest_comment').order_by('latest_comment__timestamp')

You will need to repeat the above for all the models with comments

Managing the raw SQL needed for the views can be somewhat ugly. The
last_comment_time field might be easier to implement & maintain. That
being said I have been using the above technique successfully in
production systems.

I do wish Django will one day have latest_related() functionality. I
find I need that often, and as the above shows this isn't easily
doable currently.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Bulk delete - performance / object collection

[akaariai]

A note about the upcoming 1.5: The delete() was optimized so that it
doesn't fetch the objects if it doesn't have to. The "doesn't have to"
translates to:
 - no delete signals defined for the model
 - no cascades

That being said using DSE might be the right choice for you.

 - Anssi

On 2 tammi, 14:17, George Lund  wrote:
> Thank you very much for this. I'll catch up with those threads and read
> more about DSE, which looks really interesting work.
>
> regards
>
> George
>
> On Wednesday, 2 January 2013 11:50:54 UTC, Cal Leeming [Simplicity Media
>
>
>
>
>
>
>
> Ltd] wrote:
>
> > Hi George,
>
> > This is one area I spent quite a lot of time in personally, see;
>
> >https://groups.google.com/forum/?fromgroups=#!msg/django-users/iRhWD0...
> >https://groups.google.com/forum/#!topic/django-users/hgLrwMoFLII
>
> > Bulk operations using the ORM isn't always the right thing to do - and it
> > entirely depends on what you consider bulk and acceptable performance.
>
> > You might want to look at the source code for this, to see how they handle
> > bulk operations (they implemented the same bulk_update approach mentioned
> > in the above threads)
> >http://pypi.python.org/pypi/dse
>
> > Although bypassing the ORM might feel wrong at first, sometimes it is
> > completely acceptable - you just need to make sure you don't abuse/misuse
> > it unnecessarily.
>
> > Cal
>
> > On Wed, Jan 2, 2013 at 11:29 AM, George Lund 
> > > wrote:
>
> >> I'm trying to bulk-delete several million rows from my database.
>
> >> The docs for Django 
> >> 1.3say
> >>  "this will, whenever possible, be executed purely in SQL". A pure-SQL
> >> delete is what I want in this case, so that's fine.
>
> >> However, the code is never getting as far as running any SQL.
>
> >> Interrupting the script shows that the delete method on the QuerySet is
> >> trying to use a "Collector" to construct model instances for each row I'm
> >> trying to delete. This is going to take too long (and may in fact consume
> >> all the memory available) -- I don't think it's practical to wait in this
> >> case. (I've tried waiting over half an hour!)
>
> >> (I'm looking at django.db.models.query.QuerySet.delete and 
> >> django.db.models.deletion.Collector.collect
> >> / Collector.add.)
>
> >> What's the point in doing the delete "purely in SQL" if all of the
> >> objects are getting constructed anyway? Why do they need to be "collected"
> >> before the SQL DELETE is run? The model instance in this case has no
> >> child rows, to which the delete might need to be cascaded.
>
> >> Meanwhile I can construct the SQL by hand easily enough, but I feel this
> >> isn't doing things the right way.
>
> >> thanks for any help
>
> >> George
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Django users" group.
> >> To view this discussion on the web visit
> >>https://groups.google.com/d/msg/django-users/-/W4LqKzcnlaYJ.
> >> To post to this group, send email to 
> >> django...@googlegroups.com
> >> .
> >> To unsubscribe from this group, send email to
> >> django-users...@googlegroups.com .
> >> For more options, visit this group at
> >>http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: The method all() in overrided models.Manager doesn't works as expected

[akaariai]

Interesting approach :)

Some pointers:
  - The Page object has an automatic primary key. You should probably
change the path to be the primary key.
  - .all() should not return a list of objects, instead it should
return a QuerySet which returns a list of objects when iterated.

You might be better of using just a regular FileField. Store the
objects in the DB, remove the path field and store the content in a
FileField. That is the way this is supposed to be done in Django. You
can likely make your approach work but it wont be easy.

 - Anssi

On 31 joulu, 04:52, Shuge Lee  wrote:
> I'm writing a simple CMS, it stores page file data on local file system
> instead of SQL server.
>
> I do overrides the model Page's Manager, but it not works as expected,
> problems
>
>    - PageManager.all(it calls PageQuerySet?) doesn't works
>    - I can create Page object in Admin, but can't view it or delete it
>
> Source code is herehttps://github.com/shuge/mcms/blob/master/pages/models.py
>
> I have read Django Manager 
> docs a
> couple of times, but it doesn't help.
>
> Thanks for reply.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: memory leak? Am I taking crazy pills?

[akaariai]

On 22 marras, 08:27, Dan Ancona  wrote:
> python 2.7.3, django 1.4.2
>
> Our app (a fairly simple tastypie API that runs some machine learning foo on 
> mongodb records) is deployed to heroku and is leaking memory at a pretty good 
> clip. Hitting heroku memory errors after about an hour or two, under 
> significant load.
>
> Wandered through some various profiling options and eventually started to get 
> suspicious of tastypie. So I tried ripping that out. And then everything 
> else, until I finally got down to a django view that has nothing but this in 
> it:
>
> def stack(request):
>    gc.set_debug(gc.DEBUG_LEAK)
>    print "Uncollectable garbage", gc.garbage
>    return HttpResponse('hi')

If you set gc.DEBUG_LEAK then the GC will not free anything, instead
it will append garbage to gc.garbage. So, getting everything into
gc.garbage is expected in that case.

You will want to remove the gc.set_debug() call.

Here is an example without the call:

import gc
from django.http import HttpResponse

# Objects which have __del__ defined and are part of reference cycle
will be uncollectable.
class A(object):
def __del__(self):
pass
class B(object):
pass

def stack(request):
a = A()
b = B()
# Lets create a cycle
a.b = b
b.a = a
print(gc.garbage)
return HttpResponse('hi')


Load the view a couple of times and you will see garbage accumulating.
It will contain only the A objects - they are uncollectable because of
the __del__ method.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: IntegrityError at /add/ PRIMARY KEY must be unique

[akaariai]

.create() creates the object into DB, then .save(force_insert=True)
tries to insert the same object with the same ID again -> integrity
error. Seems like you don't need the second save at all.

 - Anssi

On 27 syys, 15:39, Amyth Arora  wrote:
> Not quite sure, but it seems like the save() method is being called twice
> so it is trying to create insert an entry with a primary key that already
> exists.
>
>
>
>
>
>
>
>
>
> On Thu, Sep 27, 2012 at 2:23 PM, enemybass  wrote:
> >  http://wklej.org/id/837631/- models, traceback, urls, views, database
> > structure
>
> > After adding data to database I have error (data is added correctly but
> > displays an error):
>
> > Exception Type: IntegrityError at /add/Exception Value: PRIMARY KEY must be 
> > unique
>
> > def add(request):
> >    if request.user.is_authenticated():
> >            user = request.user
> >                    b = Product.objects.get(id=3)
> >                    e = b.basket_set.create(name='basket_name', owner=user)
> >            e.save(force_insert=True)
>
> >    return render_to_response('basket.html')
>
> > How to fix it?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msg/django-users/-/mdIBvzeje1gJ.
> > To post to this group, send email to django-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > django-users+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/django-users?hl=en.
>
> --
> Thanks & Regards
> 
>
> Amyth [Admin - Techstricks]
> Email - aroras.offic...@gmail.com, ad...@techstricks.com
> Twitter - @mytharorahttp://techstricks.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: database error and connection._rollback

[akaariai]

On 23 elo, 08:30, Mike Dewhirst  wrote:
> This ... [1]
>
> from django.db import connection
> connection._rollback()
>
> ... solved a problem for me in unit testing when I catch a deliberate
> DatabaseError.
>
> Django 1.4
> Python 2.7
> PostgreSQL 9.1
>
> My question: Is it safe enough to use a method with a leading underscore?

Generally no. In this specific case you might be safe. However, if you
use _commit() in Django's standard TestCase you might make your
testing DB dirty and some other test could break because of that. This
kind of bug is horrible to debug.

> Is there a better method?

Use TransactionTestCase which is meant for this kind of testing. In
TransactionTestCase you can safely use .commit() and .rollback(). The
cost is slower test execution. See: [https://docs.djangoproject.com/en/
dev/topics/testing/#django.test.TransactionTestCase].

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: From Django 1.1 to Django 1.3.1 - Goes Very Slow

[akaariai]

On 21 elo, 19:04, ydjango  wrote:
> Thank you Paul. I will use those tools.
>
> I also plan to replace to some ORM queries by raw sql to avoid performance
> impact of cloning and multi db enhancements.

If you can, please try to pinpoint the reason for regressions, and if
it seems to be in Django's ORM, then post to django-developers or
create a ticket in Django's trac. If there is some major regressions
in the ORM it would be very valuable to know about them.

Some possible explanations:
  - The DB configuration isn't exactly as it was before.
  - The ORM generates different query strings leading to slow query
plans.
  - Using F() expressions and long filter chains can lead to slow
queryset .clone() speed (see https://code.djangoproject.com/ticket/16759)

You can try to pinpoint these by:
  - Checking if the generated SQL is the same between 1.1 and 1.3 by
using qs.query.as_str() in 1.1 and str(qs.query) in 1.3.
  - Check if the generation of the queryset takes too much time.
Easiest way is to generate a dummy script which does something like
this:
from datetime import datetime
start = datetime.now()
for i in range(0, 100):
# Do not execute the query, just construct the queryset!
qs = SomeModel.objects.filter(cond1).filter(cond2)... # Replace
with problematic queryset from production.
print datetime.now() - start

And see if there are any meaningful differences.

If the answer to the above is "no meaningful differences, the same SQL
generated" then it is reasonable to suspect the DB configuration.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Calculations in Queries

[akaariai]

On 16 elo, 03:51, Callum Bonnyman  wrote:
> I've tried a few things but i cannot get the correct date formats to
> calculate the difference, so far i have this:
>
>     popular_posts = Post.objects.extra(select={
>     'popularity': '(' + datetime.datetime.now() + ' - created) * views'
>     })
>
> It shows an error, i've tried googling it but not had much luck so far

It seems you have two errors here:
  1. you should not use datetime.datetime.now() as part of the
querystring, use select_params (see the docs) to pass the time to the
query.
  2. the "timedelta" * views calculation might need some casting. It
might be your DB isn't happy about doing multiplication of an integer
and an interval.

You might want to store the popularity into the model itself. The
problem with your approach is that if you have a lot of posts, you
can't use indexing for the popularity. So, to fetch the top 50 popular
posts, the DB will need to sort all the posts. This will be slow if
you have non-trivial amount of posts.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: union of two QuerySets

[akaariai]

On 1 elo, 19:15, Tim Chase  wrote:
> On 08/01/12 10:28, lex P rez wrote:
>
> > Hi,
>
> > it's better  Person.objects.filter(models.Q(first_**name__startswith='mic'),
> > models.Q(first_**name__startswith='joh'))
> > (only one query...)
>
> I'm pretty sure this will get you the intersection (it uses AND)
> rather than the union (which would be using OR).  So I think you want
>
>  from django.db.models import Q
>  Person.objects.filter(
>    Q(first_name__startswith="mic") |
>    Q(first_name__startswith="joh")
>    )
>
> using the "|" (OR) operator to join the two Q objects.
>
> -tkc
>
> For reference, you can read at
>
> https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db...
>
> https://docs.djangoproject.com/en/dev/topics/db/queries/#complex-look...

As stated earlier the correct way to combine two existing querysets is
to do: combined = qs1 | qs2.

There are some limitations to what kind of querysets will work, but
for plain querysets which have just filers applied things should just
work.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: MySQL total overheat (somewhat complex database)

[akaariai]

On 28 heinä, 21:44, lubos  wrote:
> Hello,
>
> I have a quite sophisticated database with frequently interconnected tables
> and on the top level table, Django produces queries like this:
>
> SELECT `data_schedule`.`id`, `data_schedule`.`process_id`,
> `data_schedule`.`hardware_id`, `data_schedule`.`task_id`,
> `data_schedule`.`start`, `data_schedule`.`times`, `data_schedule`.`done`,
> `data_process`.`id`, `data_process`.`string_id`,
> `data_process`.`operation_id`, `data_string`.`id`,
> `data_string`.`immutable`, `data_operation`.`id`,
> `data_operation`.`string_id`, T5.`id`, T5.`immutable`,
> `data_hardware`.`id`, `data_hardware`.`string_id`,
> `data_hardware`.`workspace_id`, `data_hardware`.`warmup`, T7.`id`,
> T7.`immutable`, `data_workspace`.`id`, `data_workspace`.`string_id`,
> T9.`id`, T9.`immutable`, `data_task`.`id`, `data_task`.`request_id`,
> `data_task`.`thread_id`, `data_task`.`amount`, `data_request`.`id`,
> `data_request`.`order_id`, `data_request`.`item_id`,
> `data_request`.`amount`, `data_order`.`id`, `data_order`.`string_id`,
> `data_order`.`subject_id`, `data_order`.`type`, `data_order`.`shipped`,
> `data_order`.`paid`, `data_order`.`end`, T13.`id`, T13.`immutable`,
> `data_subject`.`id`, `data_subject`.`string_id`, T15.`id`, T15.`immutable`,
> `data_item`.`id`, `data_item`.`string_id`, `data_item`.`measure_id`,
> `data_item`.`amount`, T17.`id`, T17.`immutable`, `data_measure`.`id`,
> `data_measure`.`string_id`, T19.`id`, T19.`immutable`, `data_thread`.`id`,
> `data_thread`.`string_id`, T21.`id`, T21.`immutable` FROM `data_schedule`
> INNER JOIN `data_process` ON (`data_schedule`.`process_id` =
> `data_process`.`id`) INNER JOIN `data_string` ON
> (`data_process`.`string_id` = `data_string`.`id`) INNER JOIN
> `data_operation` ON (`data_process`.`operation_id` = `data_operation`.`id`)
> INNER JOIN `data_string` T5 ON (`data_operation`.`string_id` = T5.`id`)
> INNER JOIN `data_hardware` ON (`data_schedule`.`hardware_id` =
> `data_hardware`.`id`) INNER JOIN `data_string` T7 ON
> (`data_hardware`.`string_id` = T7.`id`) INNER JOIN `data_workspace` ON
> (`data_hardware`.`workspace_id` = `data_workspace`.`id`) INNER JOIN
> `data_string` T9 ON (`data_workspace`.`string_id` = T9.`id`) INNER JOIN
> `data_task` ON (`data_schedule`.`task_id` = `data_task`.`id`) INNER JOIN
> `data_request` ON (`data_task`.`request_id` = `data_request`.`id`) INNER
> JOIN `data_order` ON (`data_request`.`order_id` = `data_order`.`id`) INNER
> JOIN `data_string` T13 ON (`data_order`.`string_id` = T13.`id`) INNER JOIN
> `data_subject` ON (`data_order`.`subject_id` = `data_subject`.`id`) INNER
> JOIN `data_string` T15 ON (`data_subject`.`string_id` = T15.`id`) INNER
> JOIN `data_item` ON (`data_request`.`item_id` = `data_item`.`id`) INNER
> JOIN `data_string` T17 ON (`data_item`.`string_id` = T17.`id`) INNER JOIN
> `data_measure` ON (`data_item`.`measure_id` = `data_measure`.`id`) INNER
> JOIN `data_string` T19 ON (`data_measure`.`string_id` = T19.`id`) INNER
> JOIN `data_thread` ON (`data_task`.`thread_id` = `data_thread`.`id`) INNER
> JOIN `data_string` T21 ON (`data_thread`.`string_id` = T21.`id`) ORDER BY
> `data_schedule`.`id` ASC
>
> MySQL, however, isn't able to process it and after few minutes it holds all
> processor performance.
>
> I would be glad for any idea.

Are you using select_related()? To me it seems you are just doing too
many joins in one query, and MySQL just can't handle it. If
select_related is the source of this problem, check if using
prefetch_related instead helps.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Is this possible using the Django ORM without raw SQL?

[akaariai]

On 28 heinä, 15:39, sbrandt  wrote:
> > On 28 heinä, 12:20, sbrandt  wrote:
> > > Hello,
>
> > > for the first time ever how to compile a query with the Django ORM :/
>
> > > There are two Models: The second model has a foreign key to the first
> > model
> > > and a type, where the combination of "first model and second model's
> > type"
> > > is unique. Let's say type 1 are dogs, type 2 are cats and every model1
> > can
> > > only have 0 or 1 dogs and cats. I know that it could have been
> > implemented
> > > as two seperate tables without types and one-to-ones, but I need to
> > query
> > > the whole model2 often and it's implemented in this way since years.
>
> > > This is not a problem for single model1 objects, since it's just a
> > second
> > > objects.get(...)-call as a lazy property in the model1.
>
> > > But now I need a query for *all* model1-objects with two additional
> > > columns: Has this model1 a cat, has this model1 a dog?
>
> > > Here's some code (Just pseudocode - doesn't work! My actual models would
> > be
> > > too complicated):
>
> > > class Model1(Model):
> > >     name = CharField(...)
>
> > > m2types = (cats, dogs)
>
> > > class Model2(Model)
> > >     m1 = ForeignKey(Model1)
> > >     type = SmallIntegerField(..., choices=m2types)
> > >     class Meta:
> > >         unique_together = ('m1', 'type')
>
> > > In SQL I would do it with left outer joins:
>
> > > SELECT m1.id, m1.name, c.id, d.id
> > > FROM model1 AS m1
> > > LEFT OUTER JOIN model2 AS c ON c.m1_id = m1.id AND c.type = 1
> > > LEFT OUTER JOIN model2 AS d ON d.m1_id = m1.id AND d.type = 2;
>
> > > So, back to my questsion: Is this possible with the Django ORM without
> > > using raw SQL?
>
> > > Note: Speed is not important. Since it is a cronjob being done half a
> > year
> > > and my models are just hundrets, iterating over every model1 and using
> > the
> > > lazy property cat and dog would be okay, and also using raw SQL would be
> > > okay since I'm tied to PostgreSQL. I'm explicitly searching for an
> > elegant
> > > solution with the Django ORM.
>
> > Hmmh, so you want to fetch every object, and "annotate" the
> > information about having a dog or a cat in the original model. I don't
> > think that can be done, although there might be some trick for this.
>
> > What you could do is use the prefetch_related() method, and then do
> > the annotation in Python code.
>
> > Something like this:
> >     objs = Model1.objects.prefetch_related('model2_set')
>
> > And in model1 you could have two properties, has_cat and has_dog
>
> > class Model1:
> >     ...
> >     def _has_dog(self):
> >         return any(obj for obj in self.model2_set if obj.type == DOG)
> >     has_dog = property(_has_dog)
> >     ...
>
> > While this isn't elegant it gets the work done... If you need to
> > filter or do something else in the ORM with the has_dog information,
> > you will not be able to do this using the above idea.
>
> > Django's ORM can be somewhat hard to use when working with reverse
> > foreign key data. There is room to improve the annotation mechanism of
> > Django. I hope the situation will improve... The prefetch_related
> > machinery has helped with many situations already.
>
> >  - Anssi
>
> Okay, so I'll take a closer look at prefetch_related. I didn't know one can
> prefetch the model_set.
>
> If I would have splitted up into one Dog and one Cat table using one-to-one
> to Model1, I could have just used select_related on the backwards relation.
> But I don't see any way to tell Django this is in fact a one-to-one
> relationship.

Well, it is not a one-to-one relationship, without the added filter on
type.

I really wish we had something like this in Django:
qs = Model1.objects.annotate(dog=ModelAnnotation('model2_set',
Q(model2_set__type='dog'))

This would do something similar to select_related - every object is
annotated with Model2 instances. You could do further operations to
the annotated model:
qs.order_by('dog__type')

The query generated would be something like this:

select ...
  from model1
   left join model2 on model2.model1_id = model1.id and model2.type =
'dog'
 order by model2.type;

I do think the above is possible to achieve, but there is some more
work to be done...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Is this possible using the Django ORM without raw SQL?

[akaariai]

On 28 heinä, 12:20, sbrandt  wrote:
> Hello,
>
> for the first time ever how to compile a query with the Django ORM :/
>
> There are two Models: The second model has a foreign key to the first model
> and a type, where the combination of "first model and second model's type"
> is unique. Let's say type 1 are dogs, type 2 are cats and every model1 can
> only have 0 or 1 dogs and cats. I know that it could have been implemented
> as two seperate tables without types and one-to-ones, but I need to query
> the whole model2 often and it's implemented in this way since years.
>
> This is not a problem for single model1 objects, since it's just a second
> objects.get(...)-call as a lazy property in the model1.
>
> But now I need a query for *all* model1-objects with two additional
> columns: Has this model1 a cat, has this model1 a dog?
>
> Here's some code (Just pseudocode - doesn't work! My actual models would be
> too complicated):
>
> class Model1(Model):
>     name = CharField(...)
>
> m2types = (cats, dogs)
>
> class Model2(Model)
>     m1 = ForeignKey(Model1)
>     type = SmallIntegerField(..., choices=m2types)
>     class Meta:
>         unique_together = ('m1', 'type')
>
> In SQL I would do it with left outer joins:
>
> SELECT m1.id, m1.name, c.id, d.id
> FROM model1 AS m1
> LEFT OUTER JOIN model2 AS c ON c.m1_id = m1.id AND c.type = 1
> LEFT OUTER JOIN model2 AS d ON d.m1_id = m1.id AND d.type = 2;
>
> So, back to my questsion: Is this possible with the Django ORM without
> using raw SQL?
>
> Note: Speed is not important. Since it is a cronjob being done half a year
> and my models are just hundrets, iterating over every model1 and using the
> lazy property cat and dog would be okay, and also using raw SQL would be
> okay since I'm tied to PostgreSQL. I'm explicitly searching for an elegant
> solution with the Django ORM.

Hmmh, so you want to fetch every object, and "annotate" the
information about having a dog or a cat in the original model. I don't
think that can be done, although there might be some trick for this.

What you could do is use the prefetch_related() method, and then do
the annotation in Python code.

Something like this:
objs = Model1.objects.prefetch_related('model2_set')

And in model1 you could have two properties, has_cat and has_dog

class Model1:
...
def _has_dog(self):
return any(obj for obj in self.model2_set if obj.type == DOG)
has_dog = property(_has_dog)
...

While this isn't elegant it gets the work done... If you need to
filter or do something else in the ORM with the has_dog information,
you will not be able to do this using the above idea.

Django's ORM can be somewhat hard to use when working with reverse
foreign key data. There is room to improve the annotation mechanism of
Django. I hope the situation will improve... The prefetch_related
machinery has helped with many situations already.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Prefetch related data when using raw()

[akaariai]

On 26 heinä, 19:12, Sencha  wrote:
> I want to prefetch related data (ideally through the 
> prefetch_related()method), however I need to use the
> raw() mapper method that will map a custom query (complicated table joins
> to filter my query properly) to my model. However when I try this I get:
>
> AttributeError: 'RawQuerySet' object has no attribute 'prefetch_related'
>
> Is there any way I can achieve the prefetching when using
> Model.objects.raw()?
>
> Many thanks!

There is no direct way. However, you could try this (completely
untested, and not part of public api):

from django.db.models.query import prefetch_related_objects
raw_qs = list(raw_qs) # this line maybe not needed...
prefetch_related_objects(raw_qs, ['a_related_lookup',
'another_related_lookup', ...])

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Excluding items from a queryset

[akaariai]

On 16 heinä, 14:38, Sithembewena Lloyd Dube  wrote:
> Hi everyone,
>
> I have a queryset of categories and would like to exclude categories with
> no items associated. I am cycling through the queryset as follows, without
> the desired result:
>
> for cat in categories:
>                 if cat.item_set.count() == 0:
>                     categories.exclude(pk=cat.pk)

Querysets are always cloned on calling a method, so the right syntax
is:
categories = categories.exclude(pk=cat.pk).
For efficiency reasons you might want to collect all the pks into a
list, and then after the loop do one:
categories = categories.exclude(pk__in=pk_list)

The queryset cloning can be expensive if you do a lot of operations.

Of course, the annotation way given downthread is likely to be the
most effective way to do this.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Ranking Queryset

[akaariai]

On 20 kesä, 20:37, Peter Ung  wrote:
> Thanks  Chris for your response; however, its not the solution I'm looking
> for. I need the ranking to go in sequential order - the solution you
> provided will just make my points, or scores in your example, the rank. I
> need, for example, the person with the highest number of scores/points with
> the rank 1 associated their record in the queryset. Then rank 2 for the
> next person with highest score/points. etc
>
> Any ideas on how to achieve that? Only solution I'm seeing at the moment is
> to add rank to the model and update rank every time a user calls the
> leaderboard - which would be expensive and slow wouldn't it?

Django's ORM doesn't support windowing functions. The problem isn't
adding the windowing function (rank() in this case) to the SELECT
clause. You can do this with .extra() currently. The real problem is
that to filter on the result, you need to push the query down into
inner subquery, then do the filter in the outer query. This isn't
supported.

So, it seems this is not doable without using raw SQL.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.4 memory issues

[akaariai]

On 12 kesä, 17:59, Greg Donald  wrote:
> Over the weekend I upgraded my project to Django 1.4.  After the
> upgrade my Apache/mod_wsgi setup began running out of memory.  I
> increased the memory capacity of my virtual servers, but that only
> extended the time until they began to swap again.  I found a blog
> article that suggested setting Apache's MaxRequestsPerChild to
> something other than 0.  After setting it to 25, Apache seems to be
> releasing memory again.
>
> My question is, why did I have to do that?  The only thing that
> changed is my Django version from 1.3 -> 1.4.  I have Debug=False in
> settings.py, so I don't think I'm experiencing the SQL memory buildup
> described on a couple of blogs.  I do have some long running scripts
> that use Django outside of Apache, so I implemented db.reset_queries()
> in those, but it had no noticeable effect.  They didn't appear to be
> using any more memory than normal, but I figured it couldn't hurt.
>
> So everything is running fine again, but something changed in Django
> 1.4 that causes my same project code to leak memory.  I'd sure like to
> figure it out.

I don't remember seeing memory leak issues reported for 1.3 -> 1.4
uprgades (which doesn't mean they do not exist).

It is very hard to say where the problem might be without further
details. I don't have much experience in memory leak debugging, but
this post for example looks promising:
http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks

 - Anssi


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How postgres data type "real" is represented in Django?

[akaariai]

On 12 kesä, 18:50, IgorS  wrote:
> Hi ALL!
>
> It seems that there is no class in Django models that represents 4-byte
> data type "real" in postgres. I looked into
> django\db\backends\postgresql_psycopg2\creation.py but could not find it
> there. The only choice seems to be FloatField, which represents "double
> precision" in postgres. But i would like to use "real" to minimize the
> index size? How can i do it?

It might be you will need to do a FloatField subclass - RealField,
which will return "REAL" as its database type from field.db_type(). It
has been a while since last dealing with custom fields so there might
be something more needed. Django's documentation should contain more
information.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How to perform a simple search ignoring characters with accents (á, é, í, ó, ú)

[akaariai]

On Jun 4, 4:20 pm, Peter of the Norse  wrote:
> One possibility is to use MySQL. By default it indexes things so that a, á, 
> and À are the same thing. There are some gotchas though: you have to make 
> sure that it’s using an appropriate character set for the languages you’re 
> using. (UTF-8 is a good choice.) There’s not a good similar solution for 
> PostgreSQL. While it is possible to write a function, and create an index on 
> that function, I haven’t found a way of searching on that index in Django. If 
> anyone knows of a way to do it, I’d love to here it.

.extra() will likely allow what you are looking for. But
using .extra() tends to get ugly fast.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Is this requirement feasible with django

[akaariai]

On Jun 3, 11:02 am, Kalpa Welivitigoda  wrote:
> Hi,
>
> I want to develop a web based laboratory test reporting system. It
> basically involves storing the records of different tests for
> different patient and get a print out of the reports. There are around
> 100 tests with different fields.
>
> The flow would be that the user enters a patient id and the tests for
> that person appears, then he selects one test and a input form appears
> to enter test results. This continues for the other tests as well and
> finally a print out is issued and a pdf of the report is generated on
> request.
>
> Since there are around 100 tests (with different fields), do I need to
> write separate models and views (to view the data input form) for each
> and every one of them or can it be made simpler at least generating
> views (because views are generated from some of the fields in the
> model) ?

There are a couple of ways to go forward I know of.

The simplest pattern is just to have all the tests as Models. This is
likely a road of a lot of pain to maintain, but it is definitely worth
trying if this works for you. Creating new Python models isn't that
complicated, and with South migrations are somewhat easy to do, too.

Another somewhat common pattern is to use EAV (entity-attribute-value)
style tables. The modeling would be something like:
class Patient:


class Test:


class PatientTest
FK(patient)
FK(test)

class PatientTestResult
FK(PatientTest)
attr_type
attr_value

So, for each patient - test combo you can store attributes of that
test. It is somewhat common to then model also metadata of what
attributes each test can potentially have, so that you can dynamically
construct the forms and/or create procedures which validate the data.
This can feel like a clever way forward, but my personal experience is
that this is actually going to be a pain to maintain. Another problem
is that most trivial queries easily becomes an unholy mess of joins +
subselects (or you need PIVOT queries, or something else like that).
For example I have a EAV system to maintain where querying pretty
basic information about an employee requires 50+ joins.

Consider carefully before going down the EAV road. Avoid storing any
validation logic in the database.

I would likely try to go with a format where you store the test
results in JSON/XML/HStore(if using PostgreSQL)/some other vendor
specific datatype. If you need to do calculations on the test results,
you can then transform the serialized format to (denormalized)
database tables for the calculations. If this transformation feels
like a lot of work, remember that if you have EAV you will still need
to do this, or write queries you really don't want to be writing. If
you have some common attributes (identifier code, date of test,
location of test, the person who did the test, etc) then model these
into SQL tables. Just the result fields should go into the serialized
format.

This is one area where there really aren't any right answers. You need
to prototype what works for your use case.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Oracle schema not working

[akaariai]

On Jun 1, 12:58 pm, Jani Tiainen  wrote:
> > Hello. The user connecting to Oracle is an ordinary user and needs to
> > prefix all tables with the schema name.
> > I've tried crafting Meta.db_table like so:
> >http://cd-docdb.fnal.gov/cgi-bin/RetrieveFile?docid=3156&version=1&fi...
>
> > But I get error
>
> > DatabaseError at /
>
> > schema "foo" does not exist
> > LINE 1: ...ty", "foo"."table_name"."address_country" FROM "foo"."...
>
> > I also tried wrapping request in a TransactionMiddleware and execute
> > this SQL before the fetching (modifying Meta accordingly):
> > MyModel.objects.raw('ALTER SESSION SET CURRENT_SCHEMA=foo')
>
> > Neither way helped. The user has the needed permissions.
>
> Since Oracle doesn't make a difference between user and schema (they're
> equivalents)
>
> Simplest thing is to create (private/public) synonyms for tables for
> user in question. Otherwise you need to prefix with schema name.
>
> See also tickethttps://code.djangoproject.com/ticket/6148

I have updated the patch in that ticket to current master. It should
now work for all core backends, though GIS is still unsupported.
Please test.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Struggling with formsets

[akaariai]

On May 29, 12:45 pm, David 
wrote:
> Anyone else have any suggestions please?
>
> Thank you

I don't think there is a way to create formsets, and then formsets
related to the first formsets. So, something like this is what you
likely need:

for p in persons:
do_what_you_did_for_single_object() # remember to prefix the
forms.

I would likely do a list persons + their attendances view with edit
link/button for each person. That is one approach worth considering,
as it is pretty easy to do.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Set additional conditions for JOIN

[akaariai]

On May 28, 10:59 pm, Boris Shemigon  wrote:
> Hi,
>
> I wonder is there a way to generate a query with joins with more than one
> condition? For example:
>
> SELECT o.id FROM object
> JOIN object_value AS ov1 ON ov1.o_id=object.id AND ov1.field_id=1 AND
> ov1.value=2
> JOIN object_value AS ov2 ON ov2.o_id=object.id AND ov2.field_id=2 AND
> ov2.value>=1
> JOIN object_value AS ov3 ON ov3.o_id=object.id AND ov3.field_id=3 AND
> ov3.value>=2
>
> It's supposed to find all objects that have all fields with corresponding
> values.
>
> Does anybody know how to code that in the frame of Django ORM. Or the only
> solution is to go down to SQL?

There is currently no way to add conditions to the join clause, not
even by using internal APIs.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Multiprocess Queryset memory consumption increase (DEBUG=False and using iterator)

[akaariai]

On May 28, 5:42 pm, pc  wrote:
> I am stumped. I am trying process a lot of data (2 million records and
> up) and once I have a QuerySet, I immediately feed it to a
> queryset_iterator that fetched results in chunks of 1000 rows each. I
> use MySQL and the DB server is on another machine (so I don't think it
> is MySQL caching).
>
> I kick off 4 sub-processes using multiprocessing.Process, but even if
> I keep it to 1, eventually, I will run out of memory.
>
> RAM usage just steadily seems to increase. When I finish processing a
> resultset, I would allocate a new resultset to the variable and I
> would expect gc to get my memory back.
>
> Any ideas?
>
> def run(self):
>      .
>
> messages=queryset_iterator(MessageDAO.get_all_messages_for_date(now))
>             self.process_messages(job,messages)
>             messages=None
>             gc.collect()
> ...
>
> def queryset_iterator(queryset, chunksize=1000):
>     pk = 0
>     last_pk = queryset.order_by('-pk')[0].pk
>     queryset = queryset.order_by('pk')
>     while pk < last_pk:
>         for row in queryset.filter(pk__gt=pk)[:chunksize]:
>             pk = row.pk
>             yield row
>         gc.collect()

Are you sure the leak is not in process_messages?

I tested something similar on PostgreSQL, and the queryset_iterator
doesn't seem to leak memory:
def queryset_iterator(queryset, chunksize=100):
pk = 0
last_pk = queryset.order_by('-pk')[0].pk
queryset = queryset.order_by('pk')
while pk < last_pk:
print len(connection.queries)
for row in queryset.filter(pk__gt=pk)[:chunksize]:
pk = row.pk
yield row
gc.collect()

for i in queryset_iterator(TestModel.objects.all()):
print memory()

where memory() is from 
http://stackoverflow.com/questions/938733/python-total-memory-used

The result seems stable, and don't indicate any memory leak.

TestModel contains 10 objects.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Why this database error in shell? (Query looks valid to me.)

[akaariai]

On May 26, 10:26 pm, Chris Seberino  wrote:
> I have Homework objects that are related to Course objects for a
> school related Django app.  I can access Course and Homework objects
> but there is one Homework query at the end of this email that bombs
> and I don't know why.
>
> ===
> First I'll show a few working queries to show that everything is sane
> and working
> ===
>
> >>> Course.objects.filter(subject="Algebra 1", year=2011, session="Fall")
>
> []
>
> >>> Course.objects.filter(year=2011)
>
> [, ,  object>, , ,  Course object>, , ,
> , ]
>
> >>> course = Course.objects.filter(year=2011)[0]
> >>> Homework.objects.filter(course=course)
>
> [, , ... etc.]
> (truncated by me)
>
> >>> Homework.objects.filter(course__session="Fall")
>
> [, , ... etc.]
> (truncated by me)
>
> 
> Here is the mystery one that bombs and I don't know why:
> 
>
> >>> Homework.objects.filter(course__year=2011)
>
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/lib/pymodules/python2.6/django/db/models/manager.py",
> line 141, in filter
>     return self.get_query_set().filter(*args, **kwargs)
>   File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line
> 550, in filter
>     return self._filter_or_exclude(False, *args, **kwargs)
>   File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line
> 568, in _filter_or_exclude
>     clone.query.add_q(Q(*args, **kwargs))
>   File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py",
> line 1128, in add_q
>     can_reuse=used_aliases)
>   File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py",
> line 1071, in add_filter
>     connector)
>   File "/usr/lib/pymodules/python2.6/django/db/models/sql/where.py",
> line 66, in add
>     value = obj.prepare(lookup_type, value)
>   File "/usr/lib/pymodules/python2.6/django/db/models/sql/where.py",
> line 299, in prepare
>     return self.field.get_prep_lookup(lookup_type, value)
>   File "/usr/lib/pymodules/python2.6/django/db/models/fields/
> related.py", line 139, in get_prep_lookup
>     raise TypeError("Related Field has invalid lookup: %s" %
> lookup_type)
> TypeError: Related Field has invalid lookup: year

Probably what happens here is that the 'year' part of the course__year
lookup is mistaken for a date lookup 'year' (you could do a query like
datetimefield__year=2011).

What version of Django are you using? 1.4 should contain a fix for
this: https://code.djangoproject.com/ticket/11670. If you can't
upgrade, it might be possible that course__year__exact=2011 will work.
Not sure though.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ordering m2m query based on through table

[akaariai]

On May 23, 11:33 pm, akaariai  wrote:
> On May 23, 3:36 pm, akaariai  wrote:
>
> > I guess (again!) that the reason for the LEFT JOIN is this call in sql/
> > compiler.py:_setup_joins()
>
> > self.query.promote_alias_chain(joins,
> >      self.query.alias_map[joins[0]].join_type == self.query.LOUTER)
>
> > This seems to promote the join to outer join too aggressively, the
> > promote_alias_chain call does not see that the join is constrained by
> > other conditions already. I don't know if this has any practical
> > effects, nor do I know how easy/hard this would be to fix. It could be
> > as easy as just starting from the first new join generated by the
> > _setup_joins() call, and leaving the pre-existing joins alone. If it
> > is that easy, it should be fixed...
>
> I have a fix for the issue mentioned above. I haven't figured out how
> to create a query where promotion from INNER JOIN to LEFT OUTER JOIN
> will cause wrong results. Fixing this is still worth it, as even if
> there isn't a query where the current promotion logic breaks the
> results, it is possible there one day will be a way to do that. There
> might be performance impacts at least for some databases.
>
> If somebody happens to figure out a query which produces wrong results
> due to the INNER -> OUTER promotion it would be good to add a test
> case for that.
>
> See this pull request for details:https://github.com/django/django/pull/90
>
>  - Anssi

Fixed in 
https://github.com/django/django/commit/8c72aa237918e31a525022f72b22cac75451af86

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ordering m2m query based on through table

[akaariai]

On May 23, 3:36 pm, akaariai  wrote:
> I guess (again!) that the reason for the LEFT JOIN is this call in sql/
> compiler.py:_setup_joins()
>
> self.query.promote_alias_chain(joins,
>      self.query.alias_map[joins[0]].join_type == self.query.LOUTER)
>
> This seems to promote the join to outer join too aggressively, the
> promote_alias_chain call does not see that the join is constrained by
> other conditions already. I don't know if this has any practical
> effects, nor do I know how easy/hard this would be to fix. It could be
> as easy as just starting from the first new join generated by the
> _setup_joins() call, and leaving the pre-existing joins alone. If it
> is that easy, it should be fixed...

I have a fix for the issue mentioned above. I haven't figured out how
to create a query where promotion from INNER JOIN to LEFT OUTER JOIN
will cause wrong results. Fixing this is still worth it, as even if
there isn't a query where the current promotion logic breaks the
results, it is possible there one day will be a way to do that. There
might be performance impacts at least for some databases.

If somebody happens to figure out a query which produces wrong results
due to the INNER -> OUTER promotion it would be good to add a test
case for that.

See this pull request for details: https://github.com/django/django/pull/90

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ordering m2m query based on through table

[akaariai]

On May 23, 2:36 pm, bruno desthuilliers
 wrote:
> > The interaction between
> > multiple references to same reverse foreign key relation in single
> > queryset is somewhat hard to remember. Could you post the generated
> > SQL?
>
> here you go - reformated for readability:
>
> SELECT
>   `blookcore_category`.`id`,
>   `blookcore_category`.`label`
> FROM
>   `blookcore_category`
>    LEFT OUTER JOIN `blookcore_classification`
>    ON (`blookcore_category`.`id` =
> `blookcore_classification`.`category_id`)
> WHERE
>    `blookcore_classification`.`blook_id` = 118
> ORDER BY
>    `blookcore_classification`.`position` ASC
>
> I don't quite get why it uses a left outer join instead of the inner
> join used when not adding the order by clause, but writing a working,
> usable ORM is not exactly a piece of cake so I won't complain about
> this ;). Anyway: the where clause still makes sure we only get the
> relevant categories.

Right you are about the query producing correct results. I should have
checked that myself.

I guess (again!) that the reason for the LEFT JOIN is this call in sql/
compiler.py:_setup_joins()

self.query.promote_alias_chain(joins,
 self.query.alias_map[joins[0]].join_type == self.query.LOUTER)

This seems to promote the join to outer join too aggressively, the
promote_alias_chain call does not see that the join is constrained by
other conditions already. I don't know if this has any practical
effects, nor do I know how easy/hard this would be to fix. It could be
as easy as just starting from the first new join generated by the
_setup_joins() call, and leaving the pre-existing joins alone. If it
is that easy, it should be fixed...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Following multiple reverse relationships

[akaariai]

On May 23, 3:06 pm, sk <6b656...@gmail.com> wrote:
> > On May 22, 12:51 pm, sk <6b656...@gmail.com> wrote:
>
> > > If I have a hierarchy of models such that:
>
> > > class A(models.Model):
> > >     name = CharField(max_length=10)
>
> > > class B(models.Model):
> > >     name = CharField(max_length=10)
> > >     a= ForeignKey(A)
>
> > > class C(models.Model):
> > >     name = CharField(max_length=10)
> > >     b= ForeignKey(B)
>
> > > If I have an instance of C, as returned by:
>
> > > c = C.objects.get(pk=1)
>
> > > How do I get the name of A?
>
> > > I realise I can go via B, but I am hoping to be able to follow the
> > > chain back to A from any arbitrary point in (a much longer) chain.
>
> > > For example:
> > > If this was repeated up to F in an identical manner.
> > > If I pass a function any instance from A to F I would want to return
> > > A.name.
>
> > > I am sure there must be a simple way to get the root data, but I am
> > > having trouble finding it.
>
> > You really do have to go through the relations through the chain -
> > there is no direct links from C to A so it is impossible to just hop
> > from C to A.
>
> > You can query the A instance by something like
> > A.objects.filter(pk=B.objects.filter(pk=c.b_id)) - this should return
> > the A instance in one query. (It might be you will need to use
> > pk__in).
>
> > Or, you could use select_related or prefetch_related to do the
> > fetching.
>
> >  - Anssi- Hide quoted text -
>
> > - Show quoted text -
>
> Fair enough, thank you for your reply.
>
> I is it posible to 'follow the chain' without knowing the exact
> instance you are in? for example can i get all foreign keys for a
> given instance, then follow them? To be honest this is just for
> interest now - I have implemented this with some hard-coding of how to
> decide where you are in the 'tree'.

You can check Model._meta.fields, Model._meta.get_field_by_name() etc.
Do a dir(ModelClass._meta) in pdb and you should see what methods/
attributes are available. The ._meta is part of the private API, so
there are no stability guarantees...

The ._meta is what Django's ORM uses when checking what lookups are
available, and which of them are foreign keys, reverse foreign keys
etc. So, it should contain all the necessary information for you.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ordering m2m query based on through table

[akaariai]

On May 23, 12:12 pm, bruno desthuilliers
 wrote:
> On May 22, 4:34 pm, akaariai  wrote:
>
> > So, you would want to do collection.songs.all() and have that ordered
> > by the sequence defined in the m2m table? I agree - this seems to be
> > somewhat hard currently.
>
> I currently have a similar scheme in one of our projects, works fine
> as far as I can tell:
>
> class Categorie(Model):
>    # 
>
> class Blook(Model):
>   # 
>   categories = models.ManyToManyField(
>         Category,
>         through='Classification',
>         related_name="blooks",
>         )
>
>   @property
>   def sorted_categories(self):
>      return self.categories.order_by("classifications__position")
>
> class Classification(Model):
>     category = models.ForeignKey(
>         Category,
>         related_name="classifications"
>         )
>
>     blook = models.ForeignKey(
>         Blook,
>         related_name="classifications"
>         )
>
>     position = models.PositiveIntegerField(
>         _(u"Position"),
>         )
>
>     class Meta:
>         unique_together = (
>             ("blook", "category"),
>             ("blook", "position"),
>             )

I would guess you would get duplicate objects returned. I think the
query works as follows:
  - Fetch all Blook's categories
  - For those categories you fetch all the classifications (not only
those which are related to Blook) and order by the position in there.
So, you could get the same category multiple times ordered by
positions not related to the blook <-> category connection.

Maybe the above code does work correctly. The interaction between
multiple references to same reverse foreign key relation in single
queryset is somewhat hard to remember. Could you post the generated
SQL?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: "Connection refused" with backend psycopg2

[akaariai]

On May 23, 11:46 am, Tanuva  wrote:
> If I connect manually using psql, I automatically get an ssl connection.
> Non-ssl isn't supposed to work in this case. Removing the sslmode from
> the django config doesn't change anything as I noted in the paste.
>
> > The error you are getting is the same you get if you would try to
> > connect to a machine with no postgres installation at all. So, one
> > possibility is that your host or port are not correct for some reason.
>
> Thats what puzzles me: if I use psycopg2 by hand
> (http://dpaste.org/Ozisz/), the connection works fine with the same
> credentials I supplied to Django. The table I created there appears in
> pgadmin.
>
> Marcel

Put a breakpoint in django/db/backends/postgresql_psycopg2/
base.py:_cursor() and see what the connect parameters are. Double
check pg_hba.conf and postgresql.conf and make sure they allow you to
connect from the webapp machine to the DB.

It is hard to help you more through a mailing list.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: "Connection refused" with backend psycopg2

[akaariai]

On May 23, 8:41 am, Tanuva  wrote:
> Am 22.05.2012 23:52, schrieb akaariai:
>
>
>
>
>
>
>
>
>
> > On May 22, 11:51 pm, Tanuva  wrote:
> >> Moin,
>
> >> lately, I wanted to switch my database from the testing sqlite file to a
> >> more productive pgsql 8.4 server. Now I'm constantly getting a
> >> "connection refused" from the psycopg2 backend:http://dpaste.org/0Ezka/
> >> Database configuration:http://dpaste.org/QPjD2/(notthat verbose...)
>
> >> Funny enough, connecting manually using psycopg2 in a python2 shell and
> >> using pgadmin to confirm changes done there works just fine, so there
> >> must be something in between.
>
> >> What information might help you help me helping myself? ;)
>
> > I would take a wild guess that this is related to pg_hba.conf. Is the
> > database on the localhost? If so, can you connect to it using 'psql -d
> > thedb', but not with 'psql -d thedb -h www0.com -U
> > the_user_in_settings'.
>
> >  - Anssi
>
> No, the db is not running on localhost, the machine sits in some
> datacentre. On that db server, your first command indeed doesn't work.
> Sadly that doesn't tell us much now I guess. The second one works fine
> both on the machine and here on my local box.

Maybe the psql -d thedb should not work. On the db server you should
probably add -U  in there.

Maybe the 'sslmode': 'require' is causing this problem? You can try if
this is causing problem by:
export SSLMODE=required
psql -U -h -p 5432 
or by just removing the sslmode from your database settings and trying
again.

The error you are getting is the same you get if you would try to
connect to a machine with no postgres installation at all. So, one
possibility is that your host or port are not correct for some reason.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: After python upgrade, User.objects.get(username= doesn't work consistently

[akaariai]

On May 22, 11:49 pm, Hanne Moa  wrote:
> I upgraded python to 2.6.8 today from an earlier 2.6, what an
> adventure! So much bizarre happenings so many places. Tonight's
> special:
>
> SomeModel.objects.get(sometextfield=sometext) always works (when there
> is one entry where sometextfield=sometext of course).
>
> auth.User.objects.get(username=sometext) works only *sometimes*.
>
> In a login-view, I first check that the user for a given username
> exists. It does! User found. Then I try to authenticate:
> auth.authenticate(user.username, password). That never works!
> Monkeypatching authenticate() shows that the
> User.objects.get(username=username) in it always returns
> User.DoesNotExist. But of course:
> User.objects.filter(username__startswith=username).get(username__endswith=username)
> always return the user in question. And the only thing that has
> changed is the minor python version.
>
> I can't get User.objects.get(username=whatever) to work in the shell either.
>
> How do I debug this further? This is for django 1.3, I need to have
> this working predictably again before I can upgrade to 1.4. I have
> logs of the sql if that'll help but the sql looks fine.

Please provide these three things for further debugging:
  - the exact Python code you try in the shell (including a real
username).
  - the SQL that generates (when not working).
  - what does happen when you try to run that SQL in manage.py dbshell
(you might need to add quotes to the parameters)

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: "Connection refused" with backend psycopg2

[akaariai]

On May 22, 11:51 pm, Tanuva  wrote:
> Moin,
>
> lately, I wanted to switch my database from the testing sqlite file to a
> more productive pgsql 8.4 server. Now I'm constantly getting a
> "connection refused" from the psycopg2 backend:http://dpaste.org/0Ezka/
> Database configuration:http://dpaste.org/QPjD2/(not that verbose...)
>
> Funny enough, connecting manually using psycopg2 in a python2 shell and
> using pgadmin to confirm changes done there works just fine, so there
> must be something in between.
>
> What information might help you help me helping myself? ;)

I would take a wild guess that this is related to pg_hba.conf. Is the
database on the localhost? If so, can you connect to it using 'psql -d
thedb', but not with 'psql -d thedb -h www0.com -U
the_user_in_settings'.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: DB joining 3 tables in django admin, while searching on the primary table

[akaariai]

On May 22, 5:10 pm, Derek  wrote:
> Please read 
> this:https://code.djangoproject.com/wiki/UsingTheMailingList#Donts:
> before continuing to post these types of messages.

In addition, doing a "django prefetch_related" search on Google might
lead to some clues here...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ordering m2m query based on through table

[akaariai]

On May 22, 7:14 am, KentH  wrote:
> I am trying to configure a m2m relationship to be ordered based on data in
> the through table. I assumed an 'order' attribute in the through-table's
> Meta class would be honored, but it's not. Bug #11850 (closed: wontfix)
> relates to this problem, but with no pointer as to a work-around.
>
> As custom ordering would seem a prime use case for a custom through class
> on m2m relations, I'm surprised how hard it is.
>
> In my case, I'm trying to order digital assets (eg sheet music) into a
> number of folios (collections). Music can appear in several collections.
> Order is manually specified for each collection. An attribute in a custom
> through table is a natural solution, but it doesn't work.
>
> Other examples which come to mind: Authors and books, or songs and
> composers. Since books are alphabetized by lead author, the order matters.
> And we wouldn't want Django to list "Lennon & McCartney" songs  as "McCartney
> & Lennon"
>
> Anyway, how is the best way to crack this nut? After studying
> `db.models.related` I tried building a custom manager on the related class,
> and override the `get_query_set` method to look for the through attribute
> on the manager to update the ordered_by clause. No luck -- but with
> prefetch interaction, this seems pretty fragile in any case.
>
> Thanks for any help.
>
> Kent.

So, you would want to do collection.songs.all() and have that ordered
by the sequence defined in the m2m table? I agree - this seems to be
somewhat hard currently. You would need to do [t.song for t in
Through.objects.filter(collection=collection).order_by('seq').select_related('song')]
or something like that... It might be nice if the related manager of
collection would have a special lookup 'through', so you could to
collection.songs.all().select_related('through'), or
collection.songs.all().order_by('through__seq'). But it would be
somewhat messy to do that currently. Implementing better custom
lookups could make this easier to implement.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Following multiple reverse relationships

[akaariai]

On May 22, 12:51 pm, sk <6b656...@gmail.com> wrote:
> If I have a hierarchy of models such that:
>
> class A(models.Model):
>     name = CharField(max_length=10)
>
> class B(models.Model):
>     name = CharField(max_length=10)
>     a= ForeignKey(A)
>
> class C(models.Model):
>     name = CharField(max_length=10)
>     b= ForeignKey(B)
>
> If I have an instance of C, as returned by:
>
> c = C.objects.get(pk=1)
>
> How do I get the name of A?
>
> I realise I can go via B, but I am hoping to be able to follow the
> chain back to A from any arbitrary point in (a much longer) chain.
>
> For example:
> If this was repeated up to F in an identical manner.
> If I pass a function any instance from A to F I would want to return
> A.name.
>
> I am sure there must be a simple way to get the root data, but I am
> having trouble finding it.

You really do have to go through the relations through the chain -
there is no direct links from C to A so it is impossible to just hop
from C to A.

You can query the A instance by something like
A.objects.filter(pk=B.objects.filter(pk=c.b_id)) - this should return
the A instance in one query. (It might be you will need to use
pk__in).

Or, you could use select_related or prefetch_related to do the
fetching.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What persists in memory across requests?

[akaariai]

On May 18, 7:54 pm, Ken J  wrote:
> I'm quite curious as to what persists in memory across requests in
> terms of django application variables.  I'm currently running a django
> app in mod_wsgi daemon mode.
>
> Because of performance concerns when dealing with large numbers of
> concurrent requests, I wanted to modify django to keep persistent DB
> connections to Postgres using a connection pool.
>
> This in turn got me wondering, how can I persist a thread pool, or
> even a simple DB connection, across requests?  I realize anything that
> is global to the wsgi entry point script will persist.  The current
> wsgi entry point I'm using is something like:
>
> import django.core.handlers.wsgi
>
> _application = django.core.handlers.wsgi.WSGIHandler()
>
> def application(environ, start_response):
>     return _application(environ, start_response)
>
> Obviously _application will remain, but since code modules are
> dynamically loaded based on URL resolvers, would the view/model/db
> connection not be destroyed once the variables referencing said
> objects go out of scope?
>
> From logging statements, it has become apparent I can in fact make DB
> connections persistent simply by not closing the DB connection after
> the request has finished.  Unfortunately, I also found this to slowly
> leak socket connections to the DB eventually making it so that I can't
> log into the DB, hence why I was looking into a connection pool.
>
> Anyways, I was hoping someone could shed some light as to the
> internals of python/django on why django/db/__init__.py is able to
> reference persistent connections.
>
> My best guess is that because
>
> connections = ConnectionHandler(settings.DATABASES)
>
> is at the top level of a module, it remains held within the python
> interpreter after being imported, thus holding a reference.
>
> Any insight would be greatly appreciated.

First, you should look into external connection poolers. PgBouncer is
excellent if you need just a connection pool. Pg-pool II is another
option, but it provides way more than just a connection pool.

If you would like to implement a connection pool in Python code, it
isn't the easiest thing to do. If you are using Django 1.4, then the
connections object will be thread local - that is, it provides a
different connection for each thread. Note that if you are using
multiple processes (instead of threads) then the processes share
nothing between each other. This is the reason you should look for
external poolers - they can see all connection attempts
simultaneously, but Python poolers are restricted to seeing one
processes at a time.

Still, if you want to experiment on poolers in Python code, I have
written a couple of attempts for a connection pool you can use as
bases of your work. https://github.com/akaariai/django-psycopg-pooled
and https://github.com/akaariai/django_pooled.

Short answer from my experiments: you do not want to implement
connection pooling in Python, at least if performance is the reason
for pooling. You could however do some other funny things, like
rewriting queries to use prepared statements.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Thread safety with view arguments in class based views

[akaariai]

On May 18, 12:17 am, Rafael Durán Castañeda
 wrote:
> I'm wondering if this is suitable for a doc bug report, it is?

I think a direct github pull request is the way to go. And yes, the
docs could be clarified here.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Model design question

[akaariai]

On May 18, 5:25 am, oneroler  wrote:
> Thanks Mike, that is what I was originally planning to do but realized
> there would be situations where that wouldn't do exactly what I wanted.
> For example, if there is a business that only has the strategy 'wholesale'
> assigned, using ForeignKey would still allow me to assign a different
> strategy to a division.  I was hoping to find a solution where the strategy
> for a division is constrained by the strategies assigned to its respective
> business.

Django doesn't make this particularly easy. You should create the
constraint in the database by using custom SQL, and then constraint
the assignable objects in Python in your view code. The Strategy and
Business models will create three tables into the database, one of the
tables is the many-to-many table. The m2m table's structure should be
something like this (check manage.py sqlall output):

create table business_strategy(
id serial primary key,
business_id integer references strategy(id),
strategy id integer references business(id),
unique (business_id, strategy_id)
)

Now, what you need to do is create a foreign key pointing to
business_id, strategy_id for your division table. The model should
look like this:

class Division(models.Model):
business = models.ForeignKey(Business)
name = models.CharField()
strategy = models.ForeignKey(Strategy)

this creates a table (
 id serial primary key,
 business_id integer references business(id),
 name varchar(n),
 strategy_id integer references strategy(id),
)

You will need to drop the strategy_id -> strategy(id) foreign key by
hand and add a new one: business_id, strategy_id ->
business_strategy(business_id, strategy_id). Check your database
vendors docs for details how to do this. You will want this constraint
to be deferred if available on your DB.

So, now you should have correct constraint in the database. You will
still have some problems: changing division's business changes the set
of available strategies. On UI side you will probably need to create
an AJAX call on business changed to fetch the available strategies. In
forms code you will need to do something like this:

class DivisionForm(forms.ModelForm):
class Meta:
  model = Division
def __init__(self, **kwargs):
  # Note, intentionally restrict the forms usage to only
kwargs.
  super(DivisionForm, self).__init__(**kwargs)
  business_id = None
  if 'initial' in kwargs:
  business_id = kwargs['initial'].business_id
  if 'data' in kwargs and 'business_id' in kwargs['data']:
  business_id = kwargs['data']['business_id']
  self.fields['strategy'].queryset =
Strategy.objects.filter(business_set=business_id)

So, the restriction should be doable both in DB and Django, but isn't
trivial to do. The above is pseudo-code style, so the example code
will likely not work by copy-pasting...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Deferred fields problem.

[akaariai]

On May 14, 3:29 pm, Jani Tiainen  wrote:
> 14.5.2012 14:50, akaariai kirjoitti:
>
>
>
>
>
>
>
>
>
>
>
> > On May 14, 2:37 pm, Jani Tiainen  wrote:
> >> Hi,
>
> >> I have in my database quite a bunch of a models where I do have quite
> >> large fields and I'm using .only('pk', 'identifier') to fetch only those
> >> two fields - mainly to make serverside natural sort for a data. Database
> >> is legacy one and it's being used with other external programs as well
> >> so changing it is not possible.
>
> >> After sorting I just need to get slice of data and fetch rest of fields
> >> in one query. I haven't found simple way to fetch rest of the fields in
> >> one request.
>
> >> So what I would like to do:
>
> >> qs = MyModel.objects.all().only('pk', 'identifier')
>
> >> list_of_models = sort_naturally(qs, 'identifier')
>
> >> sublist_of_models = list_of_models[10:20]
>
> >> for model in sublist_of_models:
> >>       model.fetch_all_deferred_fields()
> > While not totally same, how about:
> > for model in MyModel.objects.all(pk__in=[obj.pk for obj in
> > sublist_of_models]):
> > or maybe this will work, too:
> > for model in MyModel.objects.all(pk__in=sublist_of_models):
>
> > You will lose the sorting, so you will need to resort the models
> > again. If your model instances are changed, you will lose those
> > changes.
>
> > If the above isn't enough the best you can do currently is to create a
> > helper method which will do the merge manually.
>
> That would probably do, and now you mentioned it I already have merge
> function that can merge arbitrary dicts to models... ;)
>
> "Dare to be stupid" like Weird Al Yankovic sings...
>
> Of course my DB backend Oracle has limit of (I think it was 1000)
> elements in parameter list so it would require additional work if
> someone wants to work with more than 1k elements in a sublist.

I don't know the version of Django you are using, but at least 1.4
should automatically know how to split the list into multiple parts to
work around the 1000 parameters limit in Oracle. See:
https://github.com/django/django/blob/stable/1.4.x/django/db/models/sql/where.py#L181

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: database routers from html code

[akaariai]

On May 14, 11:43 am, Antonimus  wrote:
> Hi, I'm new to django and web framework. What I'm trying to do is to
> select what database to connect from html template select. In the
> login form, besides user and password, it's a select field with
> database alias to connect:
>      
>       {% for db in db %}
>              {{ db.DbCode }} |
> {{ db.DbName }}
>       {% endfor %}
>    
> where DbCode is db_alias from settings.py.
> How to pass DbCode value to routers.py?
> Thanks

you will need a middleware and use threading.local.

The idea is that the login form will set the used db alias into the
user's session. The middleware will then set a threading.local value
somewhere based on the session db value. Finally the router will read
the threading.local value and do routing based on that value.

However, I do not recommend the above. Instead, I would either try to
pass the DB to use explicitly (by the "using" queryset parameter), or
create a special database alias ("session_db" perhaps) which always
points to the user's selected DB. This alias does not need to exist in
the settings.py file. Again, a middleware is needed. Set the
connections['session_db'] to point to the right database
(connections['session_db'] = connections[session.login_db]) NOTE!
works only in 1.4+! Now, again use the "using" queryset argument, but
you can always set this to "session_db" for the queries which should
go to the session db. It is now explicit which queries go to the
session's database, which not.

To do any of the above you will need some knowledge of Python and/or
Django's database routing/connections implementation. For the latter
look into django.db.util, django.db.__init__ and django.db.backends.

Be careful, you are dealing with potential concurrency issues. Getting
things wrong will likely lead to confusing data corruption issues. Be
extra-careful with transaction handling. The default transaction
handling only works for the default database alias - so you will want
to make sure the user's database alias is also under transaction
control.

I hope the above helps more than confuses... :)

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Deferred fields problem.

[akaariai]

On May 14, 2:37 pm, Jani Tiainen  wrote:
> Hi,
>
> I have in my database quite a bunch of a models where I do have quite
> large fields and I'm using .only('pk', 'identifier') to fetch only those
> two fields - mainly to make serverside natural sort for a data. Database
> is legacy one and it's being used with other external programs as well
> so changing it is not possible.
>
> After sorting I just need to get slice of data and fetch rest of fields
> in one query. I haven't found simple way to fetch rest of the fields in
> one request.
>
> So what I would like to do:
>
> qs = MyModel.objects.all().only('pk', 'identifier')
>
> list_of_models = sort_naturally(qs, 'identifier')
>
> sublist_of_models = list_of_models[10:20]
>
> for model in sublist_of_models:
>      model.fetch_all_deferred_fields()
While not totally same, how about:
for model in MyModel.objects.all(pk__in=[obj.pk for obj in
sublist_of_models]):
or maybe this will work, too:
for model in MyModel.objects.all(pk__in=sublist_of_models):

You will lose the sorting, so you will need to resort the models
again. If your model instances are changed, you will lose those
changes.

If the above isn't enough the best you can do currently is to create a
helper method which will do the merge manually.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: postgresql connection is not closed

[akaariai]

On May 14, 9:05 am, shariq  wrote:
> Hi,
> I am running django-1.2 with postgresql.
> I am facing the problem.
> Django is not closing the connection and the connection is
> retained by db-server until DBA goes and kill the connection manually.

Are you using multiple databases? Is this happening in "batch job"
processing? That is, the connection is left open in situations where
you are not doing normal HTTP request serving.

When using connections outside of request serving you are responsible
for closing the connections manually yourself. However, in request
processing situations Django will close the connections automatically
when the request is processed.

There is definitely some room for improving the documentation of how
to use Django connections safely in batch job processing. Leaving open
transactions behind in multidb situations is too easy to do currently,
and closing connections isn't too well documented either.

Currently you will need to do something like this:
from django.db import connections
def close_connections():
for conn in connections.all():
conn.close()
from django.db.transactions import commit_on_success
#repeat for every DB you are going to use in your batch job.
@commit_on_success(using='default')
def _my_batch_job():
# real code here

def my_batch_job():
try:
_my_batch_job()
finally:
close_connections()

It would be nicer if you could do this instead:
@commit_on_success(using=['default', 'other'], close_connections=True)
def my_batch_job():
#real code here

In addition I would like to have an "explicit transactions only" mode,
where using a database which you haven't explicitly mentioned in
commit_on_success/commit_manually etc decorator will be treated as an
error. The reason for this is:
@commit_on_success
def my_batch_job():
for SomeModel.objects.all():
do something here...
the above code looks safe, but it might be that
SomeModel.objects.all() actually queries some other database than the
default on - and you have IDLE IN TX error right there.

The above went a little off-topic. So in short: you are responsible
for closing connections manually except in request processing, where
connections are closed automatically.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django psycopg error

[akaariai]

It seems the browsershots really uses psycopg, and you will need that
to use the software. The latest releases of the psycopg 1 series are
from late 2010 so you have a good chance to get it working.

 - Anssi

On May 13, 1:52 am, Mike Di Domenico 
wrote:
> It really says psycopg, not psycopg2. I am not entirely sure what is being
> imported for I am trying to install Browsershots Central
> Server,http://code.google.com/p/browsershots/source/checkout, and I have
> never used Python/Django before.
>
> Thanks for all of your help,
> Mike
>
>
>
>
>
>
>
> On Friday, May 11, 2012 3:58:07 PM UTC-4, akaariai wrote:
>
> > On May 11, 7:37 pm, Mike Di Domenico 
> > wrote:
> > > I have the python-psycopg2 package installed and the settings for my
> > engine
> > > are 'django.db.backends.postgresql_psycopg2' and I am still getting the
> > > error.
>
> > > When I do "python manage.py shell" I get the error "Error: No module
> > named
> > > psycopg".
>
> > > The contents of manage.py are the following:
> > > from django.core.management import execute_manager
> > > import settings # Assumed to be in the same directory.
>
> > > if __name__ == "__main__":
> > >     execute_manager(settings)
>
> > Do you really get the error "No module named psycopg"? No 2 in there?
> > Might it be you are trying to import psycopg somewhere in the project
> > - django doesn't try to import psycopg, only psycopg2...
>
> >  - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django unit test. Access the 'real data base'

[akaariai]

On May 11, 4:38 pm, Gelonida  wrote:
> For one unit test I would like to use the real data base and not the
> test data base.
>
> Basically this unit test shall verify that the server database fulfills
> certain consistency criterea.
>
> How can I write one django test, that uses the 'default' (non test database)
>
> I tried to use Model.objects.using('default').all() but still seem to
> receive an empty list (thus the contents of the test data base)

The real database is not available in testing (it's a feature). If you
want to access the real database, you will have to do it yourself,
something like this might work in 1.4:
def test_something():
try:
old_db = django.db.connections['default']
django.db.connections['default'] =
old_db.__class__(old_db.settings_dict)
# you might want to print the settings dict to see what you
need to update...
django.db.connections['default'].settings_dict['NAME'] =
real_db_name
# test the db here
finally:
   django.db.connections['default'] = old_db

The above relies on internals of Django, and in addition if something
goes wrong, it is possible the real database gets flushed (Django's
testing framework aggressively flushes databases). I haven't tested
the above code. Test carefully and don't blame me if something goes
wrong.

I am not sure but I guess the testing framework misses the concept of
read-only database. That might be useful for some testing cases.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django psycopg error

[akaariai]

On May 11, 7:37 pm, Mike Di Domenico 
wrote:
> I have the python-psycopg2 package installed and the settings for my engine
> are 'django.db.backends.postgresql_psycopg2' and I am still getting the
> error.
>
> When I do "python manage.py shell" I get the error "Error: No module named
> psycopg".
>
> The contents of manage.py are the following:
> from django.core.management import execute_manager
> import settings # Assumed to be in the same directory.
>
> if __name__ == "__main__":
>     execute_manager(settings)

Do you really get the error "No module named psycopg"? No 2 in there?
Might it be you are trying to import psycopg somewhere in the project
- django doesn't try to import psycopg, only psycopg2...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django timezone doesn't show the right time?

[akaariai]

I see I am talking about concepts which are not yet familiar to you.

By template I mean Django's HTML template engine. This is introduced
in part 3 of the tutorial.

The reason for the errors is that Poll is not an instance, it is the
class so it doesn't have pub_date. poll isn't assigned to yet. You
could do poll = Poll.objects.all()[0], and then poll would be assigned
the first Poll object instance, fetched from the database.

 - Anssi

On May 1, 9:45 pm, Dan Santos  wrote:
> Hi thanks for taking the time to explain to me!
>
> Unfortunately I still don't understand the outputs that I get in Python
> console, maybe it's because I don't understand the concept of templates.
> But this is what I tried to do in Python console.
>
> >>> p=Poll(question="Giddeup'ah!", pub_date=timezone.now())
> >>> p.save()
> >>> p.id
> 4
> >>> p.question
> "Giddeup'ah!"
> >>> *p.pub_date*
>
> datetime.datetime(2012, 5, 1, *18, 25*, 39, 804854, tzinfo=<*UTC*>)
>
> >>> *Poll.objects.all()*
>
> [,  2012-04-26 10:30:23.624835+00:00>,
> ,  2012-05-01 *18:25*:39.804854*+00:00*>]
>
> p.pub_date and Poll.objects.all() show the same time info, so how do I see
> the time with the simplest of templates?
>
> Also when I tried typing "poll.pub_date" in different ways I get these
> errors.
>
> >>> *Poll.pub_date*
>
> Traceback (most recent call last):
>   File "", line 1, in 
> AttributeError: type object 'Poll' has no attribute 'pub_date'
>
> >>> *poll.pub_date*
>
> Traceback (most recent call last):
>   File "", line 1, in 
> NameError: name 'poll' is not defined
>
> >>> *poll.pub_date()*
>
> Traceback (most recent call last):
>   File "", line 1, in 
> NameError: name 'poll' is not defined
>
> >>> *Poll.pub_date()*
>
> Traceback (most recent call last):
>   File "", line 1, in 
> AttributeError: type object 'Poll' has no attribute 'pub_date'
>
> Here's a better overview of what my relevant files looks like and what
> setup I have tried to use.  Maybe it will help in finding if I
> have mis-configured 
> something.http://www.python-forum.org/pythonforum/viewtopic.php?f=19&t=34082
>
>
>
>
>
>
>
> On Tuesday, May 1, 2012 3:16:33 PM UTC+2, akaariai wrote:
>
> > On Apr 30, 11:03 pm, Dan Santos  wrote:
> > > Hi I'm a total programming newbie!
>
> > > ### INTRO ###
> > > I have been following this tutorial and the timezone outputs confuses
> > > me.
> >https://docs.djangoproject.com/en/1.4/intro/tutorial01/#playing-with-...
>
> > > And I'm still confused after reading these explanations, I basically
> > > need some really dumbed down answers to understand this timezone
> > > business:
>
> > > TIME_ZONE setting: How does it work?
> >http://groups.google.com/group/django-users/browse_thread/thread/bebb...
>
> > > ### QUESTION ###
> > > * Europe/Brussels has UTC+2 during summer time.
>
> > > When I run this as I follow the Django tutorial, then the time (20:34)
> > > is behind by 2 hours to my local time and UTC displays +00:00.
> > > Shouldn't it display 22:34 +02:00 instead for a server located at
> > > Europe/Brussels?
>
> > > >>> Poll.objects.all()
>
> > > ]
>
> > Internally Django works in UTC . When you display a value in a
> > template or in a form it will be converted to the currently active
> > time zone (by default settings.TIMEZONE). So, when you just do
> > Poll.objects.all() you will see the UTC time, as this is Python
> > internal representation. If you would do {{poll.pub_date}} in the
> > template, it would be displayed in the time zone you have currently
> > active.
>
> >  - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django timezone doesn't show the right time?

[akaariai]

On Apr 30, 11:03 pm, Dan Santos  wrote:
> Hi I'm a total programming newbie!
>
> ### INTRO ###
> I have been following this tutorial and the timezone outputs confuses
> me.https://docs.djangoproject.com/en/1.4/intro/tutorial01/#playing-with-...
>
> And I'm still confused after reading these explanations, I basically
> need some really dumbed down answers to understand this timezone
> business:
>
> TIME_ZONE setting: How does it 
> work?http://groups.google.com/group/django-users/browse_thread/thread/bebb...
>
> ### QUESTION ###
> * Europe/Brussels has UTC+2 during summer time.
>
> When I run this as I follow the Django tutorial, then the time (20:34)
> is behind by 2 hours to my local time and UTC displays +00:00.
> Shouldn't it display 22:34 +02:00 instead for a server located at
> Europe/Brussels?
>
> >>> Poll.objects.all()
>
> ]

Internally Django works in UTC . When you display a value in a
template or in a form it will be converted to the currently active
time zone (by default settings.TIMEZONE). So, when you just do
Poll.objects.all() you will see the UTC time, as this is Python
internal representation. If you would do {{poll.pub_date}} in the
template, it would be displayed in the time zone you have currently
active.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Slow page in admin

[akaariai]

On Apr 25, 4:22 pm, Jonas Ghyllebert  wrote:
> Hi,
>
> Thanks for the quick reply.
>
> when we test the application local, it works like a charm.
> But when we deployed it on Google App engine using Cloud SQL, the
> application works now and then and this particular page is much slower than
> the others.
> When there's an error, Django shows the 500 - internal server error.
>
> These are my models:
> *Zip*
>
> class Zip(models.Model):
>
>     zip_code = models.IntegerField()
>
>     zip_name = models.CharField(max_length=75)
>
>     zip_lang = models.CharField(max_length=2)
>
>     province = models.ForeignKey(Province)
>
>     class Meta:
>
>         ordering = ["zip_name"]
>
>         verbose_name = "postcode"
>
>         verbose_name_plural = "postcodes"
>
>     def __unicode__(self):
>         return self.zip_name + ' (' + unicode(self.zip_code) +')'
>
> *
> *
> *Region*
>
> class Region(models.Model):
>
>     region_name = models.CharField(max_length=75)
>
>     office = models.ForeignKey(Office)
>
>     region_primary = models.BooleanField()
>
>     def __unicode__(self):
>
>         return self.region_name
>
>     class Meta:
>
>         verbose_name = "regio"
>
>         verbose_name_plural = "regio's"
>
> There is an extra table which stores which zips are included in one Region:
> *Zip_per_region*
>
> class Zip_per_region(models.Model):
>
>     region = models.ForeignKey(Region)
>
>     zip = models.ForeignKey(Zip)
>
>     class Meta:
>
>         verbose_name = "postcode"

I am afraid I can't help you. It seems the issue is related to Google
App Engine and Django interactions, and I know nothing about GAE. I
hope somebody else is able to help you out.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Slow page in admin

[akaariai]

On Apr 25, 3:35 pm, Jonas Ghyllebert  wrote:
> I'm new to Django and I would like to apologize if it's been asked before.
>
> I have two models *Region *and *Zip*. In the admin system, Zip is an inline
> model in Region
>
> Whenever i want to add a Region it shows directly 25 listboxes with zips.
> This however takes a long time to load the page and results in an internal
> error.
> There are 2,774 records in the zip table.
> I suspect the cause of this error is that for each list, Django connects to
> the database to fetch those records.
> Is there a way to solve this?

Your question is a little hard to answer - it is always a good idea to
include the full error message, and preferably the model definitions
etc in your post. You leave all who try to help guessing what the
problem might be...

Without knowing more about the situation I suggest you check the
queries the page loading generates (using logging, django-debug-
toolbar or your database's query logging utilities). Then check if
there is a possibility to use select_related or prefetch_related to
ease the situation.

The internal error seems a little strange. Can you share more details
about that?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django DateTimePicker only shows up for superusers inside my app

[akaariai]

Could you check if the non-admin user can download this script: "/
admin/jsi18n/". It seems downloading that script needs privileges to
access admin site.  That view just provides the javascript catalogs
for "django.conf" and "django.contrib.admin" packages. You can provide
them from different URL, see:
https://docs.djangoproject.com/en/1.4/topics/i18n/translation/#module-django.views.i18n

  - Anssi

On Apr 25, 12:40 am, Ahmad  wrote:
> I'm using django DateTimePicker widget -- AdminSplitDateTime() -- in event
> creating app, every thing goes fine when the loged in user has superuser
> permissions, but when normal user tries to access the creating form the
> widget don't show up.
>
> I don't know why it acts like this? and how to overcome this behavior?
>
> this is the the scripts included in my page
>
>     
>     
>      src="/static/admin/js/admin/RelatedObjectLookups.js">
>      src="/static/admin/js/jquery.min.js">
>      src="/static/admin/js/jquery.init.js">
>      src="/static/admin/js/actions.min.js">
>      src="/static/admin/js/calendar.js">
>      src="/static/admin/js/admin/DateTimeShortcuts.js">
>
> her is my form code
>
>     class Form(ModelForm):
>         """
>
>         """
>
>         class Meta:
>             model = Model
>             exclude = ('creator')
>         def __init__(self, *args, **kwargs):
>             super(Form, self).__init__(*args, **kwargs)
>             self.fields['start'].widget = widgets.AdminSplitDateTime()
>             self.fields['end'].widget = widgets.AdminSplitDateTime()

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: 'too many SQL variables' error with ModelMultipleChoiceField

[akaariai]

On Apr 23, 6:54 pm, "Daniel Sokolowski"
 wrote:
> +1
>
> I use SQlite3 on lower traffic production sites due to it's ease of install
> and deployment. Even though I only once I reached this 1000 variable limit I
> would like to see some django work around.

I thought a little more about this, and I find it unlikely that the
qs.filter() issue will ever be solved. The idea presented upthread
might work, but it will be a mess to implement. It adds some serious
complexity to the query generation: temp table creation, insertion
into the temp table, the where clause generation, and finally you need
to clean up the temp table. All this to work around a limitation in a
single database. An issue you can work around by recompiling SQLite.
Or you could just use a different database.

The implementation would need to be extremely good to have any chance
to get in. The work needed seems huge compared to what is gained.

So, as said upthread: there is only so much Django can hide about the
underlying database. The 1000 parameters limit for single query seems
to be one of those issues. If there is some easy trick Django will use
that. But the temp table workaround seems just too messy. Still, I
would be happy to be proven wrong here.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Problem with get_or_create

[akaariai]

On Apr 23, 4:26 pm, Murilo Vicentini 
wrote:
> Hey guys,
> I'm having a bit of a problem using the get_or_create with a
> ManyToManyField in my model.
> class Run(models.Model):
>         distros = models.ForeignKey('Distro')
>         compilers = models.ManyToManyField('Compiler', blank=True,
> null=True)
>         adapters = models.ForeignKey('Adapter')
>
>         Task = models.IntegerField()
>         Path = models.CharField(max_length = 300)
>         Date_Time = models.DateTimeField()
>
> class RunForm(ModelForm):
>         class Meta:
>                 model = Run
>
> What I'm trying to do is merely trying to save in the database if the entry
> does not exist already (hence why I'm trying to use get_or_create). Here is
> how I'm doing:
> fields = {
>          'Task': task,
>          'Path': runpath,
>          'Date_Time': datetime(2012, 4, 10, 10, 20, 0),
>          'adapters': adapters.id,
>          'distros': distros.id,    }
>
> form = RunForm(fields)
> if form.is_valid():
>           runs, created = Run.objects.get_or_create(**form.cleaned_data)

Try "runs = form.save()", get_or_create doesn't seem to be the correct
method in this situation.

If you want to skip some fields you can use the form's Meta attributes
to do that.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: 'too many SQL variables' error with ModelMultipleChoiceField

[akaariai]

On Apr 23, 10:48 am, Lukas Zilka  wrote:
> Hey Russ,
>
> thanks for your reply. I looked at the bulk insertion problem, and
> that made me think that probably using the variables themselves is the
> problem. It does not really make sense to me why anybody bothers to
> use variables in a long SQL query for the IN clause, let alone a bulk
> insertion? Does that benefit anything?
>
> Anyway, I conducted an experiment and tried a simple SQL query with
> lots of items in the IN clause to prove that its the variables
> themselves, not the limitation of IN clause in SQLite, is whats
> causing the problem. I have been successfully able to execute a query
> as "SELECT * FROM projects WHERE id IN (...)" where on the place of
> `...` there were 15,000 numbers. So SQLite has no problem with lot of
> stuff as an argument of the IN operator. Now, the only limitation is
> the length of the SQL query itself. According to the SQLite
> documentation it is 1megabyte, and that suffices for 100k+ elements.
> With that big query, you are right, it would be very impractical and
> slow for the end users to interact with the application (sending 1M of
> data to the webserver will probably be very unresponsive), so I think
> this is a fair limitation that should never be exceeded for this use.
>
> In Django code it seems that it would suffice to make a change in the
> file 'db/models/sql/where.py'. Particularly, condition on the number
> of elements of IN, and, if it is over let's say 100 of them, put them
> into the SQL query directly as a string (e.g. '1,2,3') - not as
> variables('?, ?, ?').
>
> Though, for the future, I still believe there should be a better
> solution than to rely on not reaching this limit. I would propose that
> a temporary table should be created, filled up with the right
> arguments of IN operator, and the query rewritten as a JOIN on the
> original left argument of IN (or for simplicity, and perhaps worse
> performance, a nested query) and this temporary table. That of course
> only if the number of elements on the right side of IN is more than
> some number. But this is for another discussion.
>
> My question is therefore: Will the change in Django code that I
> propose have any bad consequences or do you think it might actually
> work satisfactorily?

First the "write the query as "1, 2, 3", not as "%s, %s, %s", (1, 2,
3). The problem is SQL injection. You could do that for integer
parameters easily, but on the whole it is not a nice way.

The create temp table + join seems hard. But you could do "exists"
query instead. A query like "select * from tbl where id in (a list)"
could be rewritten to "select * from tbl where exists (select 1 from
temp_table where temp_table.id = tbl.id)". This could be handled in
sql/where.py somewhat easily.

For me this issue isn't that important. I don't use SQLite except for
testing. If somebody wants to work on this issue, I must warn that it
is possible (if not likely) that some core developer will say "too
ugly" to this solution. I might be willing to accept the solution if
it was clean enough, as this would nicely abstract away this limit of
SQLite.

So, in short: this idea is definitely worth more investigation.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: 'too many SQL variables' error with ModelMultipleChoiceField

[akaariai]

On Apr 23, 1:43 pm, Javier Guerra Giraldez  wrote:
> On Mon, Apr 23, 2012 at 3:58 AM, akaariai  wrote:
> > The last one, qs.filter(id__in=large_list) is pretty much impossible to fix.
>
> what's wrong with:
>
> qs.filter((Q(id__in=large_list[:n])|Q(id__in=large_list[n:])))  ?
>
> >   1. Use different database. I guess you have a reason to use SQLite so this
> > might not be an option.
>
> Oracle has a 1000-element limit for the IN ()  clause.  Of course,
> it does work with (IN(<1sthalf>) or IN(<2ndhalf>))

SQLite has the limit for the whole query, Oracle for just one IN
clause. For Oracle we already do that split. For SQLite we would need
to split the _whole query_ into two parts (that is, run
two .exectutes()) and then "join" the parts in Python. The joining in
Python is the hard part.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: 'too many SQL variables' error with ModelMultipleChoiceField

[akaariai]

On Monday, April 23, 2012 1:56:19 AM UTC+3, Russell Keith-Magee wrote:
>
> It's also probable that it's something that is new to 1.4 (or, at least, 
> that it manifests in slightly different ways in 1.4). One of the features 
> added in 1.4 is bulk insertion of data. This means that you can use a 
> single SQL statement to insert all your m2m relation; however, the downside 
> is that on SQLite, there is a limit of 500 insertions that can be done at 
> any one time (2 values per insertion).
>
> This is logged as ticket #17788. There have been some discussions about a 
> fix for this problem that will break the bulk insertion into batches. I 
> suspect that when this problem is fixed, your problem will go away.
>
There are at least three different instances of the 1000 variables problem. 
One is the bulk_create() which is likely to get fixed. Then there is 
delete(). This might get fixed. The last one is 
qs.filter(id__in=large_list) which will not get fixed.

The reason for fixing bulk_create() is that it is somewhat easy to do so. 
Just split the batch of objects into smaller batches and iterate. delete() 
isn't as easy, but the same split + iterate approach could work. If it gets 
too complicated then it will not be fixed, if it is easy then it might get 
fixed. The last one, qs.filter(id__in=large_list) is pretty much impossible 
to fix. The reason is you can't split that query into multiple parts. Or 
not for any non-trivial query anyways. For example ORDER BY is enough to 
break any attempt to split the query into parts.

While Django's ORM should hide limitations in the underlying database it 
can't do it perfectly. My viewpoint is that Django should hide limitations 
if it is easy to do so. It is all about cost-benefit ratio.

For your particular problem you have at least these choices:
  1. Use different database. I guess you have a reason to use SQLite so 
this might not be an option.
  2. Compile SQLite yourself with a higher parameter limit. If you need to 
deploy your software on multiple machines this might be hard to do.
  3. Work around the issues in your code. Maybe a ModelMultipleChoiceField 
subclass which does the .clean() in a safe way would work. Or better yet do 
as Russell said: try to remove the need to have 1000 options in one field 
altogether.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/QtYFVgICbC8J.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django help required with non foreign key relations

[akaariai]

On Apr 23, 8:15 am, Aditya Sriram M  wrote:
> Ohh this looks little promising..
>
> However,
>
>    1. I need a few cols of table1 and few from table 2 to be displayed. How
>    can we achieve that at the .filter() level? and
>    2. At the view level like using the list_display () for displaying them
>    in the Admin interface

If you want to get results just define the ForeignKey (or OneToOneKey)
in your Customer or User model. After this .select_related
and .prefetch_related should solve the problems you are currently
facing.

If you can't define that ForeignKey for one reason or another you need
a lot more knowledge of how Django works. All the things you need to
do are too numerous to handle in a django-users post. It is possible
there will be some problems which are very hard to solve along this
path.

In short, I recommend to just define the relation between the models.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django help required with non foreign key relations

[akaariai]

On Apr 22, 1:31 pm, Aditya Sriram M  wrote:
> File myapp/models.py has this sample code..
>
> from django.db import models
>
> # model for 'user' table in database oracle_dbuser1:user
> class User(models.Model):
>     . . .
>     customerid = models.BigIntegerField()
>
> # model for 'customer' table in database oracle_dbuser2:customer
> # Note that there is no Foreign key integrity among these legacy tables.
> class Customer(models.Model):
>     . . .
>     customerid = models.BigIntegerField()
>
> and the file myapp/admin.py has the following code:
>
> from maasusers.models import User, Customer
> from django.contrib import admin
>
> class UserAdmin(admin.ModelAdmin):
>     # A handy constant for the name of the alternate database.
>     db_one = 'dbuser1'
>     db_two = 'dbuser2'
>
>     # display in a list
>     list_display = (. . .) # question 1
>
>     def queryset(self, request):
>         result = super(UserAdmin, self).queryset(request).using(self.db_one) 
> # question 2
>         return result
>
> # Register the Poll class
> admin.site.register(User, UserAdmin)
> admin.site.register(Customer, UserAdmin)
>
> Question 1: Refer above: I want to display columns of both the tables. How
> can I achieve this? Eg.Select usr.col1, usr.col2, cust.col1, cust.col10
> from user usr, customer cust where usr.col2 = cust.col3;
>
> Question 2: How to write a corresponding queryset() function using the using
>  function?

There is no way to do a join between models if there is no relation
between them. I see two ways forward: modify one of the models to have
a foreign key - even if there is no foreign key in the DB, you can
have one in the models - or do the join in Python (fetch all
customers, then fetch users by
User.objects.filter(customerid__in=[obj.customerid for obj in
customers]). Finally join users to its customer).

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: 'builtin_function_or_method' object has no attribute 'split' after upgrading to Django 1.4

[akaariai]

On Apr 22, 4:45 am, Taras_96  wrote:
> Hi,
>
> I've just upgraded to Django 1.4, and I was getting a
> 'builtin_function_or_method' object has no attribute 'split' error
> being thrown. I traced this down to an erroneous use of distinct, I
> had:
>
> X.objects.distinct(id)
>
> Which had previously worked, but was now throwing. I've removed the
> 'id' parameter (which is also a Python keyword).
>
> Is it worth putting this into the upgrade notes for 1.4

I am not sure how important the release notes part is. However, this
is the second time I see the above problem raise by users, so maybe
improving the error message from .distinct() would be possible. It
could say something like "In 1.4 the args must be strings". If
somebody comes up with good wording for the error message it could be
possible that it will be included in future versions of 1.4.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: image KeyError

[akaariai]

On Apr 21, 11:30 pm, psychok7  wrote:
> hi there , so i am not sure if i am doing this the right way, but
> im successfully able to upload files(images) and store them in my defined
> folder  and the link in my mysql  database.
>  the error comes when i try to return the image with the HTTP RESPONSE were
> i get a image keyError.. i am not sure wht i am doing wrong..
> here is my code:
>
> def handle_uploaded_file(f,u):
>     profile=u.get_profile()
>     profile.avatar=f.name
>     destination = open('images/users/'+f.name, 'wb+')
>     for chunk in f.chunks():
>         destination.write(chunk)
>     destination.close()
>     profile.save()
>
> def upload_file(request,user_id):
>     u = get_object_or_404(User, pk=user_id)
>     if request.method == 'POST':
>         form = UploadFileForm(request.POST, request.FILES)
>         if form.is_valid():
>             handle_uploaded_file(request.FILES['file'],u)
>             uploadedImage = form.cleaned_data['image']
>
>             return HttpResponse(uploadedImage.content,mimetype="image/jpeg")
>     else:
>         form = UploadFileForm()
>
>     c = {'form': form,'user' : u}
>     c.update(csrf(request))
>     return render_to_response('accounts/upload.html', c)
>
> 
>    {% csrf_token %}
>    
>        {{form.as_table}}
>     
>     
> 
>
> thanks in advance

Your question is very hard to answer without the full error message.
Some wild guesses of what is going wrong: maybe the
cleaned_data['image'] is missing or it isn't a file?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Modeling Question on Many-to-Many(with through) and Proxy models

[akaariai]

On Apr 21, 8:35 pm, Arruda  wrote:
> Should I create this many-to-many table as a normal model(no proxy) and set
> the foo foreingkey to Foo1 instead of Foo?
> Or should I use only FooBar as many-to-many and create some specific
> queries that would convert ALL the returned foo objects as Foo1 instance?

I am not sure if I am answering your question... But here is a dirty
hack to try:
qs = SomeModel.objects.all()
qs.model = SomeModelProxy
print list(qs)

You should get proxy model instances from the queryset.

Now, I just quickly tested this a minute ago, and it seems to work in
a very simple cases. The above uses internals of Django's ORM in a way
it was not designed to be used. So mandatory warnings: If anything
breaks, though luck. A minor version upgrade could break your code
without any notice.

So, in the m2m case I guess I would create a property:
class SomeModel:
 foos = M2M(Foo)

 def _foo1s(self):
   qs = self.foos.all()
   qs.model = Foo1
   return qs
 foo1s = property(_foo1s)

The above might just work. Or not. Once again, be very cautious if you
use any of the above.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django 1.4, Postgresql and partitioned table

[akaariai]

On Apr 18, 3:49 pm, Michaël Journo 
wrote:
> Well you were right. The insert didn't return the id, so that was the
> problem.
>
> I used your trick (customizing the save method) and it seems to work
> just fine.

Glad it helped.

A mandatory warning: you are using non-public parts of Django's
backends, and the trick isn't something django-devs will consider
supported usage. So, you are on your own if anything breaks. If at all
possible try to return the ID from the insert.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How to release the caching of QuerySet

[akaariai]

On Apr 18, 2:48 pm, Kejun He  wrote:
 > (2)
> About the query if be executed
>
> if the code like below, and do no operation on variable tmp:>>>tmp = 
> Demo.objects.filter(pid=p_id)
>
> The query would not be executed actually untill do some operation on
> tmp(exp:tmp.count())
>
> if just do "Demo.objects.filter(pid=p_id)", the query would be executed at
> once and back the result.

If you just do Demo.objects.filter(pid=p_id) the query will not be
executed. Of course, if you do that in the Python shell, it will be
executed, as Python shell will then print the return value (in this
case, the generated queryset). So, in your test() method the query
should not execute.

> I would update Django in my machine, and repeat to check if problem exists
> on the latest version.

If you can try this on 1.4 it would be great. Be sure to run the
test() method 10-20 times so that the garbage collector has a reason
to clean up trash. If the leak is present in 1.4 there is a bug
somewhere. In that case try to provide a self-contained demo project.
It would help a lot in debugging this.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How to release the caching of QuerySet

[akaariai]

On Apr 18, 1:05 pm, Kejun He  wrote:
> and define a method named test in views.py. like below:
>
> def test(p_id):
>     # Just execute the filter 1000 times
>     for i in xrange(1000):
>         Demo.objects.filter(pid=p_id)
>
   
> run the test function again:>>>test(1)
>
> The occupied memory increase again.
>
> But the memory never been released until I input exit() to quit the python
> shell.

If I understand correctly, you are never actually executing the query?
You just do "Demo.objects.filter(pid=p_id)" repeatedly, and never do
anything with the result. This should definitely not result in memory
leak.

Django 1.2 isn̈́'t supported any more, and I could not repeat your
problem using current Django trunk head.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django 1.4, Postgresql and partitioned table

[akaariai]

The problem seems to be that INSERT INTO partitioned_table values(...)
RETURNING ID isn't for some reason returning the ID. Can you confirm
this is the case by running the insert query manually from dbshell?

I believe 1.3 used SELECT
currval(pg_get_serial_sequence('partitioned_table', 'id')) instead of
RETURNING ID. So, that might explain this issue. You could try to set
connection.features.can_return_id_from_insert to False just before
executing any insert query into this table (most conveniently done by
overriding the model's save() method). This is an ugly hack, but it
might make your code work in 1.4...

 - Anssi

On Apr 17, 3:42 pm, Michaël Journo 
wrote:
> Hello,
>
> I use postgresql 9.1 with postgresql_psycopg2 on debian (actually this
> also happens on my mac). I recently moved to django 1.4 and I got this
> error while creating a simple object :
> o = Object()
> o.fieldA = ..
> 
> o.save()
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> lib/python2.6/site-packages/django/db/models/base.py", line 463, in
> save
>     self.save_base(using=using, force_insert=force_insert,
> force_update=force_update)
>   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> lib/python2.6/site-packages/django/db/models/base.py", line 551, in
> save_base
>     result = manager._insert([self], fields=fields,
> return_id=update_pk, using=using, raw=raw)
>   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> lib/python2.6/site-packages/django/db/models/manager.py", line 203, in
> _insert
>     return insert_query(self.model, objs, fields, **kwargs)
>   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> lib/python2.6/site-packages/django/db/models/query.py", line 1576, in
> insert_query
>     return query.get_compiler(using=using).execute_sql(return_id)
>   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> lib/python2.6/site-packages/django/db/models/sql/compiler.py", line
> 914, in execute_sql
>     return self.connection.ops.fetch_returned_insert_id(cursor)
>   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
> lib/python2.6/site-packages/django/db/backends/__init__.py", line 529,
> in fetch_returned_insert_id
>     return cursor.fetchone()[0]
> TypeError: 'NoneType' object is unsubscriptable
>
> My postgresql table for object is a partioned one, based on a trigger
> on insert that look for the right child table. I have a hard time
> finding information about this, except one post 
> herehttp://postgresql.1045698.n5.nabble.com/INSERT-INTO-RETURNING-with-pa...
> suggesting to use rule instead of triggers for handling insert
> returning clause (and it seems it really can handle only ONE returning
> clause, i.e. return object.id).
>
> And in any way, what I really don't understand is that everything
> worked just fine in django 1.3.
>
> I don't know this is a real issue or me handling table partitioning
> the wrong way.
>
> Thank you for your help.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django ORM - query help

[akaariai]

On Apr 12, 4:01 pm, David  wrote:
> The above ORM statement however does not look as elegant to read as I have
> come to expect from Django though. The resulting SQL doesn't seem too
> shabby however.

.distinct(fields) + .order_by() is pretty low level stuff - that is
why it would be nice if there was a dedicated API for queries like
this. Something like:

> qs.first_related('log', order_by='-modified_on')
OUT: qs annotated with log entries.

Turning that into efficient and cross-DB compliant SQL isn't the
easiest thing to do. If you want N first objects instead of one, then
it is going to be even harder.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django ORM - query help

[akaariai]

On Apr 12, 11:52 am, David  wrote:
> > Hi Jani
>
> > That was very helpful. Is there a way to include select_related into that
> > query? or do I have to list every single field I would like to return using
> > values()?
>
> > last_deleted = ModificationLog.objects.values('thing__id', '
> > thing__prefix', ' thing __first_name', ' thing__last_name', '
> > thing__company_name', 'thing__creator', ' thing __created_on',
> > 'modifier').annotate(deletion_date=Max('modified_on')).filter(thing__deleted=1).order_by('-deletion_date')[:20]
>
> > This for example works, but I don't have access to User. Thing contains a
> > FK to the User model, but I'm unable to do in the template {{
> > object_list.creator.get_full_name }}
>
> > But I am a lot lot closer than I was before, thank you.

One reason why such queries are hard is that they aren't easy to
express using SQL. The left join is correct from ORM perspective: it
can't know that each entry will have at least one log entry attached.
I am not sure how to tell that to the ORM without doing ugly trick
using sql/query.py internals.

If you happen to be using PostgreSQL and Django 1.4 you could probably
do something like this (using DISTINCT ON support added in 1.4):

> ml_objs = ModificationLog.objects.distinct('thing__id').order_by('thing__id', 
> 'modified_on').select_related('thing').

You could then swap the objects around:

> [setattr(ml.thing, 'last_mod', ml) for ml in ml_objs]
> things = [ml.thing for ml in ml_objs]

I am not sure at all the above will work, or that the .distinct() and
order_by() calls are correct for your use case. But DISTINCT ON could
work here...

In general fetching the latest object(s) per related category is a
common operation, and it would be extremely nice if prefetch_related
and/or select_related would support that out of the box. It is a very
commonly repeating pattern: show me the latest mail in each thread,
show me the latest edit for each article, show the highest rating
employee per project and so on. A modification of this theme is "show
me the N latest edits per category" which is pretty hard to do
efficiently using the current ORM APIs.

So, if somebody has ideas how to implement this into prefetch_related/
select_related I am all ears.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: PostgreSQL Introspection Bug

[akaariai]

On Apr 11, 4:39 pm, Thomas Guettler  wrote:
> I fixed a postgreSQL introspection bug, but unfortunately it is still in 
> stage "new".
>
> Can someone please review it?
>
>    https://code.djangoproject.com/ticket/17785

I hope to find time to look at this. No promises, but added to my
rather long TODO list...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Model - create a filter and use it as argument

[akaariai]

On Apr 5, 1:00 am, Ivo Marcelo Leonardi Zaniolo 
wrote:
> Hi there,
>
> I'm need to pass a QuerySet as an argument, like a function pointer that
> will be
> evaluate in other time. Somebody has one idea how I can do this?
> I dont want to create a "box" method to encapsulate it.
>
> So, can anyone help me?

QuerySets are evaluated only when accessed. So, you can just do:

def method(qs):
qs = qs.filter(somethingelse=bar)
list(qs)

qs = SomeModel.objects.filter(something=foo)
method(qs)

The queryset will be evaluated at list(qs) time.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: custom manager to control access

[akaariai]

On Apr 4, 3:15 am, Mike Dewhirst  wrote:
> I have now discarded the idea :)
>
> I'm not very comfortable with thread locals. I need a bullet-proof
> approach which makes data from other companies invisible to members of
> this company. I suppose a view decorator is the way to go but I would
> have preferred a deeper model-level approach.

You have two ways to pass this kind of information around:
  1. explicitly as parameters
  2. using thread locals (or other "global state").

So, you need to use explicit parameters. You should do something like
this:

class MemberManager(models.Manager):
def visible_for_user(self, user):
# Replace the below filter condition with appropriate logic.
return self.get_query_set().filter(company=user.company)

Now, instead of using Member.objects.all() you should use
Member.objects.visible_for_user(request.user) in your code. I think
you can set the manager default for relations, so that you could do
group.membership.visible_for_user(request.user), but I haven't ever
done that. You will need to be careful when coding so that you don't
accidentally show or modify data from other companies. The way I would
do this is have the logic in the Model/Manager classes, and then use
that logic in your view code.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: AttributeError at /admin/coltrane/entry/add/ 'Entry' object has no attribute 'pubdate'

[akaariai]

Do you get a stacktrace from the error? You should get one, and it
should point out the code-path to the erroneous pubdate usage. If you
do not get one for one reason or another, search your project for the
string "pubdate" and go through all locations containing that string.

The code snippets you have below seem clean to me.

 - Anssi

On Apr 3, 10:59 pm, laurence Turpin 
wrote:
> Hello,
>
> I'm trying to learn Django from the book "Practical Django Projects 2nd
> Edition"
>  I'm using Django 1.3.1 and Python 2.7.2 on Windows 7
> Also I'm using Eclipse Indigo with pydev
>
> The current project I'm working on is to create a Weblog called coltrane.
> coltrane is an application that exists in a project called
> "cmswithpythonanddjango"
>
> Inside the models.py is a class called Entry which holds the fields used to
> Enter a weblog entry.
> When I go to localhost:8000/admin/    I get the option to add and Entry.
> I can select this option fill out sample data, but when I try to save it.
> I get the following error.
>
> AttributeError at /admin/coltrane/entry/add/
>
> 'Entry' object has no attribute 'pubdate'
>
> There is a field called pub_date in Entry
>
> It's as if I have made a typing error somewhere and typed pubdate instead of 
> pub_date.
>
> Below is the models.py, views.py , url.py , and admin.py files
>
> //
>
> Here is my models.py
>
> /
>
> from django.db import models
> import datetime
> from tagging.fields import TagField
> from django.contrib.auth.models import User
> from markdown import markdown
>
> class Category(models.Model):
>     title = models.CharField(max_length=250, help_text="Maximum 250
> characters.")
>     slug = models.SlugField(help_text="Suggested value automatically
> generated from title. Must be unique.")
>     description = models.TextField()
>
>     class Meta:
>         ordering = ['title']
>         verbose_name_plural = "Categories"
>
>     def __unicode__(self):
>             return self.title
>
>     def get_absolute_url(self):
>         return "/categories/%s/" % self.slug
>
> class Entry(models.Model):
>     LIVE_STATUS = 1
>     DRAFT_STATUS = 2
>     HIDDEN_STATUS = 3
>     STATUS_CHOICES = (
>     (LIVE_STATUS, 'Live'),
>     (DRAFT_STATUS, 'Draft'),
>     (HIDDEN_STATUS, 'Hidden')
>     )
>
>     # Core fields
>     title = models.CharField(max_length=250, help_text = "Maximum 250
> characters. ")
>     excerpt = models.TextField(blank=True, help_text = "A short summary of
> the entry. Optional.")
>     body = models.TextField()
>     pub_date = models.DateTimeField(default = datetime.datetime.now)
>
>     # Fields to store generated HTML
>     excerpt_html = models.TextField(editable=False, blank=True)
>     body_html = models.TextField(editable=False, blank=True)
>
>     # Metadata
>     author = models.ForeignKey(User)
>     enable_comments = models.BooleanField(default=True)
>     featured = models.BooleanField(default=False)
>     slug = models.SlugField(unique_for_date = 'pub_date', help_text =
> "Suggested value automatically generated from title. Must be unique.")
>     status = models.IntegerField(choices=STATUS_CHOICES,
> default=LIVE_STATUS, help_text = "Only entries with live status will be
> publicly displayed.")
>
>     # Categorization.
>     categories = models.ManyToManyField(Category)
>     tags = TagField(help_text = "Separate tags with spaces.")
>
>     class Meta:
>         verbose_name_plural = "Entries"
>         ordering = ['-pub_date']
>
>     def __unicode__(self):
>         return self.title
>
>     def save(self, force_insert=False, force_update=False):
>         self.body_html = markdown(self.body)
>         if self.excerpt:
>             self.excerpt_html = markdown(self.excerpt)
>             super(Entry, self).save(force_insert, force_update)
>
>     def get_absolute_url(self):
>         return "/weblog/%s/%s/" %
> (self.pub_date.strftime("%Y/%b/%d").lower(), self.slug)
>
> ///
>
> Here is the views.py
>
> //
>
> from django.shortcuts import render_to_response
>
> from cmswithpythonanddj

Re: Django database-api

[akaariai]

On Apr 3, 11:27 am, KasunLak  wrote:
> Hi,
>
> No, sorry for not giving more details. What I am asking is once we
> syncdb the generated scripts (create table). Can we see the method
> implementation of database access api?
>
> For example: If we have created and deployed a model called Poll
> (tutorial example in django site)
>
> p = Poll.objects.get(pk=1).
>
> Can we see the implementation of this get(pk) method? I think these
> are some objects in the database, just wonder this is possible?

If you mean what SQL it generates there are two methods inbuilt to
django. If you have settings.DEBUG = True, you can use:
> p = Poll.objects.get(pk=1).
> django.db import connection
> print connection.queries[-1]

Another option:
> print Poll.objects.filter(pk=1).query
This one does work only for QuerySet, so if the method you are
interested in executes the query immediately (like .get()), then you
can't use this method.

Neither of the above will give you correctly quoted querystring, as
that is simply not available to Django. The database backend libraries
do the quoting, and Django can't get the quoted query string back from
the libraries.

If you want to see the Python code implementation, look at django/db/
models/query.py, which uses django/db/models/sql/query.py. Be warned:
the sql/query.py will take some time to understand, it is one of the
most complex pieces of code in Django.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Filtering results before & after update?

[akaariai]

On Apr 1, 4:24 am, symmitchry  wrote:
> I have a straightforward form that simply reduces an item count by 1, for a
> selected product. (The form essentially "ships" a product, so need to
> decrease remaining term_length by 1).
>
> I have the following code:
>
>             cd = form.cleaned_data
>
>             q = Subscription.objects.filter(product_key__exact =
> cd['product'])
>             q = q.filter(status__iexact='Active')
>             q = q.filter(term_length__gt=0)
>
>             # Decrement our terms remaining by 1.
>             rec_count = q.update(term_length=F('term_length') - 1)
>
>             return render_to_response('subs/shipped.html',{ 'q':
> q, 'rec_count': rec_count })
>
> What happens is that the results I want to display on the "shipped" page
> are still being filtered according to the criteria I used to select the
> rows for updating. Since I have a >= 0 filter, I do not see those rows that
> were reduced from 1 to 0. I can't use {{ q|length }} to get number of
> records updated, for example.
>
> How would I go about returning, for example, a list of the freshly updated
> records from BEFORE they were updated. I can't search after the fact, since
> status and term_length would be Inactive and 0 for a huge number of
> records; there will be nothing distinct about the ones that were last
> changed.
>
> Thanks for the help.
>
> M.

It seems your best bet is to do something like this:

to_be_updated = list(q.select_for_update())
q.filter(pk__in=to_be_updated).update(...)

The above is written in a way trying to guarantee that the
to_be_updated set is exactly the same as the actual set you are going
to update. However, you need to fetch the objects from the database
which can be costly. If you don't want to do that, you could fetch
just the PK list or add another column to the update discriminating
the set (modified_by=someid). It could be useful if you could directly
get the updated PK list from .update(), but that is not supported by
the ORM.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.