Admin - Selector Inlines in 1.3?
Hi, As far as I know this is the only major chunk of Zain's SOC work that didn't make it into 1.2 There's a ticket for it: http://code.djangoproject.com/ticket/12509 that's at 'design decision needed' What would be needed to get this rather nice feature into 1.3? cheers, Andy -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
importing fixtures to postgres fails to set sequences correctly
Hi guys,
there seems to be a problem with django postgres backend, when
importing data from fixtures. Data are imported correctly, but the
sequences for primary keys are set incorrecly on models that have
generic.GenericRelation field. See example:
As a demo, those are just two models with generic relation.
models.py:
-
from django.db import models
from django.contrib.contenttypes import generic
from django.contrib.contenttypes.models import ContentType
class Tag(models.Model):
name = models.CharField(max_length=30)
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = generic.GenericForeignKey('content_type', 'object_id')
class Post(models.Model):
name = models.CharField(max_length=30)
text = models.TextField()
tags = generic.GenericRelation('blog.Tag')
The loaddata management command calls at the end
connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql command
to update sq1 sequences. Let's see the result:
./manage shell
---
In [1]: from django.db import DEFAULT_DB_ALIAS, connections
In [2]: from django.core.management.color import no_style
In [3]: from proj.blog.models import Post
In [4]: connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql(no_style(), [Post])
Out[4]:
['SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_post";',
'SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_tag";']
As you can see, the problem is in the last "SELECT". Postgres backend
"thinks" that Post.tags is m2m relation with usual m2m sql table and
tries to update it's pk sequence. The table is of course non existant
and it resets Post.pk sequence to max(Tag.pk), this is obviously
incorrect behaviour and results with potential DatabaseErrors
duplicate key on blog_post table.
Removing Post.tags and accessing related tags directly works as a
workaround, but not very nice and comfotable one..
Do you agree that I'll start a ticket on that?
I was looking to the code to make a patch, but since the error is
caused by "magic" m2m field in application in contrib, I hasitate to
make postgres backend "aware" of any generic.GenericRelation fields.
On the other hand, I'm not sure it's possible to edit just
GenericRelation class to fix that.
Regards,
Ales
--
Ales Zoulek
+420 604 332 515
Jabber: ales.zou...@gmail.com
--
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: importing fixtures to postgres fails to set sequences correctly
On Thu, Jul 15, 2010 at 6:09 PM, Ales Zoulek wrote:
> Hi guys,
>
> there seems to be a problem with django postgres backend, when
> importing data from fixtures. Data are imported correctly, but the
> sequences for primary keys are set incorrecly on models that have
> generic.GenericRelation field. See example:
>
> As a demo, those are just two models with generic relation.
>
> models.py:
> -
> from django.db import models
> from django.contrib.contenttypes import generic
> from django.contrib.contenttypes.models import ContentType
>
>
> class Tag(models.Model):
> name = models.CharField(max_length=30)
> content_type = models.ForeignKey(ContentType)
> object_id = models.PositiveIntegerField()
> content_object = generic.GenericForeignKey('content_type', 'object_id')
>
>
> class Post(models.Model):
> name = models.CharField(max_length=30)
> text = models.TextField()
> tags = generic.GenericRelation('blog.Tag')
>
>
> The loaddata management command calls at the end
>
> connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql command
>
> to update sq1 sequences. Let's see the result:
>
> ./manage shell
> ---
> In [1]: from django.db import DEFAULT_DB_ALIAS, connections
> In [2]: from django.core.management.color import no_style
> In [3]: from proj.blog.models import Post
> In [4]: connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql(no_style(),
> [Post])
> Out[4]:
> ['SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
> coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_post";',
> 'SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
> coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_tag";']
>
>
> As you can see, the problem is in the last "SELECT". Postgres backend
> "thinks" that Post.tags is m2m relation with usual m2m sql table and
> tries to update it's pk sequence. The table is of course non existant
> and it resets Post.pk sequence to max(Tag.pk), this is obviously
> incorrect behaviour and results with potential DatabaseErrors
> duplicate key on blog_post table.
> Removing Post.tags and accessing related tags directly works as a
> workaround, but not very nice and comfotable one..
>
>
>
> Do you agree that I'll start a ticket on that?
I haven't physically run the same code, but what you describe
certainly sounds plausible. Generic relations are a common edge case
that gets forgotten. On top of that, sequence resets on Postgres have
changed recently (post Django 1.2-final), so it's entirely possible
that a bug has slipped in.
A ticket would be most appreciated.
> I was looking to the code to make a patch, but since the error is
> caused by "magic" m2m field in application in contrib, I hasitate to
> make postgres backend "aware" of any generic.GenericRelation fields.
> On the other hand, I'm not sure it's possible to edit just
> GenericRelation class to fix that.
You shouldn't need to make the PostgreSQL backend aware of the
existence of GenericRelation. Fields contain a 'serialize' option that
is used to determine if a relation should be serialized as part of a
fixture. Not deserializing GenericRelations is the primary reason for
the existence of this setting. I'd need to check to be certain, but I
think the reason that GenericRelations can't be serialized will turn
out to be exactly the same as the reason they have their sequences
reset.
If it turns out that this isn't the case, I have no problem adding
another flag that *does* identify the right use case (i.e., not
"is_generic_relation", but "requires_sequence_reset", or something
like it).
Of related interest would be to check if this issue existed previously
(i.e., before the recent PostgreSQL sequence changes); if the issue
didn't exist in Django 1.2 final, then checking the diff between the
old and new implementations may also shed some light on potential
solutions.
Yours,
Russ Magee %-)
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: importing fixtures to postgres fails to set sequences correctly
Okay,
ticket created as #13941
I'll try to find out more and let you know in this thread.
Thanks,
A.
--
Ales Zoulek
+420 604 332 515
Jabber: ales.zou...@gmail.com
--
On Thu, Jul 15, 2010 at 1:40 PM, Russell Keith-Magee
wrote:
> On Thu, Jul 15, 2010 at 6:09 PM, Ales Zoulek wrote:
>> Hi guys,
>>
>> there seems to be a problem with django postgres backend, when
>> importing data from fixtures. Data are imported correctly, but the
>> sequences for primary keys are set incorrecly on models that have
>> generic.GenericRelation field. See example:
>>
>> As a demo, those are just two models with generic relation.
>>
>> models.py:
>> -
>> from django.db import models
>> from django.contrib.contenttypes import generic
>> from django.contrib.contenttypes.models import ContentType
>>
>>
>> class Tag(models.Model):
>> name = models.CharField(max_length=30)
>> content_type = models.ForeignKey(ContentType)
>> object_id = models.PositiveIntegerField()
>> content_object = generic.GenericForeignKey('content_type', 'object_id')
>>
>>
>> class Post(models.Model):
>> name = models.CharField(max_length=30)
>> text = models.TextField()
>> tags = generic.GenericRelation('blog.Tag')
>>
>>
>> The loaddata management command calls at the end
>>
>> connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql command
>>
>> to update sq1 sequences. Let's see the result:
>>
>> ./manage shell
>> ---
>> In [1]: from django.db import DEFAULT_DB_ALIAS, connections
>> In [2]: from django.core.management.color import no_style
>> In [3]: from proj.blog.models import Post
>> In [4]: connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql(no_style(),
>> [Post])
>> Out[4]:
>> ['SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
>> coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_post";',
>> 'SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
>> coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_tag";']
>>
>>
>> As you can see, the problem is in the last "SELECT". Postgres backend
>> "thinks" that Post.tags is m2m relation with usual m2m sql table and
>> tries to update it's pk sequence. The table is of course non existant
>> and it resets Post.pk sequence to max(Tag.pk), this is obviously
>> incorrect behaviour and results with potential DatabaseErrors
>> duplicate key on blog_post table.
>> Removing Post.tags and accessing related tags directly works as a
>> workaround, but not very nice and comfotable one..
>>
>>
>>
>> Do you agree that I'll start a ticket on that?
>
> I haven't physically run the same code, but what you describe
> certainly sounds plausible. Generic relations are a common edge case
> that gets forgotten. On top of that, sequence resets on Postgres have
> changed recently (post Django 1.2-final), so it's entirely possible
> that a bug has slipped in.
>
> A ticket would be most appreciated.
>
>> I was looking to the code to make a patch, but since the error is
>> caused by "magic" m2m field in application in contrib, I hasitate to
>> make postgres backend "aware" of any generic.GenericRelation fields.
>> On the other hand, I'm not sure it's possible to edit just
>> GenericRelation class to fix that.
>
> You shouldn't need to make the PostgreSQL backend aware of the
> existence of GenericRelation. Fields contain a 'serialize' option that
> is used to determine if a relation should be serialized as part of a
> fixture. Not deserializing GenericRelations is the primary reason for
> the existence of this setting. I'd need to check to be certain, but I
> think the reason that GenericRelations can't be serialized will turn
> out to be exactly the same as the reason they have their sequences
> reset.
>
> If it turns out that this isn't the case, I have no problem adding
> another flag that *does* identify the right use case (i.e., not
> "is_generic_relation", but "requires_sequence_reset", or something
> like it).
>
> Of related interest would be to check if this issue existed previously
> (i.e., before the recent PostgreSQL sequence changes); if the issue
> didn't exist in Django 1.2 final, then checking the diff between the
> old and new implementations may also shed some light on potential
> solutions.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-develo
Re: importing fixtures to postgres fails to set sequences correctly
Okay, so after a little bit more investigation on that topic, I've found out:
1] It's changed since changeset 13328
(http://code.djangoproject.com/changeset/13328)
2] Previous versions DID reset the sequence thru GenericRelation
fields, but it did it on the right table so it worked. It just called
useless code. In our example, the second query would set blog_tag
sequence to max(blog_tag.id)
3] I've created a patch and a test, that keeps the older behaviour.
Please see the ticket.
4] We can consider find a way how to skip the second sql query altogether.
A.
--
Ales Zoulek
+420 604 332 515
Jabber: ales.zou...@gmail.com
--
On Thu, Jul 15, 2010 at 2:01 PM, Ales Zoulek wrote:
> Okay,
>
> ticket created as #13941
>
> I'll try to find out more and let you know in this thread.
>
> Thanks,
>
> A.
> --
> Ales Zoulek
> +420 604 332 515
> Jabber: ales.zou...@gmail.com
> --
>
>
>
> On Thu, Jul 15, 2010 at 1:40 PM, Russell Keith-Magee
> wrote:
>> On Thu, Jul 15, 2010 at 6:09 PM, Ales Zoulek wrote:
>>> Hi guys,
>>>
>>> there seems to be a problem with django postgres backend, when
>>> importing data from fixtures. Data are imported correctly, but the
>>> sequences for primary keys are set incorrecly on models that have
>>> generic.GenericRelation field. See example:
>>>
>>> As a demo, those are just two models with generic relation.
>>>
>>> models.py:
>>> -
>>> from django.db import models
>>> from django.contrib.contenttypes import generic
>>> from django.contrib.contenttypes.models import ContentType
>>>
>>>
>>> class Tag(models.Model):
>>> name = models.CharField(max_length=30)
>>> content_type = models.ForeignKey(ContentType)
>>> object_id = models.PositiveIntegerField()
>>> content_object = generic.GenericForeignKey('content_type', 'object_id')
>>>
>>>
>>> class Post(models.Model):
>>> name = models.CharField(max_length=30)
>>> text = models.TextField()
>>> tags = generic.GenericRelation('blog.Tag')
>>>
>>>
>>> The loaddata management command calls at the end
>>>
>>> connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql command
>>>
>>> to update sq1 sequences. Let's see the result:
>>>
>>> ./manage shell
>>> ---
>>> In [1]: from django.db import DEFAULT_DB_ALIAS, connections
>>> In [2]: from django.core.management.color import no_style
>>> In [3]: from proj.blog.models import Post
>>> In [4]: connections[DEFAULT_DB_ALIAS].ops.sequence_reset_sql(no_style(),
>>> [Post])
>>> Out[4]:
>>> ['SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
>>> coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_post";',
>>> 'SELECT setval(pg_get_serial_sequence(\'blog_post\',\'id\'),
>>> coalesce(max("id"), 1), max("id") IS NOT null) FROM "blog_tag";']
>>>
>>>
>>> As you can see, the problem is in the last "SELECT". Postgres backend
>>> "thinks" that Post.tags is m2m relation with usual m2m sql table and
>>> tries to update it's pk sequence. The table is of course non existant
>>> and it resets Post.pk sequence to max(Tag.pk), this is obviously
>>> incorrect behaviour and results with potential DatabaseErrors
>>> duplicate key on blog_post table.
>>> Removing Post.tags and accessing related tags directly works as a
>>> workaround, but not very nice and comfotable one..
>>>
>>>
>>>
>>> Do you agree that I'll start a ticket on that?
>>
>> I haven't physically run the same code, but what you describe
>> certainly sounds plausible. Generic relations are a common edge case
>> that gets forgotten. On top of that, sequence resets on Postgres have
>> changed recently (post Django 1.2-final), so it's entirely possible
>> that a bug has slipped in.
>>
>> A ticket would be most appreciated.
>>
>>> I was looking to the code to make a patch, but since the error is
>>> caused by "magic" m2m field in application in contrib, I hasitate to
>>> make postgres backend "aware" of any generic.GenericRelation fields.
>>> On the other hand, I'm not sure it's possible to edit just
>>> GenericRelation class to fix that.
>>
>> You shouldn't need to make the PostgreSQL backend aware of the
>> existence of GenericRelation. Fields contain a 'serialize' option that
>> is used to determine if a relation should be serialized as part of a
>> fixture. Not deserializing GenericRelations is the primary reason for
>> the existence of this setting. I'd need to check to be certain, but I
>> think the reason that GenericRelations can't be serialized will turn
>> out to be exactly the same as the reason they have their sequences
>> reset.
>>
>> If it turns out that this isn't the case, I have no problem adding
>> another flag that *does* identify the right use case (i.e., not
>> "is_generic_relation", but "requires_sequence_reset", or something
>> like it).
>>
>> Of related interest would be to ch
Invalid SQL generated by objects.all()[:1]?
Hi, I've just been informed by a MonetDB developer that LIMIT and ORDER BY clauses are not allowed by the SQL spec in sub-selects. I filied a bug with them because the Django code: Event.objects.all()[:1] was failing because of invalid SQL. I don't have access to the SQL spec so I can't quote you verse and chapter. Has this come up before? How can I workaround this in the adapter? Thanks, m -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Invalid SQL generated by objects.all()[:1]?
On Thu, Jul 15, 2010 at 1:00 PM, Mark Bucciarelli wrote: > Hi, > > I've just been informed by a MonetDB developer > that LIMIT and ORDER BY clauses are not > allowed by the SQL spec in sub-selects. > > I filied a bug with them because the Django > code: > > Event.objects.all()[:1] > > was failing because of invalid SQL. > > I don't have access to the SQL spec so I > can't quote you verse and chapter. > > Has this come up before? > > How can I workaround this in the adapter? > > Thanks, > > m > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > It may be that they aren't allowed by the spec, but the only database (that Django includes an adapter for) that doesn't support them (AFAIK) is MySQL. Further, Event.objects.all()[:1] doesn't generate a subquery at all. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Invalid SQL generated by objects.all()[:1]?
On Thu, Jul 15, 2010 at 2:04 PM, Alex Gaynor wrote: > > It may be that they aren't allowed by the spec, but the only database > (that Django includes an adapter for) that doesn't support them > (AFAIK) is MySQL. > Cool, so that adapter has a workaround? > > Further, Event.objects.all()[:1] doesn't generate a > subquery at all. > Hmm, well something in Django did. Doesn't matter, really. Or are you saying Django shouldn't generate invalid SQL? Thanks, m -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Invalid SQL generated by objects.all()[:1]?
On Thu, Jul 15, 2010 at 1:14 PM, Mark Bucciarelli wrote: > On Thu, Jul 15, 2010 at 2:04 PM, Alex Gaynor wrote: >> >> It may be that they aren't allowed by the spec, but the only database >> (that Django includes an adapter for) that doesn't support them >> (AFAIK) is MySQL. >> > > Cool, so that adapter has a workaround? > >> >> Further, Event.objects.all()[:1] doesn't generate a >> subquery at all. >> > > Hmm, well something in Django did. Doesn't matter, > really. > > Or are you saying Django shouldn't generate invalid > SQL? > > Thanks, > > m > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > I don't think there's a workaround, I'm pretty sure MySQL will just give you an error if you try to do that. And yes, if what you say about the SQL spec is true, then Django will emit semantically invalid SQL. It's also emitting SQL that can be perfectly understood by a lot of databases. Despite SQL being a spec most DBs implement it slightly differently, that's why we have different backends for different DBs. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Invalid SQL generated by objects.all()[:1]?
On Thu, Jul 15, 2010 at 2:18 PM, Alex Gaynor wrote: > > And yes, if what you say about the SQL spec is true, then Django will > emit semantically invalid SQL. It's also emitting SQL that can be > perfectly understood by a lot of databases. Despite SQL being a spec > most DBs implement it slightly differently, that's why we have > different backends for different DBs. > embrace and extend. hey, it's worked for others ... bwah hah hah. ;) huh, regarding no workaround. i have a cursor wrapper, i guess i can use a regex in execute(). not sure why a subselect would ever need an order by anyway. i'll have to figure out exactly what django operation generated that sql. thanks for your quick response. m -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Invalid SQL generated by objects.all()[:1]?
On 07/15/2010 01:28 PM, Mark Bucciarelli wrote: not sure why a subselect would ever need an order by anyway. i'll have to figure out exactly what django operation generated that sql. The only time I've needed an ORDER BY in a subselect involved a LIMIT/TOP in the subselect SELECT ... FROM tblFoo WHERE StatementID IN ( SELECT -- TOP 5 -- SQL Server id FROM tblStatements ORDER BY StatementDate Desc LIMIT 5 -- just about everybody else uses LIMIT ) which has -tkc -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: importing fixtures to postgres fails to set sequences correctly
> > there seems to be a problem with django postgres backend, when > > importing data from fixtures. Data are imported correctly, but the > > sequences for primary keys are set incorrecly on models that have > > generic.GenericRelation field. See example: I have similar code, and was griping in #django-dev yesterday about the regression. I came up with an almost identical fix (using `m2m_db_table`), so I've accepted the ticket. > Of related interest would be to check if this issue existed previously > (i.e., before the recent PostgreSQL sequence changes); if the issue > didn't exist in Django 1.2 final, then checking the diff between the > old and new implementations may also shed some light on potential > solutions. It's definitely a side-effect of r13328 -- I have production code that works fine on 1.2.X, but crashes on trunk without the patch when restoring fixtures. My only question is whether there was an original reason that `db_table` was used or if it was simply a copy & paste oversight from the previous loop. -Justin -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Accessible Command Output using self.stdout & self.stderr
Related to the issue with stdout/stderr: It is ofcourse possible to have wrapper in the Command class for stdout and stderr output that redirect all data to system's stdout in most cases. But this might not work for all output since we don't have control about all management commands of thirdparty apps. A more "secure" approach of capturing output is to overwrite `sys.stdout` and `sys.stderr` with your own file like objects. A short example: import sys class MyStdout(object): def __init__(self): self.data = '' def write(self, s): self.data += s myout = MyStdout() sys.stdout = myout print "Hello, World!" assert myout.data == "Hello, World!\n" This is not as clean as the self.stdout approach but will capture *all* output instead of only the ouput of commands that will support the new API - which might be what you want if you provide a webinterface. I'm aware that Russ is not a big fan of Monkeypatching :) but I think this would be a legal case - since there is no other way of intercepting the real stdout output (I'm not sure but this might be even documented in python docs, that its ok to overwrite sys.stdout and sys.stderr with your own objects). Just want to point out that its possible - not that its the only way to go :) Gregor 2010/7/13 Russell Keith-Magee : > On Tue, Jul 13, 2010 at 2:23 PM, maxweld wrote: >> I would like to propose that all commands in core.management be >> modified to enable command output to be directed to self.stdout and >> self.stderr so that the command output can be captured and made >> accessible to an application. >> >> I use a virtual web hosting environment where no shell access is >> permitted. I would like to set up an administrative web interface >> which enables commands to be executed and the command output >> redisplayed to the user. However, as commands currently send output >> directly to stdout or stderr, the comand output is not readily >> accessible (Django 1.2.1). >> >> Discussions with Russ McGee on Django Users suggested that some > > Magee. I'm not Scottish :-) > >> commands, for testing system purposes, are now sending to self.stdout >> and self.stderr, allowing the target to be redefined and the output to >> be captured. Only dumpdata and loaddata commands have currently been >> converted in trunk. I am proposing extending this to all commands in >> core.management, and suggest that this should be promoted as the norm >> for custom developed commands also. >> >> If agreed, the work required to implement would be: >> a) convert existing commands >> b) document the feature >> c) improve documentation of the existing feature allowing applications >> to construct their own commands >> d) improve documentation of how to execute commands programatically >> (as opposed to in a shell) > > Seconded, and carried. I've already said I'll accept any patch that > implements any (or preferably all) of these points. > >> If there is agreement, I am happy to undertake the above although >> would appreciate a mentor being available for advice from time to time >> as this would be my first contribution. > > I'm happy to give you any feedback you need; if you post to django-dev > when you need assistance or feedback, I'll try to respond; if I don't, > someone else hopefully will. > >> Longer term, if an administrative web interface to commands is to be >> provided, we would need to devise an interface that adapted well into >> the existing administrative interface. However, I see this as a second >> stage in the process. First we need the command output to be >> available. > > There may well be a place for these commands to be exposed via a web > interface (especially if you're building a shared hosting type > environment), but I'm not convinced that this is something that is > appropriate to be baked into Django's core. > > Firstly, it's a bit of an edge case; the biggest use case for this > sort of facility is to support a virtual server for which you don't > have shell access. I don't have any firm numbers, but I suspect that > this is an edge case of Django deployments. > > Secondly, some management commands aren't really well suited to > execute on the same server that is serving them. It may be fine to > have one server executing management commands on another, though > (especially if you're setting up some sort of push-button hosting > environment). > > Thirdly, some management commands can be time consuming or have the > potential for introducing database locks; as such, they aren't well > suited to execution in the request/response cycle. This means you > should be looking at implementing this using a job queue, and Django > is agnostic when it comes to such issues. > > That said, if such an administrative interface were to be introduced, > it would almost certainly be as a contrib application. The usual path > for adding new contrib applications is for them to start as a > standalone projec
Re: Invalid SQL generated by objects.all()[:1]?
On Thu, Jul 15, 2010 at 12:04 PM, Alex Gaynor wrote: > On Thu, Jul 15, 2010 at 1:00 PM, Mark Bucciarelli wrote: >> Hi, >> >> I've just been informed by a MonetDB developer >> that LIMIT and ORDER BY clauses are not >> allowed by the SQL spec in sub-selects. >> >> I filied a bug with them because the Django >> code: >> >> Event.objects.all()[:1] >> >> was failing because of invalid SQL. >> >> I don't have access to the SQL spec so I >> can't quote you verse and chapter. >> >> Has this come up before? >> >> How can I workaround this in the adapter? >> >> Thanks, >> >> m >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers" group. >> To post to this group, send email to django-develop...@googlegroups.com. >> To unsubscribe from this group, send email to >> django-developers+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-developers?hl=en. >> >> > > It may be that they aren't allowed by the spec, but the only database > (that Django includes an adapter for) that doesn't support them > (AFAIK) is MySQL. Further, Event.objects.all()[:1] doesn't generate a > subquery at all. Er, I'm pretty sure that MySQL does support both ORDER BY and LIMIT/OFFSET. At least, it is documented as supporting them. The supported database that does have a problem with LIMIT/OFFSET is Oracle. The backend contains a workaround, which is implemented by a custom SQLCompiler in django.db.backends.oracle.compiler, and it involves wrapping the original query in a subquery. The MonetDB backend may be doing something similar, which would explain why that query is generating a subquery. All of the internally supported databases support ORDER BY, so I can't offer any hints there. Ian -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Invalid SQL generated by objects.all()[:1]?
On Thu, Jul 15, 2010 at 5:57 PM, Ian Kelly wrote: > > Er, I'm pretty sure that MySQL does support both ORDER BY and > LIMIT/OFFSET. At least, it is documented as supporting them. > Right you are. There are few restrictions on the type of statements in which subqueries can be used. A subquery can contain many of the keywords or clauses that an ordinary SELECT can contain: DISTINCT, GROUP BY, ORDER BY, LIMIT, joins, index hints, UNION constructs, comments, functions, and so on. ref: http://dev.mysql.com/doc/refman/4.1/en/subqueries.html > > The supported database that does have a problem with LIMIT/OFFSET is > Oracle. The backend contains a workaround, which is implemented by a > custom SQLCompiler in django.db.backends.oracle.compiler, and it > involves wrapping the original query in a subquery. > Ouch. Ok, so it's possible to handle this in the adapter. > > The MonetDB > backend may be doing something similar, which would explain why that > query is generating a subquery. > I don't think so. I'm maintaining the adapter, such as it is [1]. I need to do some homework on this. Thanks, m [1] http://github.com/mbucc/monetdb-python -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Accessible Command Output using self.stdout & self.stderr
On Fri, Jul 16, 2010 at 5:39 AM, Gregor Müllegger wrote: > Related to the issue with stdout/stderr: > > I'm aware that Russ is not a big fan of Monkeypatching :) True :-) > but I think this > would be a legal case - since there is no other way of intercepting the real > stdout output (I'm not sure but this might be even documented in python docs, > that its ok to overwrite sys.stdout and sys.stderr with your own objects). If you want to get really technical, there *is* another way it can be done -- at least, if you're on a *NIX; you can redirect the file descriptors that represent stdout/stderr. It's low-level plumbing, but it works. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: importing fixtures to postgres fails to set sequences correctly
On Thu, Jul 15, 2010 at 6:05 PM, Justin Bronn wrote: >> > there seems to be a problem with django postgres backend, when >> > importing data from fixtures. Data are imported correctly, but the >> > sequences for primary keys are set incorrecly on models that have >> > generic.GenericRelation field. See example: > > I have similar code, and was griping in #django-dev yesterday about > the regression. I came up with an almost identical fix (using > `m2m_db_table`), so I've accepted the ticket. > >> Of related interest would be to check if this issue existed previously >> (i.e., before the recent PostgreSQL sequence changes); if the issue >> didn't exist in Django 1.2 final, then checking the diff between the >> old and new implementations may also shed some light on potential >> solutions. > > It's definitely a side-effect of r13328 -- I have production code that > works fine on 1.2.X, but crashes on trunk without the patch when > restoring fixtures. My only question is whether there was an original > reason that `db_table` was used or if it was simply a copy & paste > oversight from the previous loop. > I think so. And now I remember actually we discussed about this on #django-dev a couple of weeks ago but I forgot to open a ticket: http://botland.oebfare.com/logger/django-dev/2010/6/23/1/#02:04-2311026 Regards, -- Ramiro Morales | http://rmorales.net -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: MySQL index hints
Simon --
Slicing result sets is clearly the developer's responsibility IMHO.
Even with hard limits, MySQL still fails miserably in many situations;
hence the proposal to support hinting from the ORM.
Russ --
Firstly, re namespacing. No worries, let's just keep it RDBMS-
specific, ie --
MySQL - with_hints(use={}, force={}, ignore={})
Oracle - with_hints(index={}, ...)
That gives plenty of name space to breathe in.
Next --
> Using the filter/exclude name sounds interesting, but skips the first
> step of putting an index hint on the origin table. For example,
> Book.objects.filter(author__name='foo') -- now put an index hint on
> Book; now put one on Author. If Author and Book are an a m2m relation
> -- put a hint on the m2m table.
How about --
{'book': 'book_idx', 'author': 'author_idx', 'author__name':
'm2m_idx'}
Or, if you want to cover absolutely every possibility --
{'book': 'book_idx', 'author__name__author': 'author_idx',
'author__name': 'm2m_idx'}
That is, require indexes on the right side of joins to be named
explicitly, just in case Author is used in another join.
Sounds complicated, but obviously edge case. Common usage will be
simple, eg {'book':'book_idx'}.
Let me know what you think.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
