Re: App Engine port
On Tue, 2009-02-17 at 02:58 -0800, Waldemar Kornewald wrote: > No comments on the more efficient perms/contenttypes implementation? If you were hoping for something from me, I don't have any time to think about it. I'm not in the business of predicting the future in a general sense, beyond things I've thought hard about and are working on directly. Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
No comments on the more efficient perms/contenttypes implementation? I'd really like to know if that is an option because if it's not I should better make app-engine-patch 1.0 compatible with the less efficient implementation. Hmm, or I could emulate those db tables in the backend, but that wouldn't be very nice. Bye, Waldemar Kornewald --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On 14 Feb., 03:08, Malcolm Tredinnick wrote: > On Thu, 2009-02-12 at 02:14 -0800, Waldemar Kornewald wrote: > > On 12 Feb., 03:51, Malcolm Tredinnick > > wrote: > > > > Is the plan somewhere on the wiki? > > > > No, because it's only something I'm pulling together slowly in my head. > > > I hope you've read my wiki page, so that in the end Django is flexible > > enough to work even with App Engine's limitations (e.g., it's > > practically impossible to implement a TransactionMiddleware): > >http://code.djangoproject.com/wiki/AppEngine > > I've had a quick read of it, but nothing serious, since I have a million > other things to do at the moment that are higher priority. However, the > overall impression from reading that is that most of those problems > aren't something Django itself will need to worry about. It's things > that the Query (and other django/db/models/sql/*) replacement will need > to handle. Yes, for most problems that's correct. Most important changes required on the Django side: * many-to-many relations need to be handled more flexibly (could be stored as an attribute on some model) * batch save/delete (at least very simple/naive implementations) must be supported (otherwise efficient websites are impossible with App Engine) > Ditto with things like replacing Permissions and ContentTypes -- that > will be up to you to fake somehow in the eventual AppEngine code. > They're part of Django, so if you choose not to use them (and I really > don't understand why you wouldn't, since they're just normal models -- > the argument seems to be that you don't want to have an equivalent of > syncdb for some reason, but that doesn't necessarily seem to be a > required constraint), you have to work around that. Regarding syncdb in general: Since you don't have shell access to the server, something like syncdb would have to be emulated with a view that is remotely accessible or using the provider's background processing facility. Unfortunately, you might hit a request limitation if you run syncdb in a single request (also applies to background processing), so syncdb should be spread across multiple requests, but it's not designed to do that and the implementation wouldn't be easy. In general, whenever syncdb isn't absolutely necessary it shouldn't be used. Regarding faked Permission/ContentType models: We could probably create all entities within a single request (up to 500 individual write operations in a single request should be doable). My point, though, is that faked models would be more efficient and much simpler (e.g., auth backend's get_group_permissions() could be reduced from 3 queries plus one client-side JOIN to 1 query without JOINs on App Engine). Also, installations/upgrades are easier if you don't need syncdb and everything magically just works. This change might not be an option for SQL, but couldn't Django at least provide a more efficient alternative for cloud backends? > > SimpleDB is even worse because of "eventual consistency" which means > > there are no transactions and absolutely no integrity guarantees. > > No, it's easier. Since there are no constraints of that nature, it's not > something we have to worry. Designing in features that are impossible > isn't a good goal to have. If you're using a storage backend that > doesn't support some kind of transactional integrity, then you simply > don't have transactional integrity. That's a constraint you accept when > choosing that storage backend. I wasn't suggesting to emulate transactions somehow, but pointing out that some backends might have very restrictive limitations and that it might be difficult to handle that in a general way. In the end, the API might look similar, but depending on the backend the semantics could be ridiculously different. That's still better than no common API, though. Bye, Waldemar Kornewald --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
Seems like my message didn't reach the server. Sending again. Sorry if you receive this twice. On 14 Feb., 03:08, Malcolm Tredinnick wrote: > On Thu, 2009-02-12 at 02:14 -0800, Waldemar Kornewald wrote: > > On 12 Feb., 03:51, Malcolm Tredinnick > > wrote: > > > > Is the plan somewhere on the wiki? > > > > No, because it's only something I'm pulling together slowly in my head. > > > I hope you've read my wiki page, so that in the end Django is flexible > > enough to work even with App Engine's limitations (e.g., it's > > practically impossible to implement a TransactionMiddleware): > >http://code.djangoproject.com/wiki/AppEngine > > I've had a quick read of it, but nothing serious, since I have a million > other things to do at the moment that are higher priority. However, the > overall impression from reading that is that most of those problems > aren't something Django itself will need to worry about. It's things > that the Query (and other django/db/models/sql/*) replacement will need > to handle. Yes, for most problems that's correct. Most important changes required on the Django side: * many-to-many relations need to be handled more flexibly (could be stored as an attribute on some model) * batch save/delete (at least very simple/naive implementations) must be supported (otherwise efficient websites are impossible with App Engine) > Ditto with things like replacing Permissions and ContentTypes -- that > will be up to you to fake somehow in the eventual AppEngine code. > They're part of Django, so if you choose not to use them (and I really > don't understand why you wouldn't, since they're just normal models -- > the argument seems to be that you don't want to have an equivalent of > syncdb for some reason, but that doesn't necessarily seem to be a > required constraint), you have to work around that. Regarding syncdb in general: Since you don't have shell access to the server, something like syncdb would have to be emulated with a view that is remotely accessible or using the provider's background processing facility. Unfortunately, you might hit a request limitation if you run syncdb in a single request (also applies to background processing), so syncdb should be spread across multiple requests, but it's not designed to do that and the implementation wouldn't be easy. In general, whenever syncdb isn't absolutely necessary it shouldn't be used. Regarding faked Permission/ContentType models: We could probably create all entities within a single request (up to 500 individual write operations in a single request should be doable). My point, though, is that faked models would be more efficient and much simpler (e.g., auth backend's get_group_permissions() could be reduced from 3 queries plus one client-side JOIN to 1 query without JOINs on App Engine). Also, installations/upgrades are easier if you don't need syncdb and everything magically just works. This change might not be an option for SQL, but couldn't Django at least provide a more efficient alternative for cloud backends? > > SimpleDB is even worse because of "eventual consistency" which means > > there are no transactions and absolutely no integrity guarantees. > > No, it's easier. Since there are no constraints of that nature, it's not > something we have to worry. Designing in features that are impossible > isn't a good goal to have. If you're using a storage backend that > doesn't support some kind of transactional integrity, then you simply > don't have transactional integrity. That's a constraint you accept when > choosing that storage backend. I wasn't suggesting to emulate transactions somehow, but pointing out that some backends might have very restrictive limitations and that it might be difficult to handle that in a general way. In the end, the API might look similar, but depending on the backend the semantics could be ridiculously different. That's still better than no common API, though. Bye, Waldemar Kornewald --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
Basic demand side problem. When an app stands up and solves the real issues (which, as the DjangoCon presentation proved, there were many) involving App Engine and Django, then I (and the demographic of developers I statistically represent) will become excited. -S On Feb 6, 9:51 am, David Stenglein wrote: > If so, why the seeming lack of excitement around App Engine? Are > google engineers not involved in the community? > > -Dave > > On Fri, Feb 6, 2009 at 7:10 AM, Russell Keith-Magee > > wrote: > > > On Fri, Feb 6, 2009 at 8:32 PM, Waldemar Kornewald > > wrote: > > >> Hi Russell, > > >> On 6 Feb., 11:34, Russell Keith-Magee wrote: > >>> I would suggest to you that the broader project of "modifying the > >>> django.db.models interface to be fully independent of SQL" is much > >>> more likely to get core developer support. We (the Django core) will > >>> be very receptive to any suggestions on how django.db.models needs to > >>> be modified in order to support non-SQL backends -- especially if you > >>> can demonstrate that your suggestions aren't just theoretical, but are > >>> clearly required by an actual backend implementation. > > >> Agreed, making models SQL-independent is the most important (and most > >> complicated) part. Do the people secretly working on App Engine > >> integration try to implement that part? > > > I haven't seen their code, so I can't say for certain, but I imagine > > that they have a small collection of patches for the main Django tree > > that enables their backend to be independent of SQL. > > >>> If you are maintaining an external project handling AppEngine support > >>> for Django, then that project's wiki is the right place for > >>> documentation about AppEngine support within Django. While AppEngine > >>> support isn't part of Django's core, Django's wiki isn't the right > >>> place to be providing AppEngine specific instructions - especially if > >>> the first instruction is "install this third party project that isn't > >>> formally affiliated with the Django project at this time". > > >> I can't find anything like that in my wiki page. It describes what is > >> necessary for a completely new port that has practically nothing to do > >> with app-engine-patch apart from being able to reuse some of the > >> existing code. It could be useful for any porting effort. If I keep it > >> in my wiki nobody else can work on that page. It's almost useless that > >> way. > > > Ok - sounds like they could be some good contributions. I just wanted > > to make sure you were not planning on making the Django wiki the home > > of the "how to use app-engine-patch" documentation. > > > Yours, > > Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Thu, 2009-02-12 at 02:14 -0800, Waldemar Kornewald wrote: > On 12 Feb., 03:51, Malcolm Tredinnick > wrote: > > > Is the plan somewhere on the wiki? > > > > No, because it's only something I'm pulling together slowly in my head. > > I hope you've read my wiki page, so that in the end Django is flexible > enough to work even with App Engine's limitations (e.g., it's > practically impossible to implement a TransactionMiddleware): > http://code.djangoproject.com/wiki/AppEngine I've had a quick read of it, but nothing serious, since I have a million other things to do at the moment that are higher priority. However, the overall impression from reading that is that most of those problems aren't something Django itself will need to worry about. It's things that the Query (and other django/db/models/sql/*) replacement will need to handle. Ditto with things like replacing Permissions and ContentTypes -- that will be up to you to fake somehow in the eventual AppEngine code. They're part of Django, so if you choose not to use them (and I really don't understand why you wouldn't, since they're just normal models -- the argument seems to be that you don't want to have an equivalent of syncdb for some reason, but that doesn't necessarily seem to be a required constraint), you have to work around that. > SimpleDB is even worse because of "eventual consistency" which means > there are no transactions and absolutely no integrity guarantees. No, it's easier. Since there are no constraints of that nature, it's not something we have to worry. Designing in features that are impossible isn't a good goal to have. If you're using a storage backend that doesn't support some kind of transactional integrity, then you simply don't have transactional integrity. That's a constraint you accept when choosing that storage backend. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On 12 Feb., 03:51, Malcolm Tredinnick wrote: > > Is the plan somewhere on the wiki? > > No, because it's only something I'm pulling together slowly in my head. I hope you've read my wiki page, so that in the end Django is flexible enough to work even with App Engine's limitations (e.g., it's practically impossible to implement a TransactionMiddleware): http://code.djangoproject.com/wiki/AppEngine SimpleDB is even worse because of "eventual consistency" which means there are no transactions and absolutely no integrity guarantees. > > App Engine support requires at least that Model and probably a few > > other classes can be overridden (maybe partially) > > For example, > > save_base() makes a few queries to check if the row already exists, > > but this would have to be done differently on App Engine (hopefully > > with a transaction). > > No. Backends should require piecemeal futzing around at that level --the > Model level should be reasonably abstracted from backend specifics. We > should, instead, fix the problem more generically by moving those tests > into the django/db/models/sql/ level in some fashion. Yes, that's better, of course. > That's one of those "we'll do it when we need it" tasks. Since this is > the first time anybody's mentioned, we now have a reason to look at it. I really hope that you'll make this possible in version 1.2. I'd like help with implementing the actual App Engine backend. Bye, Waldemar Kornewald --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Wed, 2009-02-11 at 09:08 -0800, Waldemar Kornewald wrote: > Hi Malcolm, > > On 10 Feb., 05:25, Malcolm Tredinnick > wrote: > > I have a reasonably fleshed out plan to make things easier here in the > > Django 1.2 timeframe. The rough idea is that everything under > > django/db/models/sql/ could be replaced with a module of the developer's > > choosing (probably set via a setting). That package is the only place > > that really cares much about SQL. > > > > So somebody wanting a GAE backend or an Hadoop backend or something else > > would write something that behaved the same way as the Query class (and > > subclasses) and could be called by the QuerySet class appropriately. > > Is the plan somewhere on the wiki? No, because it's only something I'm pulling together slowly in my head. > App Engine support requires at least that Model and probably a few > other classes can be overridden (maybe partially) > For example, > save_base() makes a few queries to check if the row already exists, > but this would have to be done differently on App Engine (hopefully > with a transaction). No. Backends should require piecemeal futzing around at that level --the Model level should be reasonably abstracted from backend specifics. We should, instead, fix the problem more generically by moving those tests into the django/db/models/sql/ level in some fashion. That's one of those "we'll do it when we need it" tasks. Since this is the first time anybody's mentioned, we now have a reason to look at it. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
Hi Malcolm, On 10 Feb., 05:25, Malcolm Tredinnick wrote: > I have a reasonably fleshed out plan to make things easier here in the > Django 1.2 timeframe. The rough idea is that everything under > django/db/models/sql/ could be replaced with a module of the developer's > choosing (probably set via a setting). That package is the only place > that really cares much about SQL. > > So somebody wanting a GAE backend or an Hadoop backend or something else > would write something that behaved the same way as the Query class (and > subclasses) and could be called by the QuerySet class appropriately. Is the plan somewhere on the wiki? App Engine support requires at least that Model and probably a few other classes can be overridden (maybe partially). For example, save_base() makes a few queries to check if the row already exists, but this would have to be done differently on App Engine (hopefully with a transaction). BTW, Mitchell Garnaat, the creator of boto (popular AWS library for Python), might be interested in helping with a SimpleDB port: http://groups.google.com/group/boto-users/msg/c68e9456d2c6393e Bye, Waldemar Kornewald --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Tue, Feb 10, 2009 at 12:36 PM, David Larlet wrote: > > > Le 10 févr. 09 à 17:53, Alex Gaynor a écrit : > > David, I think I mentioned this to you, but I believe that by using > > the new F() objects we can actually remove the raw SQL entirely, I'd > > need to put some more thought into this(and perhaps Malcolm already > > has) but it should be possible. > > Alex, I agree with you (not yet tried though), it's only a timing > issue. There are two possibilities: moving then reF()actoring or doing > both in the meantime/same patch. I adopted the first one because I got > no answer to my previous email and that's the blocking point to me, I > thought there were more chance that it get commited quickly keeping > the patch as simple as possible. > Now, if you think it's important to do both and provide some feedback > on the way I did the move I'll be glad to work on it. > > Regards, > David > > > > > From my perspective they don't need to be at once, but there's no point in moving a bunch of code if the new location won't make sense once we remove the raw SQL. I'll take a look at the patch this afternoon and give some thoughts. As for the F(), the question of whether that should be specific to SQL backeneds is interesting, I don't know enough about non-SQL backends to know if that type of query makes sense. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire "The people's good is the highest law."--Cicero --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
Le 10 févr. 09 à 17:53, Alex Gaynor a écrit : > David, I think I mentioned this to you, but I believe that by using > the new F() objects we can actually remove the raw SQL entirely, I'd > need to put some more thought into this(and perhaps Malcolm already > has) but it should be possible. Alex, I agree with you (not yet tried though), it's only a timing issue. There are two possibilities: moving then reF()actoring or doing both in the meantime/same patch. I adopted the first one because I got no answer to my previous email and that's the blocking point to me, I thought there were more chance that it get commited quickly keeping the patch as simple as possible. Now, if you think it's important to do both and provide some feedback on the way I did the move I'll be glad to work on it. Regards, David --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Tue, Feb 10, 2009 at 10:36 AM, David Larlet wrote: > > > Le 10 févr. 09 à 05:25, Malcolm Tredinnick a écrit : > > There's also > > simple enough stuff like removing the last vestiges of raw SQL from a > > couple of places (most notable related fields, but that should > > probably > > go after, or in tandem with, a rewrite of related fields to clean them > > up in general). > > That's the goal of #10109. Good to know there is eventually an > interest in this. > > Regards, > David > > http://code.djangoproject.com/ticket/10109 > > > > > David, I think I mentioned this to you, but I believe that by using the new F() objects we can actually remove the raw SQL entirely, I'd need to put some more thought into this(and perhaps Malcolm already has) but it should be possible. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire "The people's good is the highest law."--Cicero --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
Le 10 févr. 09 à 05:25, Malcolm Tredinnick a écrit : > There's also > simple enough stuff like removing the last vestiges of raw SQL from a > couple of places (most notable related fields, but that should > probably > go after, or in tandem with, a rewrite of related fields to clean them > up in general). That's the goal of #10109. Good to know there is eventually an interest in this. Regards, David http://code.djangoproject.com/ticket/10109 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Feb 10, 4:25 am, Malcolm Tredinnick wrote: > I have a reasonably fleshed out plan to make things easier here in the > Django 1.2 timeframe. That's good to know. I see there's already a hook for a custom query class which would work well as far as querying is concerned, but doesn't seem to be so useful for inserting, updating and deleting, and date and count queries. That's because these are implemented as subclasses of Query, and although we can substitute our own Query subclass for Query, as is done in the Oracle backend, what we actually need is to be able to also affect the behaviour of the subclasses. If one wanted e.g. some special syntax for an INSERT query, how would one hook into the machinery to deliver custom SQL for insertions? Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
Malcolm, Russell, James, Thanks for taking the time to respond so thoroughly to my questions. I think that these questions are rooted in the unrealistic idea that something that looks like it would be good relationship- or PR-wise is also a good development priority. As I expressed, I've simply thought that GAE would be a no-brainer, but that is obviously not the case when viewed against other development work with more concrete demand. Now it is time for me to put-up or shut-up. Hopefully it will not be the latter (and making a post like this can be a good motivator)! -Dave On Mon, Feb 9, 2009 at 11:20 PM, Malcolm Tredinnick wrote: > > On Fri, 2009-02-06 at 09:51 -0500, David Stenglein wrote: >> I have to ask a question here. Why is there such reticence regarding >> App Engine? It would >> seem to me that App Engine has been a feather in the cap for Django. A >> lot of people don't know Django and at a previous job, I was able to >> say that "Google chose django" for App Engine to help validate my use >> of it. > > To echo everybody else; There simply isn't any reticence. > >> >> If so, why the seeming lack of excitement around App Engine? Are >> google engineers not involved in the community? > > Possibly. If they are, they aren't posting from their @google.com > addresses, as far as I can see on this list. > > I would point out that the original GAE integration with Django could > have been implemented quite differently and a lot of this could have > Just Worked out of the box. However, there were reasonably (not > perfectly) valid corporate and legal reasons why Google chose not to > and/or couldn't do that. So the thing works both ways. Google made some > choices and this thread is now about working around those choices to > integrate more nicely. > > Regards, > Malcolm > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Fri, 2009-02-06 at 03:32 -0800, Waldemar Kornewald wrote: > Hi Russell, > > On 6 Feb., 11:34, Russell Keith-Magee wrote: > > I would suggest to you that the broader project of "modifying the > > django.db.models interface to be fully independent of SQL" is much > > more likely to get core developer support. We (the Django core) will > > be very receptive to any suggestions on how django.db.models needs to > > be modified in order to support non-SQL backends -- especially if you > > can demonstrate that your suggestions aren't just theoretical, but are > > clearly required by an actual backend implementation. > > Agreed, making models SQL-independent is the most important (and most > complicated) part. Do the people secretly working on App Engine > integration try to implement that part? I have a reasonably fleshed out plan to make things easier here in the Django 1.2 timeframe. The rough idea is that everything under django/db/models/sql/ could be replaced with a module of the developer's choosing (probably set via a setting). That package is the only place that really cares much about SQL. So somebody wanting a GAE backend or an Hadoop backend or something else would write something that behaved the same way as the Query class (and subclasses) and could be called by the QuerySet class appropriately. There are a few more details involved there than just that one setting, since it requires documenting the expectations (vs the currently convenient assumptions) that Queryset will make on the Query class -- probably that's pretty much what's there now, although I have a list somewhere of a couple/three things I wanted to look at. There's also simple enough stuff like removing the last vestiges of raw SQL from a couple of places (most notable related fields, but that should probably go after, or in tandem with, a rewrite of related fields to clean them up in general). There's also some thinking I have to do about where to draw the line for things like custom lookup types on fields -- how much do we encourage (with the noisy end of a cattle prod) people to be good extension-oriented citizens down the track and where can we help. We've tried pretty hard to keep the SQL-agnostic portions of the API separates from the database bits. I've made Russell's life a bit harder lately by asking for -- fairly insistently -- such changes in the aggregates patch to help with this goal going forwards, for example. The public API for querysets is fairly SQL agnostic, with the exception of Black Sheep extra(), and in that latter case, sometimes there's just nothing you can do about that. We take a fair bit of grief both here and in django-users for having this position, actually. Lots of requests for compromises to allow some half-breed combinations of raw SQL and querysets, which has hidden traps at both the design level and the extension level we're talking about here. So the changes you're talking about are certainly possible. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Fri, 2009-02-06 at 09:51 -0500, David Stenglein wrote: > I have to ask a question here. Why is there such reticence regarding > App Engine? It would > seem to me that App Engine has been a feather in the cap for Django. A > lot of people don't know Django and at a previous job, I was able to > say that "Google chose django" for App Engine to help validate my use > of it. To echo everybody else; There simply isn't any reticence. > > If so, why the seeming lack of excitement around App Engine? Are > google engineers not involved in the community? Possibly. If they are, they aren't posting from their @google.com addresses, as far as I can see on this list. I would point out that the original GAE integration with Django could have been implemented quite differently and a lot of this could have Just Worked out of the box. However, there were reasonably (not perfectly) valid corporate and legal reasons why Google chose not to and/or couldn't do that. So the thing works both ways. Google made some choices and this thread is now about working around those choices to integrate more nicely. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Fri, Feb 6, 2009 at 11:51 PM, David Stenglein wrote: > > I have to ask a question here. Why is there such reticence regarding > App Engine? It would > seem to me that App Engine has been a feather in the cap for Django. A > lot of people don't know Django and at a previous job, I was able to > say that "Google chose django" for App Engine to help validate my use > of it. Reticence isn't a particularly accurate description of the situation. The Django Core developers are all volunteers, so our time is limited. We work on the things that scratch our particular itches. Speaking personally (and I suspect the same is probably true of the rest of the core developers), I don't have a lot of use for App Engine at the moment, and I have a lot of other pressing concerns, so I'm not spending a lot of personal time on the problem. For me, App Engine is in a similar camp to Python 3: a very interesting idea, but not a development priority _right now_. Given time, I expect this to change (commercial hosts will start providing Python 3 support/Google will take App Engine out of preview and into full release), but for the moment, I have a lot of other issues that affect me on a daily basis, and are therefore much more compelling for me to work on. That said, if Google (or anyone else, for that matter) wants to throw some engineers at addressing this issue, we will gratefully accept the assistance. We are in no way opposed to improving App Engine support. If a well designed, well written patch were to arrive on our doorstep today, I suspect you would find that patch in Django's trunk in short order. > It may be coincidental, but I thought that a regularized release > schedule (rather than just "use SVN") seemed to come about after the > App Engine announcement. This was complete coincidence. Our v1.0 timeline was decided upon independent of any App Engine announcement or plans. Google hasn't actively engaged the Django community to coordinate release plans, schedules, or feature lists. > Has App Engine created a noticeable increase in developers interested > in using Django? There has certainly been some interest from App Engine that flows into Django. You don't have to hang out on Django-users for too long to find questions that reference App Engine. > If so, why the seeming lack of excitement around App Engine? This question presupposes that there should be excitement. Rephrasing the question, why should we be excited? On the one hand - the tools they are exposing (BigTable, easy deployment environment, etc) are very exciting developments, and they clearly work well for Google itself. However, it's not like Apache and *SQL installations are going to go away overnight. There are some very good reasons to _not_ use App Engine - not the least of which is the fact that Google lists App Engine as a "preview" product (it doesn't even make it to Googles infamous "Beta" stage :-). At the moment, the vast majority of Django's audience (and 100% of the audience that pays my salary) aren't using App Engine, and wouldn't be able to (for commercial reasons) even if they wanted to. Yes, it's flattering to have one of the largest software companies in the world using Django. However, that doesn't mean that the Django Core developers are all going to drop everything and make App Engine their development priority. > Are > google engineers not involved in the community? There are a few tickets in Django's ticket tracker that I know have come from Google engineers, but those individuals haven't made any big noises about App Engine support. By way of evidence: when the v1.1 feature suggestion list was opened, nobody suggested App Engine support as a potential feature. Django is an open source project. If Google wants App Engine support in Django, they're free to engage with the community and make it happen. If any other company wants App Engine Support in Django, they're free to engage with the community and make it happen. If any individual wants App Engine Support in Django, they're free to engage with the community and make it happen. If you're not willing to write the code yourself, then you either need to convince someone to engage with the community and make it happen, pay someone to engage with the community and make it happen, or wait until it becomes a big enough itch for someone that it gets scratched. Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Fri, Feb 6, 2009 at 8:51 AM, David Stenglein wrote: > I have to ask a question here. Why is there such reticence regarding > App Engine? I'm curious as to what reticence you think there is; since, as Russell said, code to get a straight Django with all features working on App Engine would be welcome, I don't really get the same vibe. The only thing is that we're past the point where major features like this can be proposed for Django 1.1; we're aleady a bit behind schedule getting out the first alpha release, and delaying it even further isn't something we can do at the moment. -- "Bureaucrat Conrad, you are technically correct -- the best kind of correct." --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
I have to ask a question here. Why is there such reticence regarding App Engine? It would seem to me that App Engine has been a feather in the cap for Django. A lot of people don't know Django and at a previous job, I was able to say that "Google chose django" for App Engine to help validate my use of it. I just glanced at the app engine project page and there is even an article about porting your apps to run in any standard Django environment. It may be coincidental, but I thought that a regularized release schedule (rather than just "use SVN") seemed to come about after the App Engine announcement. Has App Engine created a noticeable increase in developers interested in using Django? If so, why the seeming lack of excitement around App Engine? Are google engineers not involved in the community? Sorry to be posing so many questions, but I've found this aspect of Django development particularly puzzling. -Dave On Fri, Feb 6, 2009 at 7:10 AM, Russell Keith-Magee wrote: > > On Fri, Feb 6, 2009 at 8:32 PM, Waldemar Kornewald > wrote: >> >> Hi Russell, >> >> On 6 Feb., 11:34, Russell Keith-Magee wrote: >>> I would suggest to you that the broader project of "modifying the >>> django.db.models interface to be fully independent of SQL" is much >>> more likely to get core developer support. We (the Django core) will >>> be very receptive to any suggestions on how django.db.models needs to >>> be modified in order to support non-SQL backends -- especially if you >>> can demonstrate that your suggestions aren't just theoretical, but are >>> clearly required by an actual backend implementation. >> >> Agreed, making models SQL-independent is the most important (and most >> complicated) part. Do the people secretly working on App Engine >> integration try to implement that part? > > I haven't seen their code, so I can't say for certain, but I imagine > that they have a small collection of patches for the main Django tree > that enables their backend to be independent of SQL. > >>> If you are maintaining an external project handling AppEngine support >>> for Django, then that project's wiki is the right place for >>> documentation about AppEngine support within Django. While AppEngine >>> support isn't part of Django's core, Django's wiki isn't the right >>> place to be providing AppEngine specific instructions - especially if >>> the first instruction is "install this third party project that isn't >>> formally affiliated with the Django project at this time". >> >> I can't find anything like that in my wiki page. It describes what is >> necessary for a completely new port that has practically nothing to do >> with app-engine-patch apart from being able to reuse some of the >> existing code. It could be useful for any porting effort. If I keep it >> in my wiki nobody else can work on that page. It's almost useless that >> way. > > Ok - sounds like they could be some good contributions. I just wanted > to make sure you were not planning on making the Django wiki the home > of the "how to use app-engine-patch" documentation. > > Yours, > Russ Magee %-) > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On 6 Feb., 13:10, Russell Keith-Magee wrote: > Ok - sounds like they could be some good contributions. I just wanted > to make sure you were not planning on making the Django wiki the home > of the "how to use app-engine-patch" documentation. Hmm, now that you mention it, this sounds like an interesting idea. :p I've moved the document here: http://code.djangoproject.com/wiki/AppEngine Bye, Waldemar Kornewald --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Fri, Feb 6, 2009 at 8:32 PM, Waldemar Kornewald wrote: > > Hi Russell, > > On 6 Feb., 11:34, Russell Keith-Magee wrote: >> I would suggest to you that the broader project of "modifying the >> django.db.models interface to be fully independent of SQL" is much >> more likely to get core developer support. We (the Django core) will >> be very receptive to any suggestions on how django.db.models needs to >> be modified in order to support non-SQL backends -- especially if you >> can demonstrate that your suggestions aren't just theoretical, but are >> clearly required by an actual backend implementation. > > Agreed, making models SQL-independent is the most important (and most > complicated) part. Do the people secretly working on App Engine > integration try to implement that part? I haven't seen their code, so I can't say for certain, but I imagine that they have a small collection of patches for the main Django tree that enables their backend to be independent of SQL. >> If you are maintaining an external project handling AppEngine support >> for Django, then that project's wiki is the right place for >> documentation about AppEngine support within Django. While AppEngine >> support isn't part of Django's core, Django's wiki isn't the right >> place to be providing AppEngine specific instructions - especially if >> the first instruction is "install this third party project that isn't >> formally affiliated with the Django project at this time". > > I can't find anything like that in my wiki page. It describes what is > necessary for a completely new port that has practically nothing to do > with app-engine-patch apart from being able to reuse some of the > existing code. It could be useful for any porting effort. If I keep it > in my wiki nobody else can work on that page. It's almost useless that > way. Ok - sounds like they could be some good contributions. I just wanted to make sure you were not planning on making the Django wiki the home of the "how to use app-engine-patch" documentation. Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
Hi Russell, On 6 Feb., 11:34, Russell Keith-Magee wrote: > I would suggest to you that the broader project of "modifying the > django.db.models interface to be fully independent of SQL" is much > more likely to get core developer support. We (the Django core) will > be very receptive to any suggestions on how django.db.models needs to > be modified in order to support non-SQL backends -- especially if you > can demonstrate that your suggestions aren't just theoretical, but are > clearly required by an actual backend implementation. Agreed, making models SQL-independent is the most important (and most complicated) part. Do the people secretly working on App Engine integration try to implement that part? On 6 Feb., 11:53, Russell Keith-Magee wrote: > On Fri, Feb 6, 2009 at 5:51 PM, Waldemar Kornewald > wrote: > > > BTW, could someone please activate my "wkornewald" Trac account, so I > > can move the documentation into your wiki? > > After I pressed send, something else occurred to me: > > Before you starting loading documentation into Django's wiki, please > consider if Django's wiki is the right place for that documentation. > > If you are maintaining an external project handling AppEngine support > for Django, then that project's wiki is the right place for > documentation about AppEngine support within Django. While AppEngine > support isn't part of Django's core, Django's wiki isn't the right > place to be providing AppEngine specific instructions - especially if > the first instruction is "install this third party project that isn't > formally affiliated with the Django project at this time". I can't find anything like that in my wiki page. It describes what is necessary for a completely new port that has practically nothing to do with app-engine-patch apart from being able to reuse some of the existing code. It could be useful for any porting effort. If I keep it in my wiki nobody else can work on that page. It's almost useless that way. Bye, Waldemar Kornewald --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Fri, Feb 6, 2009 at 5:51 PM, Waldemar Kornewald wrote: > > BTW, could someone please activate my "wkornewald" Trac account, so I > can move the documentation into your wiki? After I pressed send, something else occurred to me: Before you starting loading documentation into Django's wiki, please consider if Django's wiki is the right place for that documentation. If you are maintaining an external project handling AppEngine support for Django, then that project's wiki is the right place for documentation about AppEngine support within Django. While AppEngine support isn't part of Django's core, Django's wiki isn't the right place to be providing AppEngine specific instructions - especially if the first instruction is "install this third party project that isn't formally affiliated with the Django project at this time". Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: App Engine port
On Fri, Feb 6, 2009 at 5:51 PM, Waldemar Kornewald wrote: > > A native Django port could provide us with a clean, more powerful, and > more actively developed API for App Engine development. Moreover, it > could provide a solution for provider lock-in by abstracting the > underlying API and thus making easy migrations possible. You won't get any disagreement here. Native support for AppEngine within Django would be a nice feature. Support for other storage backends (like CouchDB, Amazon SimpleDB, and so on) would also be nice to have. > I know that a lot of changes in the storage/query backend code need to > be done in order to support the App Engine datastore and similar > databases and I doubt that I can do the necessary API changes, but I > could at least help with implementing the App Engine backends > (sessions, cache, mail, db/storage). As a starting point, I've > documented a few high-level requirements here: > http://code.google.com/p/app-engine-patch/wiki/DjangoAppEnginePort > > Is there any roadmap towards App Engine support? Is there a formal roadmap? No. App Engine support wasn't mentioned on our v1.1 roadmap, and we don't plan beyond the next release. > Is anyone working on it, already? I'm not aware of any public projects. Reading between the lines, I suspect I know at least one person that is working on App Engine integration. However, I don't want to steal their thunder, so I'll leave it up to them to show their hand if they feel it is appropriate. > When will someone work on it and in which Django release can we expect > native support? Is 1.2 possible? The code isn't going to magically write itself, and it will be a non-trivial job. You can expect native support right after someone commits to implementing it :-) Whether this is possible for v1.2 is very much dependent on who puts in the work, and how much time they can put into the task. They will also need to convince a core developer that you are serious, and that they are capable of bringing the project to fruition. Providing an initial patch -- even if it is incomplete -- will go a long way to providing this proof. I would also note that our (that is, the Core Developers) initial reaction to non-SQL storage backends will be the same as it is for SQL database backends -- we will want to see public interest and continued support as a standalone project before we commit to integrating anything into the Django core. This policy is why we have an Oracle backend, but we don't have an MSSQL backend. There are, however, several external projects providing MSSQL integration. I would suggest to you that the broader project of "modifying the django.db.models interface to be fully independent of SQL" is much more likely to get core developer support. We (the Django core) will be very receptive to any suggestions on how django.db.models needs to be modified in order to support non-SQL backends -- especially if you can demonstrate that your suggestions aren't just theoretical, but are clearly required by an actual backend implementation. The goal here isn't to get AppEngine support into the core - it is to provide an interface in Django that allows you to implement an AppEngine backend (or any other backend, for that matter) as a completely standalone project. In time, this may lead to integration into the Django core, but in the interim, we would rather see a vibrant ecosystem of external plugins, rather than try and force everything into Django's trunk. > BTW, could someone please activate my "wkornewald" Trac account, so I > can move the documentation into your wiki? I just don't receive any > activation mail when registering. I tried sending to > webmas...@djangoproject.com, but that address doesn't exist. Who can I > talk to? I can mail you activation code privately. Expect an email from me shortly. Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
