Re: [GSoC 2012] Schema Alteration API proposal
On Thursday, 5 April 2012 21:25:19 UTC+5:30, Andrew Godwin wrote: > > If you plan to continue using > Django fields as type information (as South does), what potential issues > do you see there? > The only issue I can think of is the case of custom fields created by the user. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/TTHMOOOFAhYJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [GSoC 2012] Schema Alteration API proposal
On Thursday, 5 April 2012 21:25:19 UTC+5:30, Andrew Godwin wrote: > > Just thought I'd chime in now I've had a chance to look over the current > proposal (I looked at the current one you have in the GSOC system): > > - When you describe feeding things in from local_fields, are you > referring to that being the method by which you're planning to implement > things like syncdb? > Actually I am not planning to mess with syncdb and other management commands. I will only refactor django.db.backends creation functions like sql_create_model etc. to use the new API. Behaviour and functionality will be the same after refactor, so management commands like syncdb will not notice a difference. > - I'd like to see a bit more detail about how you plan to test the > code - specifically, there are some backend-specific tests you may need, > as well as some detailed introspection in order to make sure things have > applied correctly. > Currently, I can only think of things like the unique index on SQLite and oddities in MySQL mostly again from South's test suite, I will give another update before today's deadline. > - Russ is correct about your models approach - as I've said before in > other places, the models API in Django is not designed with models as > moveable, dynamic objects. > I have taken care of clearing the app cache, after migrations. Actually the entire point of using these 'Django code' based tests is that I wanted to doubly ensure that Django will behave the way its supposed to after the migrations. I could have gone with a SQL only approach e.g. 'SELECT table' after calling db.delete_table but using testing using Django code seemed a bit more comprehensive. Now, to mimic migrations, I needed to alter model definitions. The closest way to resemble actual migration scenario seemed to be to change the definitions in models.py itself. File rename/rewrite is ugly and OS dependent thats why I used a 'temporary setting' based approach. I know that messing with app cache looks a bit hackish but I cannot think of anything else for now. South has one approach to these sorts of > tests, but I'd love to see a cleaner suggestion. > Are you referring to the fake orm? Well if you are satisfied with my above explanation, there would be no need for it, since we will be using django's orm. > - There's been some discussion on south-users about the benefits of a > column-based alteration API versus a field/model-based alteration API - > why have you picked a column-based one? If you plan to continue using > Django fields as type information (as South does), what potential issues > do you see there? > Well you said it yourself above that "the models API in Django is not designed with models as moveable, dynamic objects". That is why I used a column-based approach. The advantage will be felt in live migrations. As for using Django fields for type information, I frankly cannot think of a major valid negative point for now, I will revert later today. > - Some more detail on your background would be nice - what's your > specific experience with the 3 main databases you'll be handling > (postgres, mysql, sqlite)? What was a "high voltage database migration"? > Sure. I will update it. > Sorry for the late feedback, I've been far too busy. > No problem, as long as you reply to this before the deadline :D > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/5eBCoe2syNYJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [GSoC 2012] Schema Alteration API proposal
Hi Russell, Thanks for your immense patience :-) These are some additions to my proposal above, based on your inputs: Status of current 'creation' code in django: The current code, for e.g. sql_create_model in django.db.backends.creation is a mix of *inspection* part and *sql generation* part. Since the sql generation part will (should) now be handled by our new CRUD API, I will refactor django.db.backends.creation (and other backends' creation modules) to continue using their inspection part but using our new CRUD API for sql generation. The approach will be to get the fields using model._meta.local_fields and feeding them to our new CRUD API. This will serve to be a proof of concept for my API. As for testing using Django code, my models will be something like: class UnchangedModel(models.Model): eg = models.TextField() if BEFORE_MIGRATION: class MyModel(models.Model): f1 = models.TextField() f2 = models.TextField() # Deletion of a field else: class MyModel(models.Model): f1 = models.TextField() The value of BEFORE_MIGRATION will be controlled by the testing code. A temporary environment variable can be used for this purpose. Also a revised schedule: Bonding period before GSoC: Discussion on API design Week 1: Writing tests (using 2 part checks (checking the actual database and using Django models), as discussed above) Week 2: Developing the base migration API Week 3: Developing extensions and overrides for PostgreSQL Weeks 4-5 : Developing extensions and overrides for MySQL Weeks 6-7 : Developing extensions and overrides for SQLite (may be shorter or longer (by 0.5 week) depending on how much of xtrqt's code is considered acceptable) Weeks 8-10 : Refactoring django.db backends.creation (and the PostgreSQL, MySQL, SQLite creation modules) to use the new API for SQL generation (approach discussed above) Week 11 : Writing documentaion and leftover tests, if any Week 12 : Buffer week for the unexpected On Tuesday, 3 April 2012 06:39:37 UTC+5:30, Russell Keith-Magee wrote: > > > On 03/04/2012, at 5:06 AM, j4nu5 wrote: > > > Hi Russell, > > > > Thanks for the prompt reply. > > > > * You aren't ever going to eat your own dogfood. You're spending the > GSoC building an API that is intended for use with schema migration, but > you're explicitly not looking at any part of the migration process that > would actually use that API. How will we know that the API you build is > actually fit for the purpose it is intended? How do we know that the > requirements of "step 2" of schema migration will be met by your API? I'd > almost prefer to see more depth, and less breadth -- i.e., show me a fully > functioning schema migration stack on just one database, rather than a > fully functioning API on all databases that hasn't actually been shown to > work in practice. > > > > 'Eating my own dogfood' to check whether my low level migration > primitives are actually *usable*, I believe can be done by: > > 1. Developing a working fork of South to use these primitives as I > mentioned in my project goals, or > > 2. Aiming for less 'breadth' and more 'depth', as you suggested. > > > > I did not opt for 2, since creating the '2nd level' of the migration > framework (the caller of the lower level API) is a huge beast by itself. > Any reasonable solution will have to take care of 'Pythonic' as well as > 'pseudo-SQL' migrations as discussed above. Not to mention taking care of > versioning + dependency management + backwards migrations. I am against the > development of a half baked and/or inconsistent 2nd level API layer. Trying > to fully develop such a solution even for one database will exceed the GSoC > timeline, in my humble opinion. > > Ok - there's two problems with what you've said here: > > 1) You don't make any reference in your schedule to implementing a > "working fork of South". This isn't a trivial activity, so if you're > planning on doing this, you should tell use how this is factored into your > schedule. > > 2) You're making the assumption that you need to "fully develop" a > solution. A proof of concept would be more than adequate. For example, in > the 2010 GSoC, Alex Gaynor's project was split into two bits; a bunch of > modifications to the core query engine, and a completely separate project, > not intended for merging to trunk, that demonstrated that his core query > changes would do what was necessary. You could take exactly the same > approach here; don't try to delivery
Re: [GSoC 2012] Schema Alteration API proposal
Less than a week remains for student application deadline. Can someone please comment on the above revised proposal. Thanks a lot. On Monday, 26 March 2012 01:29:35 UTC+5:30, j4nu5 wrote: > > Here is a revised proposal. > > Abstract > > -- > A database migration helper has been one of the most long standing feature > requests in Django. Though Django has an excellent database creation > helper, > when faced with schema design changes, developers have to resort to either > writing raw SQL and manually performing the migrations, or using third > party > apps like South[1] and Nashvegas[2]. > > [1] http://south.aeracode.org/ > [2] https://github.com/paltman/nashvegas/ > > Clearly Django will benefit from having a database migration helper as an > integral part of its codebase. > > From the summary on django-developers mailing list[3], the task of > building a > migrations framework will involve: > 1. Add a db.backends module to provide an abstract interface to migration >primitives (add column, add index, rename column, rename table, and so > on). > 2. Add a contrib app that performs the high level accounting of "has > migration >X been applied", and management commands to "apply all outstanding >migrations" > 3. Provide an API that allows end users to define raw-SQL migrations, or >native Python migrations using the backend primitives. > 4. Leave the hard task of determining dependencies, introspection of > database >models and so on to the toolset contributed by the broader community. > > [3] http://groups.google.com/ > group/django-developers/msg/cf379a4f353a37f8 > > I would like to work on the 1st step as part of this year's GSoC. > > > Implementation plan > > -- > The idea is to have a CRUD interface to database schema (with some > additional > utility functions for indexing etc.) with functions like: > * create_table > * rename_table > * delete_table > * add_column > and so on, which will have the *explicit* names of the table/column to be > modified as its parameter. It will be the responsibility of the higher > level > API caller (will not be undertaken as part of GSoC) to translate > model/field > names to explicit table/column names. These functions will be directly > responsible for modifying the schema, and any interaction with the database > schema will take place by calling these functions. Most of these functions > will come from South. > > These API functions will also have a "dry-run" or test mode, in which they > will output raw SQL representation of the migration or display errors if > they > occur. This will be useful in: > 1. The MySQL backend. MySQL does not have transaction support for schema >modification and hence the migrations will be run in a dry run mode > first >so that any errors can be captured before altering the schema. > 2. The django-admin commands sql and sqlall that return the SQL (for > creation >and indexing) for an app. They will capture the SQL returned from the > API >running in dry run mode. > > As for the future of the current Django creation API, it will have to be > refactored (not under GSoC) to make use of the 'create' part of our new > CRUD > interface, for consistency purposes. > > The GeoDjango backends will also have to be refactored to use the new API. > Since, they build upon the base code in db.backends, firstly db.backends > will > have to be refactored. > > Last year xtrqt had written, documented and tested code for at least the > SQLite backend[4]. As per Andrew's suggestion, I would not be relying too > much > on that code but some parts can still be salvaged. > > [4] https://groups.google.com/ > forum/?fromgroups#!searchin/django-developers/xtrqt/django-developers/pSICNJBJRy8/Hl7frp-O-dMJ > > > Schedule and Goal > > -- > Week 1 : Discussion on API design and writing tests > Week 2-3 : Developing the base migration API > Week 4 : Developing extensions and overrides for PostgreSQL > Week 5-6 : Developing extensions and overrides for MySQL > Week 7-8.5 : Developing extensions and overrides for SQLite (may be > shorter or > longer (by 0.5 week) depending on how much of xtrqt's code is > considered acceptable) > Week 8.5-10: Writing documentaion and leftover regression tests, if any &g
Discussion on Ticket #16304
I have written a patch for ticket #16304 (https:// code.djangoproject.com/ticket/16304). The ticket adds support for the HTML5 input placeholder attribute (WHAT Working Group placeholder description - http://www.whatwg.org/specs/web-apps/current-work/multipage/common-input-element-attributes.html#the-placeholder-attribute), in Forms as well as ModelForms. I was wondering if a larger undertaking can be taken to make Django HTML5 aware. Options may include support for new input types like tel, email etc. which currently are rendered simply as text and ModelForms/ Forms have to manually override widgets to achieve desired results. Comments please. P.S. The submitted patch lacks tests. I could not figure out how to write tests for the UI. If that is acceptable, can core devs please accept the patch? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Student Proposal for Google Summer of Code 2012
Hi, Google Summer of Code 2012 has been announced. http://google-opensource.blogspot.com/2012/02/google-summer-of-code-2012-is-on.html I want to work on extending Django's database migration capabilities, as a student. Database migrations have always been a pain in Django and most people rely on third party apps like South or Nashvegas for migrations. I feel migration capabilities should be made part of the main codebase. Specifically, adding a new field to an existing model and issuing 'manage.py syncdb' has no effect since manage.py does not issue ALTER TABLE commands. Adding such functionality might be difficult to implement but will be a boon to new users (especially those migrating from Rails who are used to out of the box migration support). Is this project worthy of being taken up for this year's Summer of Code? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
