Re: Using AWS Redshift database Django 1.6

[jpk]

Ah, I see!  I was mistaken, then.  So the base db backend's
sql_create_model is what generates the create table with the constraint?
 Then a db backend subclass just needs to override that to do whatever
dbms-specific stuff it wants?  Is there documentation on what all the
methods in a django.db.backends module are supposed to do?  (The docstrings
in the code seem decent enough, just wondering if there's anything more
detailed than that.)

Thanks!
jpk


On Thu, May 29, 2014 at 1:48 PM, C. Kirby <mist...@gmail.com> wrote:

> I'll answer what I can from a quick review of the codebase.
>
>
> On Thursday, May 29, 2014 12:36:39 PM UTC-5, jpk wrote:
>>
>> Yeah, I came across that as well, but it didn't work on 1.6.  I cloned
>> and debugged it up until I got the exact same error as the paste in my
>> first message. :(
>>
>> So, a few questions:
>>
>>- Am I correct in my analysis that the syncdb command alone is
>>generating the table creation sql that includes the constraint redshift is
>>barfing on?
>>
>> No, syncdb (or in 1.7+ migrate) eventually calls the correct db backend
> to generate the sql. You can see from the code
> <https://github.com/django/django/blob/stable/1.6.x/django/core/management/commands/syncdb.py>
> that there is no SQL there, just calls to the backend sql generation (In
> particular look at line 96
> <https://github.com/django/django/blob/stable/1.6.x/django/core/management/commands/syncdb.py#L96>
> )
>
>>
>>- If I were to patch the syncdb command code, is it reasonable to
>>consult the database backend for settings related to which constraints are
>>allowed/not?
>>
>> I would not change the sync code. A better path would probably be to
> start with the postgres backend and make the changes necessary for redshift
> (I assume that is what the django-redshift project started to do)
>
>>
>>- Does it make more sense for the database backend code to do the
>>table creation instead of syncdb?  (As in, db backends provide might an
>>abstracted create table method that syncdb could just call instead of
>>building its own sql.)
>>
>> I know more than I did a day ago, but I'm still not super hip on how db
>> backends are implemented and how the rest of the framework interacts with
>> them.  So if the stuff I'm asking doesn't make sense then do put me on the
>> right path. :)
>>
>> Thanks!
>> jpk
>>
>
> Kirby
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/5a094392-088f-4062-b85a-2a0a24bf192b%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/5a094392-088f-4062-b85a-2a0a24bf192b%40googlegroups.com?utm_medium=email_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
john p. kiffmeyer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAEQ4iqVi6DWt4FqCyoGBhq%3D7KBJd-tvkRMMc5Fh_ES9-M9bTLQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using AWS Redshift database Django 1.6

[jpk]

Yeah, I came across that as well, but it didn't work on 1.6.  I cloned and
debugged it up until I got the exact same error as the paste in my first
message. :(

So, a few questions:

   - Am I correct in my analysis that the syncdb command alone is
   generating the table creation sql that includes the constraint redshift is
   barfing on?
   - If I were to patch the syncdb command code, is it reasonable to
   consult the database backend for settings related to which constraints are
   allowed/not?
   - Does it make more sense for the database backend code to do the table
   creation instead of syncdb?  (As in, db backends provide might an
   abstracted create table method that syncdb could just call instead of
   building its own sql.)

I know more than I did a day ago, but I'm still not super hip on how db
backends are implemented and how the rest of the framework interacts with
them.  So if the stuff I'm asking doesn't make sense then do put me on the
right path. :)

Thanks!
jpk



On Thu, May 29, 2014 at 12:01 PM, C. Kirby <mist...@gmail.com> wrote:

> Wow, redshift lets you save quite a bit of data.
> It looks like someone started in on a redshift backend, I found something
> on github (https://github.com/binarydud/django-redshift)
> It appears to be unmaintained, and maybe not even complete, but you might
> get some idea's of where work needs to be done from it.
>
> Kirby
>
>
> On Thursday, May 29, 2014 9:13:55 AM UTC-5, jpk wrote:
>>
>> Greetings,
>>
>> I'm attempting to set up a django project using amazon redshift as the
>> database backend.  I started a fresh 1.6.5 project and set up the database
>> portion of settings.py to use the redshift cluster. With just the default
>> apps (admin, auth, contenttypes, sessions, messages, and staticfiles), I
>> ran syncdb and got this: https://dpaste.de/y98N.  Compared to vanilla
>> postgre, redshift doesn't support some things, including some constraints,
>> which is what's making syncdb blow up.
>>
>> What worries me in that backtrace is, the table creation sql that has the
>> offending constraint in it is being generated in commands/syncdb.py, so
>> that's before we even get to the postgresql_psycopg2 backend code, right?
>>  So hacking on the postgres backend to make it jive with redshift wouldn't
>> help, here.  (Correct me if I'm wrong!)
>>
>> Anyway, what wisdom can you share?  What's the best way to approach this?
>>  Is trying to get this to work even a good idea?  Any guidance would be
>> greatly appreciated.
>>
>>
>> Thanks!
>> jpk
>>
>>
>>
>> --
>> john p. kiffmeyer
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/8bd4e6b5-42c5-4779-86af-cb04b708540d%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/8bd4e6b5-42c5-4779-86af-cb04b708540d%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
john p. kiffmeyer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAEQ4iqUF5t5A1MXQzwfy81TJ-NkngLBkMdhhN7Ra2Ki5suWd-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Using AWS Redshift database Django 1.6

[jpk]

Greetings,

I'm attempting to set up a django project using amazon redshift as the
database backend.  I started a fresh 1.6.5 project and set up the database
portion of settings.py to use the redshift cluster. With just the default
apps (admin, auth, contenttypes, sessions, messages, and staticfiles), I
ran syncdb and got this: https://dpaste.de/y98N.  Compared to vanilla
postgre, redshift doesn't support some things, including some constraints,
which is what's making syncdb blow up.

What worries me in that backtrace is, the table creation sql that has the
offending constraint in it is being generated in commands/syncdb.py, so
that's before we even get to the postgresql_psycopg2 backend code, right?
 So hacking on the postgres backend to make it jive with redshift wouldn't
help, here.  (Correct me if I'm wrong!)

Anyway, what wisdom can you share?  What's the best way to approach this?
 Is trying to get this to work even a good idea?  Any guidance would be
greatly appreciated.


Thanks!
jpk



-- 
john p. kiffmeyer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAEQ4iqWJWH7s01osREW3j%3Dt%2BZ5tPQU8_08mtn-WWUF0a5gOAeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Ways to handle long-running server-side processing in Django

[jpk]

Ah, celery sounds like it's just what I need.  Thanks guys!

-jpk

On Mon, Apr 23, 2012 at 7:52 AM, bruno desthuilliers <
bruno.desthuilli...@gmail.com> wrote:

> On Apr 21, 11:34 pm, David Markey <ad...@dmarkey.com> wrote:
> > Sounds like the API should return a UUID that can then be used to poll,
> > while celery or similar does the heavy lifting in the background?
>
> +1
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>


-- 
john p. kiffmeyer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Ways to handle long-running server-side processing in Django

[jpk]

Hi django-users,

I've come looking for some advice.  I've built a json api using django, and
POST/PUT to one of my endpoints does some non-trivial processing.  The
gritty details are included at the end, but for the purposes of this
discussion, the important things are probably 1) The action is for
processing a user's previously uploaded data, 2) it takes a long time to
complete, and 3) probably doesn't make sense to spread over multiple api
calls.  The work that's done can take multiple minutes to complete, which
is an awful long time to wait for an http response to event begin (right?
 Or am I mistaken, there?).  Right now, that's what I'm doing.  The client
just has to wait until the processing is finished to get the response.
 This is functional, but I'm considering what can be done to make for a
better user experience and/or make it less fragile (I'm honestly not sure
what would happen if the connection died before the response goes out).

So my question is, what are my options, here?  I'm thinking the api could
respond immediately with a message like, "cool, I'll get right on that!"
and the heavy lifting happens in the background sometime after the response
goes out.  What constructs in django or python best facilitate that?  Then
what do I tell the client?  "Poll the api until it's done."?  I could also
maybe keep it all in one request like it is now, and stream progress
information back to the client as the process does stuff.  What other
approaches are there?  Am I over-thinking this? :-P  Any wisdom would be
appreciated!

Thanks!
jpk


P.S. - The gritty details:  The api call in question takes an image (that
the user previously uploaded) and a set of ground control points (pairs of
longitude,latitude on the world and x,y on the image) to generate map tiles
of the image that can be overlayed on world map tiles.  Tools from OSGeo's
excellent GDAL library are used to do this, then all the tiles are shipped
off to S3.  Once this is finished, the client has a url it can use to grab
tiles (this url is stored in the database, accessible via later api calls,
and also included in the response generated after all that processing is
done).  There's still lots I can do to optimize this process, but I don't
expect it can be made near-instant.  The other component to this is the
frontend: written using backbone.js.  I'm already using a custom
Backbone.sync(), so I suppose extra gymnastics in there (particularly in
the case of streaming progress information) to support whatever I do in the
backend related to this wouldn't be out of bounds.


-- 
john p. kiffmeyer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Using RequireJS with Django

[jpk]

Hi django-users,

I'm working on a django project with a quickly growing javascript
front-end using backbone.  The backbone application has grown to the
point where it needs to be split into modules across multiple files.
I'm looking into RequireJS to do that.

I've done some research on how to best use require in django, but
haven't found much.  My first stab at this is probably going to be
more or less: Take the single javascript file I have now, break it
into AMD modules in a sub-directory of the static directory the single
file is in, set up require's minification stuff to put the minified
javascript there in another sub-directory the same static directory,
then set up templates to conditionally include the source/minified
javascript for development/production.  Then I'll figure out some
mechanism to run the minification stuff on the server during
deployment.

So, what I'm looking for is wisdom related to carrying out that
process and maintaining it.  Has anyone set up require in a django
project before, and have any tips to share?  I'd love to hear them!

Thanks
john

--
john p. kiffmeyer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Nesting spatial filters with the __in filter will be broken until I buy you a beer.

[jpk]

Hi all,

I've recently been using the __in filter, and it's pretty nifty.
However, it seems that nesting any of geodjango's spatial filters with
the __in filter is busted.  This ticket:
https://code.djangoproject.com/ticket/17314 explains the behavior in
detail, but the précis is: "things =
Thing.objects.filter(point_field__contained=Polygon(...))" followed by
"other_things = OtherThing.objects.filter(thing__in=things)" results
in the sql compiler blowing up when the query for other_things is
supposed run.

This is a major pain in two apps I'm working on, and I'll sponsor
patches with things including but not limited to:
 * Free beer
 * Homemade cupcakes
 * Hugs

Or, if you have the inhuman inclination to refuse the aforementioned
bribery, I can probably fix the issue with some direction (which would
still be met with considerable gratitude).

Thanks,
jpk

-- 


John P. Kiffmeyer
Email/XMPP: j...@thekiffmeyer.org

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How to deploy new code to the Production Server?

[jpk]

On Fri, Dec 2, 2011 at 6:54 AM, kenneth gonsalves
 wrote:
> On Fri, 2011-12-02 at 11:29 +, Bjarni Rúnar Einarsson wrote:
>> (for example
>> as someone recommended earlier, skipping settings.py), you are IMO
>> asking for trouble and it is probably a sign that your processes are
>> broken. :-)
>
> you put your passwords and keys under version control?
> --
> regards
> Kenneth Gonsalves
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-users?hl=en.
>
>

Check out https://code.djangoproject.com/wiki/SplitSettings
Personally, I use the second example, "Multiple setting files
importing from each other".

You can put settings common to both development and production in
settings.py and keep it in version control, but have a
settings_local.py or something that has things like database
connection information (including passwords), SECRET_KEY, and whatever
else would be secret or different between development/production.
Then .gitignore settings_local.py.  I also keep a
settings_local.py.template in version control that's just a
fill-in-the-blank of settings_local.py for quick set up on a new dev
machine or production server.

-- 


John P. Kiffmeyer
Email/XMPP: j...@thekiffmeyer.org

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: How to deploy new code to the Production Server?

[jpk]

On Fri, Dec 2, 2011 at 5:30 AM, Benedict Verheyen
 wrote:
> On 2/12/2011 11:14, Andre Lopes wrote:
>> Thanks for the replies,
>>
>> I'm using Nginx + Gunicorn + Supervisor + Virtualenv
>>
>> My goal is to deploy the code to the Production in a One Click Step. I
>> think I will read on Fabric to achieve this.
>>
>> More read also about Pip, I don't know how Pip Freeze works.
>>
>> If you have some more clues, you are welcome.
>>
>>
>> Best Regards,
>>
>
> If you want 1 click, you'll need fabric, read up on that.
> As for pip, pip can be used to install the various dependencies of your
> project in your virtualenv.
> You can then list these dependencies with this command:
>    pip freeze > requirements.txt
>
> This writes them into a file named requirements.txt that you can then
> use to setup the dependencies of your virtualenv on the production server.
>
>    pip install -r requirments.txt
>
> You only do this to setup the production server, it's not needed to transfer
> code, that you do with git (or mercurial, hg, ...).
>
>
> Regards,
> Benedict
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-users?hl=en.
>
>

You might find this useful:
http://python.mirocommunity.org/video/1689/pycon-2010-django-deployment-w

It's jacobian's django deployment workshop from pycon 2010, so it's
about 3h long.  It's worth watching front to back, but he only spends
a little time on deployment tools.  (The rest is nginx, apache,
postgre, memcached, etc.)  The tl;dw is: check out fabric, puppet, and
buildout for automating these things.

Cheers,
john

-- 


John P. Kiffmeyer
Email/XMPP: j...@thekiffmeyer.org

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.