Re: registering unzipped files on the local disk
On Sun, 19 Jun 2011, francescortiz wrote:
Faheem,
In your post you write:
"This works, but I traced through the code, and found that the code
was using streaming, when clearly the optimal thing to do in this case
would be to just copy the file from the temporary location to the
permanent location"
In java copying files is performed by redericting data from a file
input stream to a file ouput stream, so your sentence sounded to me
like "Instead of copying a file I want to just copy a file" :P
Though I am no python expert, I suspect that using streams in python
is fine as well.
You could do some performance tests of stream copy vs copyfile
function in python. If you do, please share results!
Hope I help,
Francesc
Hi Francesc,
Thanks for the helpful comments. As I mention in the post, the Django
code (in the `_save` function of the `FileSystemStorage` class,
located in the file 'django/core/files/storage.py') appears to be
looking for the `temporary_file_path` attribute. If it finds it, it
executes `file_move_safe`, from 'django/core/files/move.py'. I've
included move.py below for reference.
Following your comments, I looked more carefully at what
`file_move_safe` does. This code first checks origin and destination
are not the same file. If that is so, it attempts `os.rename`, which I
think you will agree, is optimal if both the origin and destination
are on the same filesystem. If not, it copies the data across in
chunks (apparently what is referred to as streaming). This is exactly
the same code used in `FileSystemStorage.save`. So basically the only
thing missing is `os.rename` is not called if `temporary_file_path` is
not seen. Also, if the origin and destination are the same path,
currently the streaming code has to deal with this, and I'm not sure
what happens then.
I thought unix `cp` or similar was used somewhere, but no.
Anyway, the upshot is - I think you are right that there isn't a big
issue here. Regardless, the point of my post is that it didn't look
like the `_save` function of the `FileSystemStorage` class would ever
encounter the `temporary_file_path` attribute, so the bit of the code
that looks for it and executes `file_move_safe` seems like dead code,
so should be either fixed or removed.
Also thank you for the timing information. My concerns were/are more
about excessive memory usage, since I'm dealing with large files.
Regards, Faheem
###
"""
Move a file in the safest way possible::
>>> from django.core.files.move import file_move_safe
>>> file_move_safe("/tmp/old_file", "/tmp/new_file")
"""
import os
from django.core.files import locks
try:
from shutil import copystat
except ImportError:
import stat
def copystat(src, dst):
"""Copy all stat info (mode bits, atime and mtime) from src to dst"""
st = os.stat(src)
mode = stat.S_IMODE(st.st_mode)
if hasattr(os, 'utime'):
os.utime(dst, (st.st_atime, st.st_mtime))
if hasattr(os, 'chmod'):
os.chmod(dst, mode)
__all__ = ['file_move_safe']
def _samefile(src, dst):
# Macintosh, Unix.
if hasattr(os.path,'samefile'):
try:
return os.path.samefile(src, dst)
except OSError:
return False
# All other platforms: check for same pathname.
return (os.path.normcase(os.path.abspath(src)) ==
os.path.normcase(os.path.abspath(dst)))
def file_move_safe(old_file_name, new_file_name, chunk_size = 1024*64,
allow_overwrite=False):
"""
Moves a file from one location to another in the safest way possible.
First, tries ``os.rename``, which is simple but will break across
filesystems.
If that fails, streams manually from one file to another in pure Python.
If the destination file exists and ``allow_overwrite`` is ``False``, this
function will throw an ``IOError``.
"""
# There's no reason to move if we don't have to.
if _samefile(old_file_name, new_file_name):
return
try:
os.rename(old_file_name, new_file_name)
return
except OSError:
# This will happen with os.rename if moving to another filesystem
# or when moving opened files on certain operating systems
pass
# first open the old file, so that it won't go away
old_file = open(old_file_name, 'rb')
try:
# now open the new file, not forgetting allow_overwrite
fd = os.open(new_file_name, os.O_WRONLY | os.O_CREAT | getattr(os,
'O_BINARY', 0) |
(not allow_overwrite and os.O_EXCL or 0))
try:
locks.lock(fd, locks.LOCK_EX)
current_chunk = None
while current_chunk != '':
current_chunk = old_file.read(chunk_size)
os.write(fd, current_chunk)
finally:
registering unzipped files on the local disk
Hi, Can anyone help me with http://stackoverflow.com/questions/6386062/django-registering-unzipped-files-on-the-local-disk ? If you reply via email, please cc me. Thanks. Regards, Faheem. -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: using the stable api
Hi Malcolm, Thanks for the helpful reply. On Sat, 01 Aug 2009 12:24:29 +1000, Malcolm Tredinnick <malc...@pointy-stick.com> wrote: > On Fri, 2009-07-31 at 11:42 -0400, Faheem Mitha wrote: >> >> Hi everybody, >> >> I upgraded from somewhere around Django 1.0 to 1.0.2, and some things >> broke and had to be changed. In the following, f is an object of class >> 'django.core.files.uploadedfile.TemporaryUploadedFile'. I need to >> >> 1) Get the contents of the file corresponding to f. I had to change >> f['content'] to f._file.read() >> >> From the docs (specifically, 'core/files/uploadedfile.py') it looks like >> f.read() might be the right thing? > The correct way to access uploaded file data is documented here: > http://docs.djangoproject.com/en/dev/topics/http/file-uploads > /#handling-uploaded-files > That is the public API, and you are correct in thinking that read() is > the method you're after here. >> 2) Get the filename of the file corresponding to f. I had to change >> f['filename'] to f._name. > This is also documented in the above reference. Use the "name" attribute > on the UploadedFile isntance. Ok, I will change these. >> 3) Get the sessionid. I had to change request.COOKIES['sessionid'] >> to request.COOKIES[settings.SESSION_COOKIE_NAME] > The default value of settings.SESSION_COOKIE_NAME has always (and > remains) "sessionid". However, it's a little more portable to use > the latter form (request.COOKIES[settings.SESSION_COOKIE_NAME) in > your applications, since then they are usable no matter what the > setting happens to be. A Django application writer is not > necessarily in control of the settings that will be used when the > application is installed. To summarize, I've changed settings.SESSION_COOKIE_NAME from their default value, so this breaks request.COOKIES['sessionid'], and in this case, using request.COOKIES[settings.SESSION_COOKIE_NAME) is preferable, because it will work no matter what the value of settings.SESSION_COOKIE_NAME happens to be. Is that a correct summary? >> It is presumably better to use a stable API rather than less stable >> internals, > It is *always* better. :-) Ok. Thanks. Regards, Faheem Mitha. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
using the stable api
Hi everybody, I upgraded from somewhere around Django 1.0 to 1.0.2, and some things broke and had to be changed. In the following, f is an object of class 'django.core.files.uploadedfile.TemporaryUploadedFile'. I need to 1) Get the contents of the file corresponding to f. I had to change f['content'] to f._file.read() >From the docs (specifically, 'core/files/uploadedfile.py') it looks like f.read() might be the right thing? 2) Get the filename of the file corresponding to f. I had to change f['filename'] to f._name. 3) Get the sessionid. I had to change request.COOKIES['sessionid'] to request.COOKIES[settings.SESSION_COOKIE_NAME] It is presumably better to use a stable API rather than less stable internals, so my question is - what is the best expression to use in the examples above so that they are less likely to break on upgrade? Please CC me on any reply. Regards, Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
appeal for help with #9232
Hi, Can anyone help with http://code.djangoproject.com/ticket/9232 ? Currently I don't even know if this problem is reproducible. If someone could solve it, or even indicate what was wrong, if anything, I'd be happy to pay for services rendered. Thanks, Faheem --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: streaming file uploads not working with Django 1.0
On Sat, 27 Sep 2008 01:56:53 -0400 (EDT), Faheem Mitha <[EMAIL PROTECTED]> wrote: > Hi, > > I just upgraded to Django 1.0, and I've just finished fixing all my unit > tests. However, streaming file uploads are still broken. The behaviour I'm > seeing is as follows: > > When I start the upload of the file, a file which looks like eg. > /tmp/tmpGjWpee.upload appears almost immediately, but does not change in > size thereafter. The size is either empty or 16M. I haven't seen any other > values appear. The file upload completes some of the time, regardless, but > is very slow. For very large files, like 1g, it took so long I stopped it. > > A model I'm using for testing purposes appears below. > > I've seen this behavior on two independent installations of 1.0, with the > same project/application. I have a different installation with a pre-1.0 > snapshot with a patch from #2070 applied, and that works fine with an > earlier version of the same model/application, with the temporary file in > /tmp appearing and growing in size as the upload progresses, as expected. > > Suggestions for debugging would be appreciated. Please CC me on any reply. I've now created an issue for this, including a test case. http://code.djangoproject.com/ticket/9232 Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
streaming file uploads not working with Django 1.0
Hi, I just upgraded to Django 1.0, and I've just finished fixing all my unit tests. However, streaming file uploads are still broken. The behaviour I'm seeing is as follows: When I start the upload of the file, a file which looks like eg. /tmp/tmpGjWpee.upload appears almost immediately, but does not change in size thereafter. The size is either empty or 16M. I haven't seen any other values appear. The file upload completes some of the time, regardless, but is very slow. For very large files, like 1g, it took so long I stopped it. A model I'm using for testing purposes appears below. I've seen this behavior on two independent installations of 1.0, with the same project/application. I have a different installation with a pre-1.0 snapshot with a patch from #2070 applied, and that works fine with an earlier version of the same model/application, with the temporary file in /tmp appearing and growing in size as the upload progresses, as expected. Suggestions for debugging would be appreciated. Please CC me on any reply. Thanks. Sincerely, Faheem Mitha. class OldFileUpload(models.Model): upload_date = models.DateTimeField(default=datetime.now(), blank=True, editable=False) upload = models.FileField(upload_to=".") #name = models.CharField(core=True, max_length=100) name = models.CharField(max_length=100) description = models.CharField(blank=True, max_length=200) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Two Django visions running at once.
On Wed, 17 Sep 2008 13:21:50 -0700 (PDT), KillaBee <[EMAIL PROTECTED]> wrote: > Is it possible to have two versions of django running at the same time > on an Ubuntu server(9.0 and 1.0)? > It is taking to long to recode, and until I do I wanted It up and > running with the old version. > I got the tar.gz from the web site, but do I have to get rid of the > new version? I'm doing something like this. I'd recommend using a virtual machine setup. Something like Linux vservers is not difficult to setup and I've found the developers in IRC very helpful. Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: change file upload destination "on the fly" for tests
On Wed, 17 Sep 2008 14:42:40 -0400 (EDT), Faheem Mitha
<[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> I wrote some unit tests for file upload. since I didn't want the files in
> the unit tests to be uploaded to the "official locaion", I changed the
> upload location by reassiging MEDIA_ROOT to something else, '/tmp' in this
> case.
>
> With an upgrade to Django 1.0, this method no longer works. The file
> upload appears to ignore the value of MEDIA_ROOT.
>
> Can anyone suggest another method? Apparently file upload is now more
> flexible - vide http://code.djangoproject.com/wiki/FileStorageRefactor.
> I'm not sure if it is possible to use this kind of functionality to do
> this. Suggestions welcome.
I'm following up on my own message. The following approach works:
***
customstorage.py
***
import os
from django.core.files.storage import FileSystemStorage
from django.utils._os import safe_join
from django.conf import settings
class FileUploadStorage(FileSystemStorage):
def path(self, name):
print "calling FileUploadStorage class."
try:
path = safe_join(settings.MEDIA_ROOT, name)
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
return os.path.normpath(path)
***
Then in settings.py use
DEFAULT_FILE_STORAGE = "customstorage.FileUploadStorage"
Basically, overwrite the path in FileSystemStorage so it no longer
uses self.location, but instead looks up settings.MEDIA_ROOT when path
is called.
This is just a hack. I think it would be reasonable to make location
in FileSystemStorage a callable in similar fashion to the upload_to
argument in FileField. If I understand correctly, that would be make
this hack redundant. Comments appreciated - please cc me.
Faheem.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
Re: unable to import Storage from django.core.files.storage into settings.py
On Thu, 18 Sep 2008 12:16:52 -0400, Marty Alchin <[EMAIL PROTECTED]> wrote:
>
> On Thu, Sep 18, 2008 at 11:55 AM, Faheem Mitha <[EMAIL PROTECTED]> wrote:
>> Putting the following into my settings.py (checked for two different
>> projects for Dango 1.0)
>>
>> from django.core.files.storage import Storage
>>
>> gives
>>
>> [EMAIL PROTECTED]:/var/django/hg$ python manage.py validate
>> Error: Can't find the file 'settings.py' in the directory containing
>> 'manage.py'. It appears you've customized things.
>> You'll have to run django-admin.py, passing it your settings module.
>> (If the file settings.py does indeed exist, it's causing an ImportError
>> somehow.)
>>
>> Is there some reason this is not possible, or am I doing something wrong?
>> Can anyone reproduce this? Please cc me on any reply.
>
> Line 5 of django.core.files.storage looks like this:
>
> from django.conf import settings
>
> That means that your settings module imports Storage, which in turn
> imports settings. This is known as a circular import, which is never a
> good thing. So, yes, you're doing something wrong, because that
> shouldn't work.
>
> Now, the real question is: why are you trying to import Storage in
> your settings.py? What is it you're trying to accomplish? You're
> certainly using the wrong approach at the moment, but I can't
> recommend anything else without knowing what you're trying to do.
I was trying to do something like (in settings.py)
***
from customstorage import FileSystemStorage
DEFAULT_FILE_STORAGE = FileSystemStorage
***
where customstorage.py looks as below. This was failing (I think)
because the following line in customstorage.py failed to be imported
into settings.py
from django.core.files.storage import FileSystemStorage
I then belatedly realised that that the settings are strings, and did
DEFAULT_FILE_STORAGE = "customstorage.FileUploadStorage"
instead, which works.
Thanks for the reply and clarification, and sorry for the noise. If
there is anything I missed, please feel free to educate me further.
Thanks, Faheem.
***
customstorage.py
***
import os
from django.core.files.storage import FileSystemStorage
from django.utils._os import safe_join
from django.conf import settings
class FileUploadStorage(FileSystemStorage):
def path(self, name):
print "calling FileUploadStorage class."
try:
path = safe_join(settings.MEDIA_ROOT, name)
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
return os.path.normpath(path)
**
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
unable to import Storage from django.core.files.storage into settings.py
Hi, Putting the following into my settings.py (checked for two different projects for Dango 1.0) from django.core.files.storage import Storage gives [EMAIL PROTECTED]:/var/django/hg$ python manage.py validate Error: Can't find the file 'settings.py' in the directory containing 'manage.py'. It appears you've customized things. You'll have to run django-admin.py, passing it your settings module. (If the file settings.py does indeed exist, it's causing an ImportError somehow.) Is there some reason this is not possible, or am I doing something wrong? Can anyone reproduce this? Please cc me on any reply. Thanks, Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: change file upload destination "on the fly" for tests
On Wed, 17 Sep 2008 20:52:16 -0700 (PDT), akonsu <[EMAIL PROTECTED]> wrote: > > ok :) so then your solution would be to set up your own file storage > with location that you need. > > konstantin Sounds nice in theory, but I'm having trouble figuring out how to change the location dynamically, even using a file storage object. Faheem. > On Sep 17, 11:45 pm, Faheem Mitha <[EMAIL PROTECTED]> wrote: >> [This message has also been posted.] >> >> On Wed, 17 Sep 2008 20:14:54 -0700 (PDT), akonsu <[EMAIL PROTECTED]> wrote: >> > hm, i never used a callable for uplod_to and what you are saying >> > might be true, but if it is, how this behaviour would interact with >> > the possibility to use file storage for uploaded files where >> > MEDIA_ROOT would make little sense? >> > konstantin >> >> Judging by the documentation, the location (top level directory >> holding the files) is passed as an argument to the file storage >> object. In other words, the file storage object determines what the >> top level directory is. The upload_to argument corresponds to the >> subdirectory where the file is placed. >> >> I could not understand the code, so this summary may be incorrect. >> >> Faheem. >> >> > On Sep 17, 11:00 pm, Faheem Mitha <[EMAIL PROTECTED]> wrote: >> >> >> Thanks for the suggestion. However, upload_to is appended to >> >> settings.MEDIA_ROOT to determine the path the file is uploaded >> >> to. Therefore, it is not possible to (for example) change upload_to so >> >> that files are uploaded to '/tmp'. >> >> Faheem. > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: change file upload destination "on the fly" for tests
On Wed, 17 Sep 2008 20:14:54 -0700 (PDT), akonsu <[EMAIL PROTECTED]> wrote: > hm, i never used a callable for uplod_to and what you are saying > might be true, but if it is, how this behaviour would interact with > the possibility to use file storage for uploaded files where > MEDIA_ROOT would make little sense? > konstantin Judging by the documentation, the location (top level directory holding the files) is passed as an argument to the file storage object. In other words, the file storage object determines what the top level directory is. The upload_to argument corresponds to the subdirectory where the file is placed. I could not understand the code, so this summary may be incorrect. Faheem. > On Sep 17, 11:00 pm, Faheem Mitha <[EMAIL PROTECTED]> wrote: >> >> Thanks for the suggestion. However, upload_to is appended to >> settings.MEDIA_ROOT to determine the path the file is uploaded >> to. Therefore, it is not possible to (for example) change upload_to so >> that files are uploaded to '/tmp'. >> Faheem. > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: change file upload destination "on the fly" for tests
[This message has also been posted.] On Wed, 17 Sep 2008 18:40:12 -0700 (PDT), akonsu <[EMAIL PROTECTED]> wrote: > hello, > upload_to can take a callable, which can be used to change your > files' location. > konstantin Thanks for the suggestion. However, upload_to is appended to settings.MEDIA_ROOT to determine the path the file is uploaded to. Therefore, it is not possible to (for example) change upload_to so that files are uploaded to '/tmp'. Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: change file upload destination "on the fly" for tests
Hi Lisa,
Thanks for the reply.
On Wed, 17 Sep 2008 14:28:59 -0700, Lisa Dusseault
<[EMAIL PROTECTED]> wrote:
> --=_Part_17268_18727299.1221686939556
> Content-Type: text/plain; charset=ISO-8859-1
>
> I have a bunch of unit tests that do file upload too. E.g.:
>
> def testBasics(self):
> testpage = self.test_report.page_set.create(slug="testpage", title =
> "Test Page")
> self.assertEquals(testpage.report, self.test_report)
> testpage.page_spec.save("testgraphspec.json",
> ContentFile(TESTSPECSTR))
> graphspecs = testpage.graphspecs()
> assert "age1" in graphspecs.keys()
> self.assertEquals(graphspecs["age1"].slug, "age1")
> I have no idea if this is the best way or not, but it seems that the
> django unit test magic does the right thing here, and the file that
> is created upon upload gets cleaned up afterward, without any need
> to change the MEDIA location or do other painful things.
Well, hopefully it doesn't need to be that painful. :-) I'm surprised
the files get removed automatically, if that's what you mean. You are
not deleting them after the tests are run? One reason I don't want
the test files uploaded to the same directory as the regular files is
that I check to see whether the files uploaded to the location are the
correct ones, which is impossible if there are other files there
already. In general it is just cleaner, I think.
> Are you using django's "manage.py test" to run the tests?
Yes.
Faheem.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
change file upload destination "on the fly" for tests
Hi, I wrote some unit tests for file upload. since I didn't want the files in the unit tests to be uploaded to the "official locaion", I changed the upload location by reassiging MEDIA_ROOT to something else, '/tmp' in this case. With an upgrade to Django 1.0, this method no longer works. The file upload appears to ignore the value of MEDIA_ROOT. Can anyone suggest another method? Apparently file upload is now more flexible - vide http://code.djangoproject.com/wiki/FileStorageRefactor. I'm not sure if it is possible to use this kind of functionality to do this. Suggestions welcome. Thanks, Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: wsgi in django for hgwebproxy
On Fri, 22 Aug 2008, Dirkjan Ochtman wrote:
> Faheem Mitha wrote:
>> *
>> self.env = req.META where req is a django request (see__init__ of
>> hgReqWrap below).
>> *
>>
>> self.env is {'AUTH_TYPE': None, 'HTTP_COOKIE':
>> 'bixfile=d1e2ea28f5cc1d4f43a9a14e0db4970f', 'SERVER_SOFTWARE':
>> 'mod_python', 'SCRIPT_NAME':
>> None, 'REQUEST_METHOD': 'GET', 'PATH_INFO': '', 'SERVER_PROTOCOL':
>> 'HTTP/1.1', 'QUERY_STRING': None, 'CONTENT_LENGTH': 0L,
>> 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'REMOTE_USER':
>> None, 'HTTP_CONNECTION': 'keep-alive', 'SERVER_NAME':
>> 'msi.home.earth', 'REMOTE_ADDR': '192.168.1.204', 'P ATH_TRANSLATED': None,
>> 'SERVER_PORT': 443, 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; U;
>> Linux i686; en-US; rv:1.8. 0.14eol) Gecko/20070505
>> (Debian-1.8.0.15~pre080614d-0etch1) Epiphany/2.14', 'HTTP_HOST': 'msi',
>> 'HTTP_CACHE_CONTROL': 'max-age=0', 'HTTP_ACCEPT':
>> 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain
>> ;q=0.8,image/png,*/*;q=0.5', 'GATEWAY_INTERFACE': 'CGI/1.1',
>> 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'REMOTE_IDENT': None,
>> 'CONTENT_TYPE':
>> None, 'REMOTE_HOST': None, 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
>> 'HTTP_KEEP_A LIVE': '300'}
>>
>>
>
> This are just some of the common HTTP environment keys, but not the whole
> WSGI environ. You need the latter to pass to hgweb, because hgweb is a WSGI
> application (expecting WSGI requests: environ, start_response).
Hi Dirkjan,
Thanks for the helpful comment, and taking the time to respond to me on
IRC. See excerpts from conversation below.
High level summary: It seems like it should be possible to have django do
proxying for hgweb, but as yet, I haven't seen clear instructions for
doing so. The first question I'd like to have a clear answer to is
Does the default Django request for current versions of Django have wsgi
stuff in it, or not? My version doesn't, and I've seen no indication of
whether more recent versions do. Jespern's code appears to rely on that.
Faheem.
*
10:54 < djc> faheem: did you manage to get the django stuff fixed?
10:55 < faheem> djc: No, I haven't fixed the django thing. I'm replying to
your message right now, actually.
10:59 < faheem> djc: Do you think it likely that more recent django
actually has this 'wsgi.input' and stuff built into the response? jespern
seems to think so, but I haven't tried upgrading.
11:00 < djc> well, I think Django should have some facility for "mounting"
other WSGI apps
11:00 < djc> and it probably has the full WSGI environ somewhere
11:00 < djc> req.META is just not what you want
11:00 < faheem> djc: You think jespern's approach is wrong then?
11:01 < djc> oh, I don't really know about his approach
11:01 < faheem> djc: I can't find much info about wsgi + django.
11:01 < djc> what version are you toying with?
11:01 < faheem> djc: There is a wsgi.py.
11:01 < faheem> djc: Snapshot from late may. Was planning to wait for 1.0,
but could try the beta I suppose.
11:02 < faheem> djc: But the wsgi.py has pretty much zilch documentation.
I asked on #django and Magus said that people didn't used wsgi that
much. I thought it was an upcoming standard.
11:03 < djc> interesting thing to say
11:04 < faheem> djc: How so?
11:04 < djc> where's the wsgi.py at?
11:05 < djc> because if you ask me, WSGI is the canonical way to put
Python apps behind web servers
11:05 < djc> and almost everything plays nice with it
11:05 < djc> that I know of
11:05 < faheem> djc: Sec. I'll check.
11:06 < djc> ah, in core.handlers, right?
11:06 < faheem> djc: django/core/handlers
11:06 < faheem> djc: Right.
11:06 < djc> okay, so you can just use req.environ, then
11:07 < faheem> djc: Can you recommend some wsgi docs?
11:07 < faheem> djc: Can you elaborate?
11:07 < djc> the spec is easy to read
11:07 < djc> so, how are you serving this django site?
11:08 < faheem> http://www.wsgi.org/wsgi/Specifications ?
11:08 < faheem> djc: Apache + mod python.
11:08 < djc> I usually read the PEP version
11:08 < faheem> djc: Pretty standard setup.
11:08 < djc> but it's probably the same thing
11:08 < faheem> djc: http://www.python.org/dev/peps/pep-0333/ ?
11:08 < djc> okay, so it won't work with the default mod_python setup
11:08 < dj
wsgi in django for hgwebproxy
Hello everyone,
The version control system mercurial (hg) has some functionality to allow
its repositories to be viewed via the web. However, it does not have a
builtin authentication system.
I'm trying to get a setup working where I can put Django in front. Jesper
Noehr of bitbucket.org has some code that looks like it would do what I
want. Basically, it has Django act as a proxy (I hope I have the
terminology right), which sents a request to the hgweb code, and gets back
a response object.
However, this code is currently giving errors, and they are at the Django
end, so I'm CCing here in the hope that someone can set me straight about
what is going on.
I'm posting the view code from the hgwebproxy app in
https://bitbucket.org/jespern/cx/.
This basically does everything. I then follow it with the error
message I see in Django. The error message comes from the line
self.inp = self.env['wsgi.input']
Jesper thinks that a generic Django request should contain such keys, but
the request I'm seeing doesn't.
He thinks I need to upgrade Django, but before I do that, I thought I
would write and ask if the behavior has really changed so much in recent
versions. I've currently using a version from late May, and was waiting
for 1.0 to upgrade.
I looked at my current version, and it has a wsgi.py, though I have no
idea how to use it, and could not find any documentation.
>From what I understand from Jesper, _hgReqWrap wraps the Django
request object into the sort of request that hgwebdir
expects. However, I thought the whole point of wsgi was to provide a
low-level compatability layer between different Python web
applications, so shouldn't these kind of manipulations be unnecessary?
To summarize, I'm asking two things
a) What is the simplest/most direct way to fix the current code?
b) Is there a simpler way to do the same thing? Ie. get django to
proxy for hgweb?
Please CC me on any reply.
Thanks, Faheem.
*
self.env = req.META where req is a django request (see__init__ of
hgReqWrap below).
*
self.env is {'AUTH_TYPE': None, 'HTTP_COOKIE':
'bixfile=d1e2ea28f5cc1d4f43a9a14e0db4970f', 'SERVER_SOFTWARE':
'mod_python', 'SCRIPT_NAME':
None, 'REQUEST_METHOD': 'GET', 'PATH_INFO': '', 'SERVER_PROTOCOL':
'HTTP/1.1', 'QUERY_STRING': None, 'CONTENT_LENGTH': 0L,
'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'REMOTE_USER':
None, 'HTTP_CONNECTION': 'keep-alive', 'SERVER_NAME':
'msi.home.earth', 'REMOTE_ADDR': '192.168.1.204', 'P ATH_TRANSLATED':
None, 'SERVER_PORT': 443, 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; U;
Linux i686; en-US; rv:1.8. 0.14eol) Gecko/20070505
(Debian-1.8.0.15~pre080614d-0etch1) Epiphany/2.14', 'HTTP_HOST': 'msi',
'HTTP_CACHE_CONTROL': 'max-age=0', 'HTTP_ACCEPT':
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain
;q=0.8,image/png,*/*;q=0.5', 'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'REMOTE_IDENT': None,
'CONTENT_TYPE':
None, 'REMOTE_HOST': None, 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
'HTTP_KEEP_A LIVE': '300'}
views.py
import os, re, cgi
from django.http import HttpResponseRedirect, HttpResponse
from django.conf import settings
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.contrib.auth.models import User
from datetime import datetime
from mercurial.hgweb.hgwebdir_mod import hgwebdir
from mercurial import hg, ui
from mercurial import __version__
try:
from hashlib import md5 as md5
except ImportError:
from md5 import md5
class _hgReqWrap(object):
def __init__(self, req, resp):
self.django_req = req
self.env = req.META
self.response = resp
# Remove the prefix so HG will think it's running on its own.
self.env['PATH_INFO'] = self.env['PATH_INFO'].lstrip("/hg")
# Make sure there's a content-length.
if not self.env.has_key('CONTENT_LENGTH'):
self.env['CONTENT_LENGTH'] = 0
self.inp = self.env['wsgi.input']
self.form = cgi.parse(self.inp, self.env, keep_blank_values=1)
self.headers = [ ]
self.err = self.env['wsgi.errors']
self.out = [ ]
def set_user(self, username):
self.env['REMOTE_USER'] = username
def read(self, count=-1):
return self.inp.read(count)
def flush(self):
return None
def respond(self, code, content_type=None, path=None, length=0):
self.response.status_code = code
self.response['content-type'] = content_type
if path is not None and
Re: using Django unit test framework with existing production db
On Fri, 28 Mar 2008 17:10:28 -0400 (EDT), Jeff Gentry <[EMAIL PROTECTED]> wrote: >> Is there some graceful way to have the Django unit test framework >> access the "production database" for certain tests as opposed to >> creating a test db? The idea is to do a basic sanity check of the >> "production db" in case the schema has changed and the db needs to be >> rebuilt. > > http://groups.google.com/group/django-users/browse_thread/thread/e323bd3a3d7bc5ef/83995eb20dba8c27?hl=en=st=django-users+jgentry#83995eb20dba8c27 Thanks, that's very helpful. Good to know I'm not the only person with this desire. Did you manage to get a custom setup as suggested? If so, could you share it with me? Thanks, Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
using Django unit test framework with existing production db
Hi, Is there some graceful way to have the Django unit test framework access the "production database" for certain tests as opposed to creating a test db? The idea is to do a basic sanity check of the "production db" in case the schema has changed and the db needs to be rebuilt. Please CC me on any reply. Thanks, Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
dynamic url use case
Hi everyone,
I have the following situation, and I would value advice.
I have two classes, FolderUpload and FileUpload as given below. I want
a dynamic upload_to url in FileUpload, such that the upload to field
depends on the value of the folder field, specifically path field. I
realist that currently some workaround is necessary, since Django does
not yet support dynamic upload_to fields. I found the description of
some workarounds, but could not get any of them to work in the
described scenario. I'm hoping that someone will have dealt with a
similar situation, and will have some ideas.
Please cc me by email on any response.
Thanks, Faheem.
*
class FolderUpload(models.Model):
upload_date = models.DateTimeField(auto_now_add=True)
name = models.CharField(core=True, maxlength=100)
description = models.CharField(blank=True, maxlength=200)
parent_folder = models.ForeignKey('self', null=True, blank=True,
related_name='subfolders')
path = models.FilePathField(path='', editable=False)
class FileUpload(models.Model):
upload_date = models.DateTimeField(default=datetime.now(), blank=True,
editable=False)
upload = models.FileField(upload_to=settings.BIXFILE_MEDIA)
name = models.CharField(core=True, maxlength=100)
description = models.CharField(blank=True, maxlength=200)
folder = models.ForeignKey(FolderUpload, null=True, blank=True,
related_name='parentfolder', edit_inline=models.TABULAR)
**
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
media directory for testing
Hi. I'm having the following problem while testing. I'm writing a unit test for a file upload (using newforms). This works. However, the file gets written to the same media directory as is used in normal work. I'd prefer this happened somewhere where it would have no impact on normal functioning, possibly /tmp. Does anyone have suggestions on what to do? One workaround would be to redefine the media directory in the tests themselves, if I can figure out how to do that... Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
testing filesystem operations in Django
Hi, Here is a simple question about testing. I have a django file upload application which needs to operate on the filesystem as part of its functionality (create files/directories etc. under media root) I use Debian etch. So, I want a way to test filesystem operations, preferably without using an actual filesystem to avoid possible damage from a test run amuck. I was wondering how people are handling this in a test framework setup. Any ideas about this? I was thinking of possibly using a ramdisk (filesystem in RAM), but I don't know how easy or convenient it would be. Please cc me on any reply. Thanks, Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Django issue #4165
Hi,
I've been unable to get the most recent patch from #4165
(http://code.djangoproject.com/ticket/4165) to work. If anyone suggest
what I might be doing wrong, or provide a minimal working example, that
would be most helpful.
My understanding is that #4165 should work more or less out of the box
with the default admin view, once a few configurations have been made.
Specifically, I should see an upload progress bar in a model which
contains eg a FileField when I try to upload something. Note also that JS
is enabled in the browsers I'm using -- I tried a couple of different
ones.
So, I have followed the most recent instructions on the wiki as far as I
was able. See below.
The model I am using is
**
begin extract from models.py
**
class FileUpload(models.Model):
## Workaround for http://code.djangoproject.com/ticket/1030. See end of
issue.
## Faheem Mitha October 2nd 2007.
upload_date = models.DateTimeField(default=datetime.now(), blank=True,
editable=False)
upload = models.FileField(upload_to="bixfile")
name = models.CharField(core=True, maxlength=100)
description = models.CharField(blank=True, maxlength=200)
folder = models.ForeignKey(FolderUpload, null=True, blank=True,
related_name='parentfolder', edit_inline=models.TABULAR)
class Admin:
pass
**
end extract from models.py
**
I checked that a table called uploadprogress_fileprogress is created in the db,
as see below
***
begin sqlite output
***
sqlite> .dump uploadprogress_fileprogress
BEGIN TRANSACTION;
CREATE TABLE "uploadprogress_fileprogress" (
"id" integer NOT NULL PRIMARY KEY,
"remote_addr" varchar(64) NOT NULL,
"progress_id" varchar(32) NOT NULL,
"progress_text" text NOT NULL,
"last_ts" datetime NOT NULL,
UNIQUE ("remote_addr", "progress_id")
);
***
end sqlite output
***
I see the expected listing in the admin view.
**
begin admin listing for add file progress
**
Add file progress
Remote addr:
Progress id:
Last ts:
Date: Today | Calendar
Time: Now | Clock
**
begin admin listing for add file progress
**
However, there is nothing corresponding to a progress bar in FileUpload.
What am I doing wrong, if anything? If you don't have the time or
inclination to debug my setup, could you provide a minimal working
example?
Thanks, Faheem.
*
begin installation notes
*
> INSTALLATION
> The above docs don't really work. Here's a revised version.
> 1. Add #2070 to your source tree
Used 6469_streaming_file_upload.diff as uploaded to
http://code.djangoproject.com/ticket/2070. When tested separately,
this seems to work fine.
> 2. Add the latest patch (of this ticket) to your source tree.
Patched against svn 6469 + 6469_streaming_file_upload.diff, and added an empty
django/contrib/uploadprogress/__init__.py.
> 3. Setup your middleware/models to track file progress.
Did the following as detailed below:
* Added 'django.contrib.uploadprogress.middleware.FileProgressDB' to
my MIDDLEWARE_CLASSES.
* Added 'django.contrib.uploadprogress' to my INSTALLED_APPS.
* Issued a ./manage.py syncdb
> 4. Set FILE_UPLOAD_DIR to a writable folder in your settings.py file.
Set this as
FILE_UPLOAD_DIR='/tmp'.
> For (3), there are two options that come in the patch (as uploadprogress
> contrib):
> * FileProgressCached -- uses the cache framework for quicker ajax (will
> NOT work if your cache framework doesn't)
> * FileProgressDB -- uses Database models
> Depending on what you choose, continue below.
See above.
> FileProgressCached
> To enable this, add
> 'django.contrib.uploadprogress.middleware.FileProgressCached' to your
> MIDDLEWARE_CLASSES.
> FileProgressDB
> To enable this:
> * Add 'djan
file upload/download management in Django
Dear Django Users, I've been given the task of developing a web application at work. This is for uploading/downloading large data files, specifically microarray data files for bioinformatics work. I've been looking at Django and managed to get some basic framework stuff working, using the 0.96 package on Debian etch. I've also put Django behind Apache. I had some questions, which I hope are suitable for this mailing list. 1) I've noticed that the Django admin interface has the property that once a user has logged in, he can then connnect to the interface without logging in again if he (for example) closes the browser window. This may be a configuration error or something I have not enabled. In any case, this seems to me undesirable behaviour, so I would appreciate clarification. 2) How can one configure the login so that a user does not see the admin interface but goes directly to the application page? I am sure this is a straightfoward configuration. 3) Is it possible (with the admin interface) to have per folder permissions, so that only certain groups can access (read, write) a particular folder? Also, we're looking for a suitable application that can be used for file upload/download management. There appear to be a number of different candidates available, so I'd appreciate suggestions. The features we are looking for are a) Multiple upload: Specifically, this means that one can select multiple files within a single browse window by dragging the mouse (or something similar). b) A progress bar for uploads: c) Works with large file uploads: I'm not sure if this is even an issue. Is the issue wrt large streaming uploads described in http://code.djangoproject.com/ticket/2070 only a problem if the system in question does not have enough memory to keep the file in memory while it is being uploaded? Please cc me on any reply. I'll be checking via gmane in any case. Thanks, Faheem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
