Re: S3 tutorial for File Storage Refactor?
Hi Jay, Thanks for the patch, I need to spend more time on this but I had pushed your correction. Any feedback is really appreciated. Best, David Le 2 sept. 08 à 18:22, shadfc a écrit : > To fix the corrupt images error, add > content.open() > to the top of _save() on the S3Storage class (or at least before the > chunks() or read() calls). This basically causes a seek(0) on > content. I think perhaps some of the image validation reads part of > the file and doesn't reset the pointer back to the beginning. > > Jay > > On Sep 1, 10:40 am, Ramdas S <[EMAIL PROTECTED]> wrote: >> David, >> >> I think this is still not fixed. I am getting the same errors >> >> R >> >> On Aug 25, 2:02 pm, David Larlet <[EMAIL PROTECTED]> wrote: >> >>> Le 20 août 08 à 17:56, shadfc a écrit : >> With the code from the django-storages you referenced installed somewhere on PYTHONPATH, its as easy as setting a few things in your settings.py. You can see the docs for the code at http://code.larlet.fr/doc/django-s3-storage.html. Put the Required and Optional (if you want it, obviously) stuff in your settings.py and fill in the appropriate values for yourS3account. >> Now, assuming those settings are all correct, all of your FileFields (and thus, ImageFields) should store toS3into the bucket you set. You can continue to use upload_to to prefix the filename within the bucket. >> Now, I've noticed a few bugs, both of which I've notified David (the author) of: 1) if you call object.filefield.size, this code will download the entire file fromS3just to calculate the size. This happens because of the construction of the _open() method on the S3Storage class. It should not make theS3get call (which downloads the file). The workaround for now is not to use the size property with this code, but the more permanent fix is to delay reading of the file fromS3until read() is called on a S3StorageFile object. 2) I cannot get images to store correctly toS3when using an ImageField. They appear in the bucket with a small filesize change, but no software I have will recognize them as valid images. Storing images (and any other file) via a FileField works just fine and the files are not corrupted. Strangely, storing images via ImageField on the FileSystemStorage backend works just fine, so it only seems to be the combination of an ImageField while using thisS3backend that is the problem. I am not sure where the bug in this is. If you give it a try, see if you can verify this bug for me. >> Jay >> >>> Hi Jay, >> >>> I plan to work on this tonight because that's clearly a blocking >>> point. Thanks for reporting those bugs. >> >>> Best, >>> David --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: S3 tutorial for File Storage Refactor?
To fix the corrupt images error, add content.open() to the top of _save() on the S3Storage class (or at least before the chunks() or read() calls). This basically causes a seek(0) on content. I think perhaps some of the image validation reads part of the file and doesn't reset the pointer back to the beginning. Jay On Sep 1, 10:40 am, Ramdas S <[EMAIL PROTECTED]> wrote: > David, > > I think this is still not fixed. I am getting the same errors > > R > > On Aug 25, 2:02 pm, David Larlet <[EMAIL PROTECTED]> wrote: > > > Le 20 août 08 à 17:56, shadfc a écrit : > > > > With the code from the django-storages you referenced installed > > > somewhere on PYTHONPATH, its as easy as setting a few things in your > > > settings.py. You can see the docs for the code at > > >http://code.larlet.fr/doc/django-s3-storage.html. Put the Required > > > and Optional (if you want it, obviously) stuff in your settings.py and > > > fill in the appropriate values for yourS3account. > > > > Now, assuming those settings are all correct, all of your FileFields > > > (and thus, ImageFields) should store toS3into the bucket you set. > > > You can continue to use upload_to to prefix the filename within the > > > bucket. > > > > Now, I've noticed a few bugs, both of which I've notified David (the > > > author) of: > > > 1) if you call object.filefield.size, this code will download the > > > entire file fromS3just to calculate the size. This happens because > > > of the construction of the _open() method on the S3Storage class. It > > > should not make theS3get call (which downloads the file). The > > > workaround for now is not to use the size property with this code, but > > > the more permanent fix is to delay reading of the file fromS3until > > > read() is called on a S3StorageFile object. > > > 2) I cannot get images to store correctly toS3when using an > > > ImageField. They appear in the bucket with a small filesize change, > > > but no software I have will recognize them as valid images. Storing > > > images (and any other file) via a FileField works just fine and the > > > files are not corrupted. Strangely, storing images via ImageField on > > > the FileSystemStorage backend works just fine, so it only seems to be > > > the combination of an ImageField while using thisS3backend that is > > > the problem. I am not sure where the bug in this is. If you give it > > > a try, see if you can verify this bug for me. > > > > Jay > > > Hi Jay, > > > I plan to work on this tonight because that's clearly a blocking > > point. Thanks for reporting those bugs. > > > Best, > > David --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: S3 tutorial for File Storage Refactor?
David, I think this is still not fixed. I am getting the same errors R On Aug 25, 2:02 pm, David Larlet <[EMAIL PROTECTED]> wrote: > Le 20 août 08 à 17:56, shadfc a écrit : > > > > > > > With the code from the django-storages you referenced installed > > somewhere on PYTHONPATH, its as easy as setting a few things in your > > settings.py. You can see the docs for the code at > >http://code.larlet.fr/doc/django-s3-storage.html. Put the Required > > and Optional (if you want it, obviously) stuff in your settings.py and > > fill in the appropriate values for yourS3account. > > > Now, assuming those settings are all correct, all of your FileFields > > (and thus, ImageFields) should store toS3into the bucket you set. > > You can continue to use upload_to to prefix the filename within the > > bucket. > > > Now, I've noticed a few bugs, both of which I've notified David (the > > author) of: > > 1) if you call object.filefield.size, this code will download the > > entire file fromS3just to calculate the size. This happens because > > of the construction of the _open() method on the S3Storage class. It > > should not make theS3get call (which downloads the file). The > > workaround for now is not to use the size property with this code, but > > the more permanent fix is to delay reading of the file fromS3until > > read() is called on a S3StorageFile object. > > 2) I cannot get images to store correctly toS3when using an > > ImageField. They appear in the bucket with a small filesize change, > > but no software I have will recognize them as valid images. Storing > > images (and any other file) via a FileField works just fine and the > > files are not corrupted. Strangely, storing images via ImageField on > > the FileSystemStorage backend works just fine, so it only seems to be > > the combination of an ImageField while using thisS3backend that is > > the problem. I am not sure where the bug in this is. If you give it > > a try, see if you can verify this bug for me. > > > Jay > > Hi Jay, > > I plan to work on this tonight because that's clearly a blocking > point. Thanks for reporting those bugs. > > Best, > David --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: S3 tutorial for File Storage Refactor?
Le 20 août 08 à 17:56, shadfc a écrit : > > With the code from the django-storages you referenced installed > somewhere on PYTHONPATH, its as easy as setting a few things in your > settings.py. You can see the docs for the code at > http://code.larlet.fr/doc/django-s3-storage.html. Put the Required > and Optional (if you want it, obviously) stuff in your settings.py and > fill in the appropriate values for your S3 account. > > Now, assuming those settings are all correct, all of your FileFields > (and thus, ImageFields) should store to S3 into the bucket you set. > You can continue to use upload_to to prefix the filename within the > bucket. > > Now, I've noticed a few bugs, both of which I've notified David (the > author) of: > 1) if you call object.filefield.size, this code will download the > entire file from S3 just to calculate the size. This happens because > of the construction of the _open() method on the S3Storage class. It > should not make the S3 get call (which downloads the file). The > workaround for now is not to use the size property with this code, but > the more permanent fix is to delay reading of the file from S3 until > read() is called on a S3StorageFile object. > 2) I cannot get images to store correctly to S3 when using an > ImageField. They appear in the bucket with a small filesize change, > but no software I have will recognize them as valid images. Storing > images (and any other file) via a FileField works just fine and the > files are not corrupted. Strangely, storing images via ImageField on > the FileSystemStorage backend works just fine, so it only seems to be > the combination of an ImageField while using this S3 backend that is > the problem. I am not sure where the bug in this is. If you give it > a try, see if you can verify this bug for me. > > Jay Hi Jay, I plan to work on this tonight because that's clearly a blocking point. Thanks for reporting those bugs. Best, David --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: S3 tutorial for File Storage Refactor?
I don't think its as simple as that. As I said, "no software I have will recognize them as valid images". This includes GIMP, Eye of Gnome, etc. All say that it is not a valid image or is corrupted. Jay On Aug 20, 12:41 pm, Martin Diers <[EMAIL PROTECTED]> wrote: > I would suspect that #2 is a mime-type problem, which is just an S3 > property. Presuming the ImageField translates to a basic href into S3, > the mime-type property of the file would need to be set or else it is > going to come across as a Binary file. Some browsers are ok with this > non-standard behavior. Others are not, and will attempt to download > the file instead of display it. > > On Aug 20, 2008, at 10:56 AM, shadfc wrote: > > > > > With the code from the django-storages you referenced installed > > somewhere on PYTHONPATH, its as easy as setting a few things in your > > settings.py. You can see the docs for the code at > >http://code.larlet.fr/doc/django-s3-storage.html. Put the Required > > and Optional (if you want it, obviously) stuff in your settings.py and > > fill in the appropriate values for your S3 account. > > > Now, assuming those settings are all correct, all of your FileFields > > (and thus, ImageFields) should store to S3 into the bucket you set. > > You can continue to use upload_to to prefix the filename within the > > bucket. > > > Now, I've noticed a few bugs, both of which I've notified David (the > > author) of: > > 1) if you call object.filefield.size, this code will download the > > entire file from S3 just to calculate the size. This happens because > > of the construction of the _open() method on the S3Storage class. It > > should not make the S3 get call (which downloads the file). The > > workaround for now is not to use the size property with this code, but > > the more permanent fix is to delay reading of the file from S3 until > > read() is called on a S3StorageFile object. > > 2) I cannot get images to store correctly to S3 when using an > > ImageField. They appear in the bucket with a small filesize change, > > but no software I have will recognize them as valid images. Storing > > images (and any other file) via a FileField works just fine and the > > files are not corrupted. Strangely, storing images via ImageField on > > the FileSystemStorage backend works just fine, so it only seems to be > > the combination of an ImageField while using this S3 backend that is > > the problem. I am not sure where the bug in this is. If you give it > > a try, see if you can verify this bug for me. > > > Jay --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: S3 tutorial for File Storage Refactor?
I would suspect that #2 is a mime-type problem, which is just an S3 property. Presuming the ImageField translates to a basic href into S3, the mime-type property of the file would need to be set or else it is going to come across as a Binary file. Some browsers are ok with this non-standard behavior. Others are not, and will attempt to download the file instead of display it. On Aug 20, 2008, at 10:56 AM, shadfc wrote: > > With the code from the django-storages you referenced installed > somewhere on PYTHONPATH, its as easy as setting a few things in your > settings.py. You can see the docs for the code at > http://code.larlet.fr/doc/django-s3-storage.html. Put the Required > and Optional (if you want it, obviously) stuff in your settings.py and > fill in the appropriate values for your S3 account. > > Now, assuming those settings are all correct, all of your FileFields > (and thus, ImageFields) should store to S3 into the bucket you set. > You can continue to use upload_to to prefix the filename within the > bucket. > > Now, I've noticed a few bugs, both of which I've notified David (the > author) of: > 1) if you call object.filefield.size, this code will download the > entire file from S3 just to calculate the size. This happens because > of the construction of the _open() method on the S3Storage class. It > should not make the S3 get call (which downloads the file). The > workaround for now is not to use the size property with this code, but > the more permanent fix is to delay reading of the file from S3 until > read() is called on a S3StorageFile object. > 2) I cannot get images to store correctly to S3 when using an > ImageField. They appear in the bucket with a small filesize change, > but no software I have will recognize them as valid images. Storing > images (and any other file) via a FileField works just fine and the > files are not corrupted. Strangely, storing images via ImageField on > the FileSystemStorage backend works just fine, so it only seems to be > the combination of an ImageField while using this S3 backend that is > the problem. I am not sure where the bug in this is. If you give it > a try, see if you can verify this bug for me. > > Jay > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: S3 tutorial for File Storage Refactor?
With the code from the django-storages you referenced installed somewhere on PYTHONPATH, its as easy as setting a few things in your settings.py. You can see the docs for the code at http://code.larlet.fr/doc/django-s3-storage.html. Put the Required and Optional (if you want it, obviously) stuff in your settings.py and fill in the appropriate values for your S3 account. Now, assuming those settings are all correct, all of your FileFields (and thus, ImageFields) should store to S3 into the bucket you set. You can continue to use upload_to to prefix the filename within the bucket. Now, I've noticed a few bugs, both of which I've notified David (the author) of: 1) if you call object.filefield.size, this code will download the entire file from S3 just to calculate the size. This happens because of the construction of the _open() method on the S3Storage class. It should not make the S3 get call (which downloads the file). The workaround for now is not to use the size property with this code, but the more permanent fix is to delay reading of the file from S3 until read() is called on a S3StorageFile object. 2) I cannot get images to store correctly to S3 when using an ImageField. They appear in the bucket with a small filesize change, but no software I have will recognize them as valid images. Storing images (and any other file) via a FileField works just fine and the files are not corrupted. Strangely, storing images via ImageField on the FileSystemStorage backend works just fine, so it only seems to be the combination of an ImageField while using this S3 backend that is the problem. I am not sure where the bug in this is. If you give it a try, see if you can verify this bug for me. Jay --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
