[web2py] Re: google app engine, file size limits
This is fantastic! On Apr 9, 5:31 pm, howesc wrote: > Hi all, > > I implemented this today, and learned some things. I posted my > version of the upload handler as a comment to the slice > at:http://www.web2pyslices.com/main/slices/take_slice/63 > > enhancements in this version: > * parse the blob_info object out of the request, no more dummy > request > * integrate with a table that has more information then just the file > keys. > * handle deletes and updates of images by removing old files from > blobstore (assumes that this record is the only one that references a > given file) > > Hope that it's helpful for people. > > christian -- To unsubscribe, reply using "remove me" as the subject.
[web2py] Re: google app engine, file size limits
Hi all, I implemented this today, and learned some things. I posted my version of the upload handler as a comment to the slice at: http://www.web2pyslices.com/main/slices/take_slice/63 enhancements in this version: * parse the blob_info object out of the request, no more dummy request * integrate with a table that has more information then just the file keys. * handle deletes and updates of images by removing old files from blobstore (assumes that this record is the only one that references a given file) Hope that it's helpful for people. christian -- To unsubscribe, reply using "remove me" as the subject.
[web2py] Re: google app engine, file size limits
Exactly. Like you specify db.table.field.uploadfolder=os.path.join(request.folder,'uploads') you could specify db.table.field.uploadmethod='gae_blobstore' On Feb 25, 2:02 pm, villas wrote: > Alex, Thanks for this interesting information. This can make GAE > suitable for a lot of extra use. > > > It should be easy to add blobstore support to web2py. > > +1 > > Massimo, How do you imagine it would be supported? > Maybe as just an option on 'upload' type which is ignored when non-GAE? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
Alex, Thanks for this interesting information. This can make GAE suitable for a lot of extra use. > It should be easy to add blobstore support to web2py. +1 Massimo, How do you imagine it would be supported? Maybe as just an option on 'upload' type which is ignored when non-GAE? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
It should be easy to add blobstore support to web2py. On Feb 24, 5:22 am, what_ho wrote: > no problem! > > With some informal testing I found the blobstore absolutely flies > along, serves up files as fast as whatever internet connection I tried > could handle. I did not load test with multiple users, but I don't > doubt this service is built to scale. > > The blobstore is meant for file upload and download only. An > application cannot usually access data held here directly. There is > one exception at present where the GAE image API can access blobstore > data files (for example to generate a thumbnail from a large image). > But in general using the blobstore will not be suitable for every > project. > > It would be nice to build blobstore API support into a more generic > patch to be submitted for inclusion in the web2py source, like for a > new blobstore database type. The gae blobstore would be just one > possible backend - local db column / file system folder / web service > on external url could be others. So on gae and non-gae hosted web2py > apps this datatype could be supported. > > One thing I have yet to figure out with the blobstore API - I'd like > to call parseblobinfo(cgi.FieldStorage) in their API directly from > web2py code so there is no need to wrap a wgsi call to obtain details > about a new blob. But I don't yet know how to build a cgi.FieldStorage > object from the regular web2py webpage request. > > - Alex > > On Feb 23, 11:25 pm, villas wrote: > > > Wow, you only heard of blobstore yesterday --that's quick! > > If you start using it more, perhaps you would comment on the speed. > > I imagine that some people might want to just store everything in > > there, big and small files, just for the convenience of having them > > all in one place. > > Thanks for the slice! > > D -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
no problem! With some informal testing I found the blobstore absolutely flies along, serves up files as fast as whatever internet connection I tried could handle. I did not load test with multiple users, but I don't doubt this service is built to scale. The blobstore is meant for file upload and download only. An application cannot usually access data held here directly. There is one exception at present where the GAE image API can access blobstore data files (for example to generate a thumbnail from a large image). But in general using the blobstore will not be suitable for every project. It would be nice to build blobstore API support into a more generic patch to be submitted for inclusion in the web2py source, like for a new blobstore database type. The gae blobstore would be just one possible backend - local db column / file system folder / web service on external url could be others. So on gae and non-gae hosted web2py apps this datatype could be supported. One thing I have yet to figure out with the blobstore API - I'd like to call parseblobinfo(cgi.FieldStorage) in their API directly from web2py code so there is no need to wrap a wgsi call to obtain details about a new blob. But I don't yet know how to build a cgi.FieldStorage object from the regular web2py webpage request. - Alex On Feb 23, 11:25 pm, villas wrote: > Wow, you only heard of blobstore yesterday --that's quick! > If you start using it more, perhaps you would comment on the speed. > I imagine that some people might want to just store everything in > there, big and small files, just for the convenience of having them > all in one place. > Thanks for the slice! > D -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
Wow, you only heard of blobstore yesterday --that's quick! If you start using it more, perhaps you would comment on the speed. I imagine that some people might want to just store everything in there, big and small files, just for the convenience of having them all in one place. Thanks for the slice! D -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
I got the blobstore service working with web2py for up to 50MB file support on GAE. Have written up the steps I took here: http://www.web2pyslices.com/main/slices/take_slice/63 - Alex On Feb 22, 7:42 pm, what_ho wrote: > I had not heard of the blobstore service before - thank you for > pointing that out. I agree - well worth considering, up to 1GB files > the service is still free. > > On Feb 22, 6:22 pm, villas wrote: > > > > > For anything above 1Mb, I think Google are trying their best to > > promote their new 'blobstore' which is a pay service. However, it is > > still extremely cheap so may be worth considering for the > > convenience. -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
I got web2py working with blobstore, have written up details on the steps I took here: http://www.web2pyslices.com/main/slices/take_slice/63 - Alex On Feb 22, 7:42 pm, what_ho wrote: > I had not heard of the blobstore service before - thank you for > pointing that out. I agree - well worth considering, up to 1GB files > the service is still free. > > On Feb 22, 6:22 pm, villas wrote: > > > > > For anything above 1Mb, I think Google are trying their best to > > promote their new 'blobstore' which is a pay service. However, it is > > still extremely cheap so may be worth considering for the > > convenience. -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
I had not heard of the blobstore service before - thank you for pointing that out. I agree - well worth considering, up to 1GB files the service is still free. On Feb 22, 6:22 pm, villas wrote: > For anything above 1Mb, I think Google are trying their best to > promote their new 'blobstore' which is a pay service. However, it is > still extremely cheap so may be worth considering for the > convenience. -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
For anything above 1Mb, I think Google are trying their best to promote their new 'blobstore' which is a pay service. However, it is still extremely cheap so may be worth considering for the convenience. -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: google app engine, file size limits
Using rar to compress and chunk up the file at the same time could be a good option, especially if this can be done on the client (for example using a java-applet as a file download/upload manager). Chunking on the client would be the only way to go over the 10MB limit as well. GAE app development seems to be an exercise in being as efficient as you can, steering clear of restrictions, limits and time-outs. I like the idea of running rar server-side but I'd worry about hitting limits on individual page requests. For my needs I'm not so much bothered about the compression part - it's the chunking that would be great to have. I'm hopeful since this has been done for Java on GAE: http://code.google.com/p/google-file-service/wiki/GoogleFile Thanks, - Alex On Feb 22, 3:36 pm, Thadeus Burgess wrote: > Well, with RAR files you can split them up into 1MB files. So you get > the uploaded file, rar it in .9MB splits, upload to a table that > stores "chunks". > > Now when you need the file, pull the "chunks" out of your table, and > unrar them. I know that this works on the unix command line, not sure > if the unrar lib in python supports multiple > > files in memory. > > However this will be very expensive to do, if you have a 10MB file it > will most likely take at "least" double the ram to perform this > operation. > > -Thadeus > > > > On Mon, Feb 22, 2010 at 9:29 AM, what_ho wrote: > > > Has anyone tried splitting and joining one file across several blobs > > in GAE to allow uploads up to the 10M -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
