[google-appengine] Re: Text files bulk upload : any advice ?
Thank you for reading Robert. What I need on AppEngine side : * for each small : read their headers, they have some metadata going into db.Model properties, along with the file content. * send some of the files to clients, not all the files, in text format. Making a zip before upload is fine, but I don't know if there is something in AppEngine to unzip it on the server side and deal with each file. David On 13 avr, 19:05, Robert Kluin robert.kl...@gmail.com wrote: Hi David, If you just want to upload a single file, and they are all pretty small, why don't you just zip them? Or, maybe I don't understand what you are asking...? Robert On Wed, Apr 13, 2011 at 19:53, davidgm david.guyonmar...@gmail.com wrote: Hi there, I've read * how to upload one text file - that's ok. * how to bulk upload data (csv) - that's ok also. But this doesn't help with for this problem: = I need to upload a few dozen of small text files (a few kbytes each). I need to refresh them sometimes. They will be stored in datastore with a few metadata (filename and a release no). I've been thinking to merge them into one big file to split then later in GAE. But I would rather not do it this way, unless you have a nice technique to handle that. They all have special characters of all kinds and unicode... Do you have any suggestion ? Thank you -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Text files bulk upload : any advice ?
http://www.tareandshare.com/2008/09/28/Zip-Google-App-Engine-GAE/ 2011/4/14 davidgm david.guyonmar...@gmail.com Thank you for reading Robert. What I need on AppEngine side : * for each small : read their headers, they have some metadata going into db.Model properties, along with the file content. * send some of the files to clients, not all the files, in text format. Making a zip before upload is fine, but I don't know if there is something in AppEngine to unzip it on the server side and deal with each file. David On 13 avr, 19:05, Robert Kluin robert.kl...@gmail.com wrote: Hi David, If you just want to upload a single file, and they are all pretty small, why don't you just zip them? Or, maybe I don't understand what you are asking...? Robert On Wed, Apr 13, 2011 at 19:53, davidgm david.guyonmar...@gmail.com wrote: Hi there, I've read * how to upload one text file - that's ok. * how to bulk upload data (csv) - that's ok also. But this doesn't help with for this problem: = I need to upload a few dozen of small text files (a few kbytes each). I need to refresh them sometimes. They will be stored in datastore with a few metadata (filename and a release no). I've been thinking to merge them into one big file to split then later in GAE. But I would rather not do it this way, unless you have a nice technique to handle that. They all have special characters of all kinds and unicode... Do you have any suggestion ? Thank you -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp:// groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Gr, Wim den Ouden https://profiles.google.com/wdenouden, Offline webappshttp://code.google.com/p/relat/wiki/gaetips?ts=1302769343updated=gaetips#Offline_webappsdeveloper based on Javascript https://developer.mozilla.org/en/javascript, Htmlhttp://en.wikipedia.org/wiki/HTML5, Css http://en.wikipedia.org/wiki/CSS, Python http://www.python.org/, Json http://json.org/, Google app enginehttp://code.google.com/intl/nl/appengine/, and other servers. Buuring http://buuring.appspot.com, developer tipshttp://code.google.com/p/relat/wiki/gaetips -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Text files bulk upload : any advice ?
Hi, Both Python and Java have supported libraries for decompressing zip files. Once you've decompressed the files you could process them in a variety of ways, for instance by writing them to the blobstore or creating datastore entities. Robert On Thu, Apr 14, 2011 at 22:50, davidgm david.guyonmar...@gmail.com wrote: Thank you for reading Robert. What I need on AppEngine side : * for each small : read their headers, they have some metadata going into db.Model properties, along with the file content. * send some of the files to clients, not all the files, in text format. Making a zip before upload is fine, but I don't know if there is something in AppEngine to unzip it on the server side and deal with each file. David On 13 avr, 19:05, Robert Kluin robert.kl...@gmail.com wrote: Hi David, If you just want to upload a single file, and they are all pretty small, why don't you just zip them? Or, maybe I don't understand what you are asking...? Robert On Wed, Apr 13, 2011 at 19:53, davidgm david.guyonmar...@gmail.com wrote: Hi there, I've read * how to upload one text file - that's ok. * how to bulk upload data (csv) - that's ok also. But this doesn't help with for this problem: = I need to upload a few dozen of small text files (a few kbytes each). I need to refresh them sometimes. They will be stored in datastore with a few metadata (filename and a release no). I've been thinking to merge them into one big file to split then later in GAE. But I would rather not do it this way, unless you have a nice technique to handle that. They all have special characters of all kinds and unicode... Do you have any suggestion ? Thank you -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Text files bulk upload : any advice ?
Hi there, this might be helpful. I've recently developed a file uploaded for an crowd source application I'm involved in and have used a DiskFileItemFactory. Once you create a factory object you can set the size of the files that are to be uploaded. Bt default it's 10K but you cN manually set this for smaller sizes. here is some code snippets that may be helpful :- DiskFileItemFactory factory = new DiskFileItemFactory(); // create the factory for file uploading factory.setSizeThreshold(Memory); // this sets the threshold for files to be held in memory or stored on disk factory.setRepository(server folder); // defines the location on your server where the files are to be stored You should use this in conjunction with a ServletFileUpload object so you can bulk load multiple files to your server. Hope this is helpful, T. On Apr 13, 11:53 am, davidgm david.guyonmar...@gmail.com wrote: Hi there, I've read * how to upload one text file - that's ok. * how to bulk upload data (csv) - that's ok also. But this doesn't help with for this problem: = I need to upload a few dozen of small text files (a few kbytes each). I need to refresh them sometimes. They will be stored in datastore with a few metadata (filename and a release no). I've been thinking to merge them into one big file to split then later in GAE. But I would rather not do it this way, unless you have a nice technique to handle that. They all have special characters of all kinds and unicode... Do you have any suggestion ? Thank you -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
