[google-appengine] Re: Bulk upload datastore and files
How many rows? within 10m in a task queue, you should be able to upload >100k rows. If you need more, there are many tools, but the simplest one (Java) is probably: https://github.com/icoloma/queue4gae On Tuesday, 2 February 2016 21:47:23 UTC+13, pinesh shah wrote: > > Hi, > > I have a scenario mentioned below, need to understand the possible > solution for the same. > > Need to bulk upload data into datastore using CSV file. In this file each > row will have a field which should have URL for a file associated with that > row. > I am planning to use google cloud storage for uploading the file. > > > Need to understand a possible better solution in which first i can upload > files and then associate them with these rows by auto updating the CSV file > and then uploading the CSV information to datastore. > > Not sure of this is the correct forum for asking. > > Thanks for Help > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/72a52679-20d6-4dc0-b563-8cbc4eadf2b5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: Bulk Upload Images
You cannot do it with CSVs. I do not know of an easier way to do this but for my application, every image is found online so I could use the URLFetch service to fetch each image and then pass the response into db.Blob() and save it with .put() On Mon, Jul 6, 2009 at 7:23 PM, Albert albertpa...@gmail.com wrote: Hi! This might be obvious, but I'm not completely familiar with CSV's. I've tried the bulk uploader tool through the remote_api, and I've been successful at uploading some entities already. However, this time, the entity has an image (db.BlobProperty()) property. I have no idea how to insert the image files/data into the CSV. So that I can upload the data through the bulk uploader tool. I tried Microsoft Excel, but I cannot insert an image into a single cell (lol). Does anyone have an idea on how to achieve this? Thanks! -- Cheers, John Creative Director / Programmer Digital Eternal (http://www.digitaleternal.com) Branding, Web Design/Development, Internet Advertising, SEO --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk Upload Images
Hi You can put images in your csv programtaically. base64 encode them (you will need to increase the max field size in the csv reader) then in your handler base64 decode them and the db.BlobProperty the string. is myobj.image = db.BlobProperty(b64decode(val)) I am round tripping polymodels via remote_api between production and dev instances. Rgds Tim On Jul 6, 7:23 pm, Albert albertpa...@gmail.com wrote: Hi! This might be obvious, but I'm not completely familiar with CSV's. I've tried the bulk uploader tool through the remote_api, and I've been successful at uploading some entities already. However, this time, the entity has an image (db.BlobProperty()) property. I have no idea how to insert the image files/data into the CSV. So that I can upload the data through the bulk uploader tool. I tried Microsoft Excel, but I cannot insert an image into a single cell (lol). Does anyone have an idea on how to achieve this? Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk Upload Images
That is great. Thanks for that idea. On Mon, Jul 6, 2009 at 7:57 PM, Tim Hoffman zutes...@gmail.com wrote: Hi You can put images in your csv programtaically. base64 encode them (you will need to increase the max field size in the csv reader) then in your handler base64 decode them and the db.BlobProperty the string. is myobj.image = db.BlobProperty(b64decode(val)) I am round tripping polymodels via remote_api between production and dev instances. Rgds Tim On Jul 6, 7:23 pm, Albert albertpa...@gmail.com wrote: Hi! This might be obvious, but I'm not completely familiar with CSV's. I've tried the bulk uploader tool through the remote_api, and I've been successful at uploading some entities already. However, this time, the entity has an image (db.BlobProperty()) property. I have no idea how to insert the image files/data into the CSV. So that I can upload the data through the bulk uploader tool. I tried Microsoft Excel, but I cannot insert an image into a single cell (lol). Does anyone have an idea on how to achieve this? Thanks! -- Cheers, John Creative Director / Programmer Digital Eternal (http://www.digitaleternal.com) Branding, Web Design/Development, Internet Advertising, SEO --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk Upload Images
Hi Tim! Wow. Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk Upload Images
You can also simply include the filename in the csv, and make the converter function lambda x:open(x,'rb').read(). -Nick Johnson On Mon, Jul 6, 2009 at 12:57 PM, Tim Hoffmanzutes...@gmail.com wrote: Hi You can put images in your csv programtaically. base64 encode them (you will need to increase the max field size in the csv reader) then in your handler base64 decode them and the db.BlobProperty the string. is myobj.image = db.BlobProperty(b64decode(val)) I am round tripping polymodels via remote_api between production and dev instances. Rgds Tim On Jul 6, 7:23 pm, Albert albertpa...@gmail.com wrote: Hi! This might be obvious, but I'm not completely familiar with CSV's. I've tried the bulk uploader tool through the remote_api, and I've been successful at uploading some entities already. However, this time, the entity has an image (db.BlobProperty()) property. I have no idea how to insert the image files/data into the CSV. So that I can upload the data through the bulk uploader tool. I tried Microsoft Excel, but I cannot insert an image into a single cell (lol). Does anyone have an idea on how to achieve this? Thanks! -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk Upload/Download for List and Unicode properties
On Thu, May 21, 2009 at 9:11 AM, Alex Popescu wrote: For lists, things sound even more complex. I must confess that I haven't tried it yet, but (I don't know why exactly) I believe that specifying 'list' as a conversion type will not be enough. Has anyone tried any of these scenarios and have any hints for me? You can create your own conversion function. You'd have to format the list as a string, and the conversion function takes that string and convert it to a list. -- rodrigo --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk Upload/Download for List and Unicode properties
I find a blog gives a solution:
http://www.cnblogs.com/step1/archive/2009/03/12/1409899.html
Search these code in
\google_appengine\google\appengine\tools\bulkloader.py:
for (name, converter), val in zip(self.__properties, values):
if converter is bool and val.lower() in ('0', 'false', 'no'):
val = False
properties[name] = converter(val)
Change it to:
for (name, converter), val in zip(self.__properties, values):
if converter is bool and val.lower() in ('0', 'false', 'no'):
val = False
if isinstance(val,str) and not isinstance(val, unicode):
val=unicode(val,'utf-8')
properties[name] = converter(val)
And the list properties should be wrapped by quote:
field1, field2, list value1, list value2, list value3
2009/5/21 Alex Popescu the.mindstorm.mailingl...@gmail.com
I've looked through the official documentation of bulk uploader/
exporter and I couldn't find any reference on how to use it when you
have list or unicode properties.
For unicode I assume that I'll have to encode each such value to utf8
before exporting and decode it back to unicode upon import.
For lists, things sound even more complex. I must confess that I
haven't tried it yet, but (I don't know why exactly) I believe that
specifying 'list' as a conversion type will not be enough.
Has anyone tried any of these scenarios and have any hints for me?
Many thanks in advance,
./alex
--
DailyCloud: http://the.dailycloud.net
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Bulk Upload/Download for List and Unicode properties
On Thu, May 21, 2009 at 9:29 AM, 风笑雪 kea...@gmail.com wrote:
Change it to:
for (name, converter), val in zip(self.__properties, values):
if converter is bool and val.lower() in ('0', 'false', 'no'):
val = False
if isinstance(val,str) and not isinstance(val, unicode):
val=unicode(val,'utf-8')
This change is not needed. Instead of specifying str as converter,
define a conversion function and use that instead:
def convert_str(val):
return unicode(val, 'utf-8')
-- rodrigo
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Bulk upload specifying a key name
Hi ecognium, You can do this by implementing the generate_key method in your bulkloader.Loader class. See the code for details: http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/tools/bulkloader.py#3215 -Nick Johnson On Mon, May 18, 2009 at 4:54 AM, ecognium ecogn...@gmail.com wrote: Hi All, How do i specify a key name when using the bulk upload tool? I am currently using WWW:: Mechanize to upload my data one row at a time but would be nice if i can use the bulk loader as i don't want to reinvent the many options available in the tool. Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk upload specifying a key name
Fantastic, Nick, thanks for the info. Is there an equally simple way to output the key_name on bulk download? On May 18, 5:00 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi ecognium, You can do this by implementing the generate_key method in your bulkloader.Loader class. See the code for details:http://code.google.com/p/googleappengine/source/browse/trunk/python/g... -Nick Johnson On Mon, May 18, 2009 at 4:54 AM, ecognium ecogn...@gmail.com wrote: Hi All, How do i specify a key name when using the bulk upload tool? I am currently using WWW:: Mechanize to upload my data one row at a time but would be nice if i can use the bulk loader as i don't want to reinvent the many options available in the tool. Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk upload specifying a key name
Hi Dan, Unfortunately, no - as I mentioned in my other responses, currently the easiest way to do this appears to be overloading output_entities and handling the writing of entities to disk yourself. -Nick On Mon, May 18, 2009 at 4:43 PM, Devel63 danstic...@gmail.com wrote: Fantastic, Nick, thanks for the info. Is there an equally simple way to output the key_name on bulk download? On May 18, 5:00 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi ecognium, You can do this by implementing the generate_key method in your bulkloader.Loader class. See the code for details:http://code.google.com/p/googleappengine/source/browse/trunk/python/g... -Nick Johnson On Mon, May 18, 2009 at 4:54 AM, ecognium ecogn...@gmail.com wrote: Hi All, How do i specify a key name when using the bulk upload tool? I am currently using WWW:: Mechanize to upload my data one row at a time but would be nice if i can use the bulk loader as i don't want to reinvent the many options available in the tool. Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Bulk upload specifying a key name
Thanks, Nick. It works! On May 18, 5:00 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi ecognium, You can do this by implementing the generate_key method in your bulkloader.Loader class. See the code for details:http://code.google.com/p/googleappengine/source/browse/trunk/python/g... -Nick Johnson On Mon, May 18, 2009 at 4:54 AM, ecognium ecogn...@gmail.com wrote: Hi All, How do i specify a key name when using thebulkuploadtool? I am currently using WWW:: Mechanize touploadmy data one row at a time but would be nice if i can use thebulkloader as i don't want to reinvent the many options available in the tool. Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: bulk upload
Hi, This first error looks to be caused by this issue: http://code.google.com/p/googleappengine/issues/detail?id=157 For the second, to upload text fields, you can specify the value type as db.Text. -Marzia On Wed, Nov 19, 2008 at 11:02 PM, Gampesh [EMAIL PROTECTED] wrote: Hi, my bulkupolad script is working fine for the fields in loader.py is str that is: def __init__(self): fields = [ (newsid, str), (celebrity_id, str), (url_id, str), (title, str), (link, str), (description, str), (pub_date, str), (created_at, str), (updated_at, str) ] bulkload.Loader.__init__(self, News, fields) but in my data i have some some text data which have more the 500 bytes data which is range for the text so what would be the by loader.py for the text field instead of str (description, str), is field which is a text type so what should be type in place of str thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: bulk upload
yes thank Marzia in log file i got that error was in my data there are some ascii code which is not compatible with Gql Bulkupload parse. Can you please help me on this : if i want to store data in local server using same script just change is in --url=http://localhost:8080/load; then i got following error: INFO 2008-11-20 10:52:39,890 bulkload_client.py] Starting import; maximum 10 entities per post INFO 2008-11-20 10:52:39,890 bulkload_client.py] Importing 10 entities in 4673 bytes ERROR2008-11-20 10:52:41,765 bulkload_client.py] An error occurred while importing: (10061, 'Connection refused') ERROR2008-11-20 10:52:41,780 bulkload_client.py] Import failed Same script is running for google app enging data store. Thanks for the help. On Nov 19, 10:11 pm, Marzia Niccolai [EMAIL PROTECTED] wrote: Hi, What error is listed in your logs for this message? http://appengine.google.com/logs?app_id=xx -Marzia On Wed, Nov 19, 2008 at 5:36 AM, Gampesh [EMAIL PROTECTED] wrote: my bulkupload was working properly but after two or three upload of data i got following error can any body help me? Thanks INFO 2008-11-19 19:00:38,358 bulkload_client.py] Starting import; maximum 10 entities per post INFO 2008-11-19 19:00:38,358 bulkload_client.py] Importing 10 entities in 5379 bytes ERROR2008-11-19 19:00:43,828 bulkload_client.py] An error occurred while importing: Received code 500: Internal Server Error htmlhead meta http-equiv=content-type content=text/html;charset=utf-8 title500 Server Error/title /head body text=#00 bgcolor=#ff h1Error: Server Error/h1 h2The server encountered an error and could not complete your request.pIf the problem persists, please A HREF=http:// www.google.com/support/report/A your proble m and mention this error message and the query that caused it./h2 h2/h2 /body/html ERROR2008-11-19 19:00:43,858 bulkload_client.py] Import failed --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: bulk upload users
So is there any way of retrieving a user object from the user's current email address? If there isn't a way, is this for security reasons (rather, spam- prevention reasons) or some other reason? Thanks! Luigi On Oct 24, 10:48 am, Marzia Niccolai [EMAIL PROTECTED] wrote: If you know the user's email address for which you want to create profiles, you can create the user object fairly easily. new_user = users.User(email='[EMAIL PROTECTED]') Be aware (from:http://code.google.com/appengine/docs/users/userobjects.html ): If the User constructor is called with an email address that does not correspond with a valid Google account, the object will be created but it will not correspond with a real Google account. This will be the case even if someone creates a Google account with the given email address after the object is stored. A User value with an email address that does not represent a Google account at the time it is created will never match a User value that represents a real user. Also, please note that last week we announced SSL support for all *. appspot.com domains. -Marzia On Fri, Oct 24, 2008 at 7:41 AM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hi, i need to be able to bulk sign-up users for my app. This doesn't seem to be possible with Google's User model. My guess is I'll have to create a custom user model, but the lack of ssl support is a bit of a deterrent. Any thoughts on how this can be done? Thanks, -Reza --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: bulk upload version 1 format?
Hi Angel, There is only one version of the bulkload tool, which is the one currently released. I think the comment was written to acknowledge that in the future we will likely continue to improve the upload/download data functionality of App Engine, so this tool is not likely to be the final word. As for the comment about the Tools and Libraries, I think it is just a reference to an earlier version of the App Engine documentation, which was refactored prior to release. -Marzia On Sun, Oct 19, 2008 at 8:01 PM, Angel [EMAIL PROTECTED] wrote: So in looking for ways to bulk upload data with key_name using the built-in packages, I saw mention of a version 1 formatted data in google.appengine.ext.bulkload package? From the doc for the BulkLoad class: In the POST request, the body contains the data representing the entities' property values. The original format was a sequences of lines of comma-separated values (and is handled by the Load method). The current (version 1) format is a binary format described in the Tools and Libraries section of the documentation, and is handled by the LoadV1 method). Didn't find any more mention of the format or even the Tools and Libraries section. Is V1 perhaps using protocol buffers version 1? If so, I note that there is more than one mention/rumor of the version 2 PB spec being applied across the board in GAE, so should I just be patient for a bit? I ask because I am in planning stages of a data sync tool and would like to have some insight as to what to target. -angel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
