[google-appengine] Re: Local datastore import is too slow
You are right Dave. I started working with s small set of data on the local server. I also fragmented my CSV file in a set of 100 records this reduced the memory requirements. My app works fine now. Thanks, Jyoti On Nov 20, 12:58 am, David Symonds [EMAIL PROTECTED] wrote: On Thu, Nov 20, 2008 at 11:35 AM,JyotiShete-Javadekar [EMAIL PROTECTED] wrote: I am trying to load my development datastore using the bulk loader script. However the import is very slow. I had to kill the import process since it was not completed even after 12 hours. I have about 13K rows in the CSV file. One data model entity is about 300 bytes. 10 entities are imported at a time. The model has two unicode attributes, two unicode list attributes , one url and one long attribute. I use unicode.split to populate list attributes. I am running the bulk loader in a virtual machine having 512MB memory. During the import about 91% memory is utilized. I have not specified any custom index in index.yaml. Don't use the dev_appserver for that amount of data. At least in its current state it is not designed for that amount. Do you *absolutely* need to load that amount of data to do local development? Dave. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Local datastore import is too slow
On Thu, Nov 20, 2008 at 11:35 AM, Jyoti Shete-Javadekar [EMAIL PROTECTED] wrote: I am trying to load my development datastore using the bulk loader script. However the import is very slow. I had to kill the import process since it was not completed even after 12 hours. I have about 13K rows in the CSV file. One data model entity is about 300 bytes. 10 entities are imported at a time. The model has two unicode attributes, two unicode list attributes , one url and one long attribute. I use unicode.split to populate list attributes. I am running the bulk loader in a virtual machine having 512MB memory. During the import about 91% memory is utilized. I have not specified any custom index in index.yaml. Don't use the dev_appserver for that amount of data. At least in its current state it is not designed for that amount. Do you *absolutely* need to load that amount of data to do local development? Dave. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Local datastore import is too slow
I am also havin the same problem. Could you please clarify me if you got the answer On Nov 20, 5:35 am, Jyoti Shete-Javadekar [EMAIL PROTECTED] wrote: Hi, I am trying to load my development datastore using the bulk loader script. However the import is very slow. I had to kill the import process since it was not completed even after 12 hours. I have about 13K rows in the CSV file. One data model entity is about 300 bytes. 10 entities are imported at a time. The model has two unicode attributes, two unicode list attributes , one url and one long attribute. I use unicode.split to populate list attributes. I am running the bulk loader in a virtual machine having 512MB memory. During the import about 91% memory is utilized. I have not specified any custom index in index.yaml. Could someone please tell me why the import is so slow? What optimization should I do to improve the performance? I have not yet loaded the data at appspot. Above observations are for my local development server. Are there any performance numbers available for the bulk loader script? Thanks in advance, Jyoti --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
