I'm trying to parse a big (5GB) XML file (product catalog) into a google datastore. The issue I am having is it taking up a lot of memory. I was able to get the memory down from the parsing part by reading it line by line and deleting elements as I go. However something is still sticking behind.
My code is http://pastebin.com/ESARQikC I believe the issue is occuring in this specific function (process_element) def process_element(self,item): if item.tag == "programname": self.Plist.append(item.text) elif item.tag == 'name': self.Plist.append(item.text) elif item.tag == 'description': self.Plist.append(item.text) elif item.tag == 'sku': self.Plist.append(item.text) elif item.tag == 'manufacturer': self.Plist.append(item.text) elif item.tag == 'price': self.Plist.append(item.text) elif item.tag == 'buyurl': self.Plist.append(item.text) elif item.tag == 'imageurl': self.Plist.append(item.text) elif item.tag == 'advertisercategory': self.Plist.append(item.text) elif item.tag=="product": Product( programname=("%s" % self.Plist[0]), name=("%s" % self.Plist[1]), description=("%s" % self.Plist[2][0:500]), sku=("%s" % self.Plist[3]), manufacturer=("%s" % self.Plist[4]), price=("%s" % self.Plist[5]), buyurl=("%s" % self.Plist[6]), imageurl=("%s" % self.getBigImageUrl(self.Plist[7])), advertisercategory=("%s" % self.Plist[8])).put() self.count+=1 print self.count if self.count%15000 == 0: time.sleep(10000) for ob in self.Plist: del ob del self.Plist self.Plist=[] del item When I comment out the Product().put() line and run it, it can go through tons of lines without making much of a memory impact. The reason I added the sleep in the middle of it is I was thinking some subprocesses that GAE spawns were adding the data to the datastore and might need some time to operate. So I waited after adding 15000 items to see if any ram would be freed up (purged memory on the OS side as well) however it did not help. Is this something in my code or something I can't change related to adding data to a datastore. I'm stuck and confused after hours/days of playing around with it. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
