Is there a maximum size to a Python program?
I have a program that is generated from a generic process. It's job is to check to see whether records (replicated from another system) exist in a local table, and if it doesn't, to add them. I have 1 of these programs for every table in the database. Everything works well until I do the postcode table. The generated code is 5MB for a system with no current data. Normally the file would not be this big as only the changes are copied over. Python just quits, I have tried stepping through the code in the debugger but it doesn't even start. I am thinking that dynamically generating the programs to run might not be such a good idea. It would be a shame to drop it because the system needs to be generic and it runs from an XML file so the resulting code could be pretty complex, and I am new to Python. The program did generate a pyc so it was able to compile. Thoughts anyone? -- http://mail.python.org/mailman/listinfo/python-list
(UPDATE) Memory problems (garbage collection)
Thanks for the replies. I got my program working but the memory problem remains. When the program finishes and I am brought back to the PythonWin the memory is still tied up until I run gc.collect(). While my choice of platform for XML processing may not be the best one (I will change it later) I am still concerned with the memory issue. I can't believe that it could be an ongoing issue for Python's xml.dom, but nobody was able to actually point to anything in my code that may have caused it? I changed the way I was doing string manipulation but, while it may have been beneficial for speed, it didn't help the memory problem. This is more nuts and bolts than perhaps a newbie needs to be getting into but it does concern me. What if my program was a web server and periodically received these requests? I decided to try something: if __name__ == '__main__': replicate = reptorParsing() replicate.process(filename=os.path.join(os.getcwd(), "request.xml")) import gc gc.collect() Fixed the problem, though I wouldn't know why. Thought it might be something to do with my program but... if __name__ == '__main__': replicate = reptorParsing() replicate.process(filename=os.path.join(os.getcwd(), "request.xml")) del replicate Did not resolve the memory problem. Any ideas? -- http://mail.python.org/mailman/listinfo/python-list
Re: Unicode in writing to a file
Thanks yes that did it. "Peter Otten" <__pete...@web.de> wrote in message news:gspmrf$qlq$0...@news.t-online.com... > Carbon Man wrote: > >> Py 2.5 >> Trying to write a string to a file. >> self.dataUpdate.write(u"\nentry."+node.tagName+ u" = " + cValue) >> cValue contains a unicode character. node.tagName is also a unicode >> string >> though it has no special characters in it. >> Getting the error: >> UnicodeEncodeError: 'ascii' codec can't encode character u'\x93' in >> position 22: ordinal not in range(128) > > You have to decide in what encoding you want to store the data in your > file. > UTF-8 is usually a good choice. Then open it with codecs.open() instead of > the built-in open(): > > import codecs > > f = codecs.open(filename, "w", "UTF-8") > f.write(u"\nentry." + node.tagName + u" = " + cValue) > > Peter > -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory problems (garbage collection)
Thanks for the help.
I converted everything into the StringIO() format. Memory is still getting
chewed up. I will look at ElementTree later but for now I believe the speed
issue must be related to the amount of memory that is getting used. It is
causing all of windows to slow to a crawl. gc.collect() still reports the
same quantity as before.
Don't know what to try next. Updated program is below:
from xml.dom import minidom
import os
from cStringIO import StringIO
class xmlProcessing:
""" General class for XML processing"""
def process(self, filename="", xmlString=""):
if xmlString:
pass
elif filename:
xmldoc = minidom.parse(filename)
self.parse( xmldoc.documentElement )
def parseBranch(self, parentNode):
""" Process an XML branch """
for node in parentNode.childNodes:
try:
parseMethod = getattr(self, "parse_%s" %
node.__class__.__name__)
except AttributeError:
continue
if parseMethod(node):
continue
self.parseBranch(node)
del node
def parse_Document(self, node):
pass
def parse_Text(self, node):
pass
def parse_Comment(self, node):
pass
def parse_Element(self, node):
try:
handlerMethod = getattr(self, "do_%s" % node.tagName)
except AttributeError:
return False
handlerMethod(node)
return True
class reptorParsing(xmlProcessing):
""" Specific class for generating a SQLalchemy program to create tables
and populate them with data"""
def __init__(self):
self.schemaPreface = StringIO()
self.schemaPreface.write("""from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///tutorial.db', echo=False)
metadata = MetaData()
Base = declarative_base()""")
self.schemaTables = StringIO()
self.schemaFields = StringIO()
self.dataUpdate = StringIO()
self.tableDict = {}
self.tableName = StringIO()
self.tables = StringIO()
def parse(self, parentNode):
"""Main entry point to begin processing a XML document"""
self.parseBranch(parentNode)
# Properties such as schemaTables and .tables are populated by the
various methods below
fupdate=open(os.path.join(os.getcwd(), "update.py"), 'w')
if self.schemaTables:
fupdate.write("import schema\n")
f=open(os.path.join(os.getcwd(), "schema.py"), 'w')
f.write(self.schemaPreface+"\n"+self.schemaTables+
'\n' + "metadata.create_all(engine)\n"+
"print 'hello 2'")
f.close()
if self.tables:
fupdate.write(self.tables)
fupdate.close()
def do_TABLES(self, tableNode):
"""Process schema for tables"""
for node in tableNode.childNodes:
self.tableName = node.tagName
# Define a declaritive mapping class
self.schemaTables.write("""\nclass %s(Base):
__tablename__ = '%s'
""" % (self.tableName, self.tableName))
self.schemaFields = StringIO()
# allow for userA = users("Billy","Bob") via a __init__()
self.schemaInitPreface = StringIO()
self.schemaInitPreface.write("def __init__(self")
self.schemaInitBody = StringIO()
self.parseBranch(node)
self.schemaInitPreface.write("):\n")
self.schemaTables.write(self.schemaFields.read() + "\n" + \
self.schemaInitPreface.read() + \
self.schemaInitBody.read() + "\n")
def do_FIELDS(self, fieldsNode):
"""Process schema for fields within tables"""
for node in fieldsNode.childNodes:
if self.schemaFields:
self.schemaFields.write("\n")
cType = ""
# The attribute type holds the type of field
crType = node.attributes["type"].value
if crType==u"C":
cType = "String(length=%s)" % node.attributes["len"].value
elif crType==u"N" and node.attributes["dec"].value==u'0':
cType = "Integer"
elif crType==u"N":
cType = "Numeric(precision=%s, scale=%s)" %
(node.attributes["len"].value,node.attributes["dec"].value)
elif crType==u"L":
cType = "Boolean"
elif crType==u"T":
cType = "DateTime"
elif crType==u"D":
cType = "Date"
elif crType==u"M" or crType==u"G":
cType = "Text"
if node.attributes.getNamedItem("primary"):
cType += ", primary_key=True"
self.schemaFields.write("%s = Column(%s)" % (node.tagName,
cType))
self.schemaInitPreface.write(", \\\n%s" %
(node.tagName))
self.schemaInitBody.wri
Unicode in writing to a file
Py 2.5 Trying to write a string to a file. self.dataUpdate.write(u"\nentry."+node.tagName+ u" = " + cValue) cValue contains a unicode character. node.tagName is also a unicode string though it has no special characters in it. Getting the error: UnicodeEncodeError: 'ascii' codec can't encode character u'\x93' in position 22: ordinal not in range(128) Thanks. -- http://mail.python.org/mailman/listinfo/python-list
Memory problems (garbage collection)
Very new to Python, running 2.5 on windows.
I am processing an XML file (7.2MB). Using the standard library I am
recursively processing each node and parsing it. The branches don't go
particularly deep. What is happening is that the program is running really
really slowly, so slow that even running it over night, it still doesn't
finish.
Stepping through it I have noticed that memory usage has shot up from 190MB
to 624MB and continues to climb. If I set a break point and then stop the
program the memory is not released. It is not until I shutdown PythonWin
that the memory gets released.
I thought this might mean objects were not getting GCed, so through the
interactive window I imported gc. gc.garbage is empty. gc.collect() seems to
fix the problem (after much thinking) and reports 2524104. Running it again
returns 0.
I thought that garbage collection was automatic, if I use variables in a
method do I have to del them?
I tried putting a "del node" in all my for node in loops but that
didn't help. collect() reports the same number. Tried putting gc.collect()
at the end of the loops but that didn't help either.
If I have the program at a break and do gc.collect() it doesn't fix it, so
whatever referencing is causing problems is still active.
My program is parsing the XML and generating a Python program for
SQLalchemy, but the program never gets a chance to run the memory problem is
prior to that. It probably has something to do with the way I am string
building.
My apologies for the long post but without being able to see the code I
doubt anyone can give me a solid answer so here it goes (sorry for the lack
of comments):
from xml.dom import minidom
import os
import gc
class xmlProcessing:
""" General class for XML processing"""
def process(self, filename="", xmlString=""):
if xmlString:
pass
elif filename:
xmldoc = minidom.parse(filename)
self.parse( xmldoc.documentElement )
def parseBranch(self, parentNode):
""" Process an XML branch """
for node in parentNode.childNodes:
try:
parseMethod = getattr(self, "parse_%s" %
node.__class__.__name__)
except AttributeError:
continue
if parseMethod(node):
continue
self.parseBranch(node)
del node
def parse_Document(self, node):
pass
def parse_Text(self, node):
pass
def parse_Comment(self, node):
pass
def parse_Element(self, node):
try:
handlerMethod = getattr(self, "do_%s" % node.tagName)
except AttributeError:
return False
handlerMethod(node)
return True
class reptorParsing(xmlProcessing):
""" Specific class for generating a SQLalchemy program to create tables
and populate them with data"""
def __init__(self):
self.schemaPreface = """from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///tutorial.db', echo=False)
metadata = MetaData()
Base = declarative_base()"""
self.schemaTables = ""
self.schemaFields = ""
self.dataUpdate = ""
self.tableDict = {}
self.tableName = ""
self.tables = ""
def parse(self, parentNode):
"""Main entry point to begin processing a XML document"""
self.parseBranch(parentNode)
# Properties such as schemaTables and .tables are populated by the
various methods below
fupdate=open(os.path.join(os.getcwd(), "update.py"), 'w')
if self.schemaTables:
fupdate.write("import schema\n")
f=open(os.path.join(os.getcwd(), "schema.py"), 'w')
f.write(self.schemaPreface+"\n"+self.schemaTables+
'\n' + "metadata.create_all(engine)\n"+
"print 'hello 2'")
f.close()
if self.tables:
fupdate.write(self.tables)
#f=open(os.path.join(os.getcwd(), "dataUpdate.py"), 'w')
#f.write(self.dataUpdate)
#f.close()
fupdate.close()
def do_TABLES(self, tableNode):
"""Process schema for tables"""
for node in tableNode.childNodes:
self.tableName = node.tagName
# Define a declaritive mapping class
self.schemaTables += """\nclass %s(Base):
__tablename__ = '%s'
""" % (self.tableName, self.tableName)
self.schemaFields = ""
# allow for userA = users("Billy","Bob") via a __init__()
self.schemaInitPreface = "def __init__(self"
self.schemaInitBody = ""
self.parseBranch(node)
self.schemaInitPreface += "):\n"
self.schemaTables += self.schemaFields + "\n" + \
self.schemaInitPreface + \
self.schemaInitBody + "\n"
gc.collect()
def do_FIELDS(self, fieldsNode):
"""Process schema for
debuglevel for a HTTP request
#Python 2.5
# from Dive Into Python 11.5
import httplib
httplib.HTTPConnection.debuglevel = 1
import urllib2
request = urllib2.Request('http://localhost/test/atom.xml')
opener = urllib2.build_opener()
feeddata = opener.open(request).read()
It doesn't show the debug output, any ideas?
--
http://mail.python.org/mailman/listinfo/python-list
Embed a web browser into a page
Hi, I need to embed a web browser into a python page. I am coming from the MS world where I created an app that all of it's interfaces were actually web pages rendered in an Internet Explorer activex control. There was a object hook that allowed you to call into the host environment from javascript. Basically the host environment would receive the documentComplete event and call a method in the document's Javascript passing in an object reference. That reference would then be available for calls to be made from Javascript back into the host environment. I am just starting to explore the Pythonic programming jungle and I was wondering if there is a way to do something similar that would work cross-platform? I guess there is much more complexity in it when you start to go across o/s platform boundaries. The common web interface would then be Gecko or WebKit? So can someone suggest what would be required to build a cross-platform Python app that was capable of browsing HTML files, receiving events from the browser, and that allows the embedded page to call host Python modules from Javascript via an object reference? Or I am asking too much :) -- http://mail.python.org/mailman/listinfo/python-list
