Hello list,
sorry for the possibly noob question, I've googled around without much
success looking for an answer. Basically, I am given a series of this huge
Python class (a "Simulation" object), which contains an enormous amount of
information - when I cPickle it (with highest protocol), it can result to
files 200-250 MB in size, although rarely it can get up to 2 GB. I am
looking for intelligent ways to store these objects into a database. I have
to say that I don't have that much control on this Simulation class, so I
can't change its internal structure - I'm just looking for a better
alternative to what I am doing.
So, what I am doing now is basically storing this huge object as a string.
I have these two methods:
import cPickle
import zlib
import base64
def serialize(my_simulation):
my_db_object = base64.b64encode(zlib.compress(cPickle.dumps(obj,
cPickle.HIGHEST_PROTOCOL)))
return my_db_object
def deserialize(my_db_object):
my_simulation =
cPickle.loads(zlib.decompress(base64.b64decode(my_db_object)))
return simulation
I can use them to store/retrieve this big Python classes to/from the
database, but I feel it's not a particularly effective way to handle this
problem. I've tried to get my head around BLOBs and LargeBinary stuff, but
I'm not sure I'm on the right path here. I appreciate any suggestion on how
to approach the problem, to make the storing/retrieving of these objects a
bit less time/memory consuming (especially time).
On a related note, I am now using MySQL as a backend - but I am open to
suggestions about possible alternatives that may make this problem more
tractable: I have some difficulties in installing other backends (like
PostgreSQL or psycopg2), but I know I can use Access and Oracle backends. I
know that Access is not particularly famous in the SQLAlchemy world. Of
course, if any other backend has advantages over MySQL in dealing with this
issue, I may try to convince IT to get it installed on our machines.
All suggestions and comments are most welcome.
Thank you in advance for your help.
Andrea.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
