I would recommend just storing them on disk and let the OS VMM deal with
caching for speed. If you are not constrained for space I would
recommend not zlib-ing it either.
AM
On 12/3/14 1:18 PM, Andrea Gavana wrote:
Hello list,
sorry for the possibly noob question, I've googled around without
much success looking for an answer. Basically, I am given a series of
this huge Python class (a "Simulation" object), which contains an
enormous amount of information - when I cPickle it (with highest
protocol), it can result to files 200-250 MB in size, although rarely
it can get up to 2 GB. I am looking for intelligent ways to store
these objects into a database. I have to say that I don't have that
much control on this Simulation class, so I can't change its internal
structure - I'm just looking for a better alternative to what I am doing.
So, what I am doing now is basically storing this huge object as a
string. I have these two methods:
import cPickle
import zlib
import base64
def serialize(my_simulation):
my_db_object = base64.b64encode(zlib.compress(cPickle.dumps(obj,
cPickle.HIGHEST_PROTOCOL)))
return my_db_object
def deserialize(my_db_object):
my_simulation =
cPickle.loads(zlib.decompress(base64.b64decode(my_db_object)))
return simulation
I can use them to store/retrieve this big Python classes to/from the
database, but I feel it's not a particularly effective way to handle
this problem. I've tried to get my head around BLOBs and LargeBinary
stuff, but I'm not sure I'm on the right path here. I appreciate any
suggestion on how to approach the problem, to make the
storing/retrieving of these objects a bit less time/memory consuming
(especially time).
On a related note, I am now using MySQL as a backend - but I am open
to suggestions about possible alternatives that may make this problem
more tractable: I have some difficulties in installing other backends
(like PostgreSQL or psycopg2), but I know I can use Access and Oracle
backends. I know that Access is not particularly famous in the
SQLAlchemy world. Of course, if any other backend has advantages over
MySQL in dealing with this issue, I may try to convince IT to get it
installed on our machines.
All suggestions and comments are most welcome.
Thank you in advance for your help.
Andrea.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
<mailto:sqlalchemy+unsubscr...@googlegroups.com>.
To post to this group, send email to sqlalchemy@googlegroups.com
<mailto:sqlalchemy@googlegroups.com>.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
