On 12/3/14 2:23 PM, Andrea Gavana wrote:
On Wednesday, December 3, 2014 10:42:27 PM UTC+1, Jonathan Vanasco wrote:
On Wednesday, December 3, 2014 4:23:31 PM UTC-5, Ams Fwd wrote:
I would recommend just storing them on disk and let the OS VMM
deal with
caching for speed. If you are not constrained for space I would
recommend not zlib-ing it either.
I'll second storing them to disk. Large object support in all the
databases is a pain and not very optimal. Just pickle/unpickle a
file and use the db to manage that file.
Thanks to all of you who replied. A couple of issues that I'm sure I
will encounter by letting the files on disk:
1. Other users can easily delete/overwrite/rename the files on disk,
which is something we really, really do not want;
If this is windows group policies are your friends :). If this is linux,
permissions with a secondary service to access the files are a decent
choice.
2. The whole point of a database was to have everything centralized in
one place, not leaving the simulation files scattered around like a
mess in the whole network drive;
The last time I did it a post processing step in my data pipeline
organized the files based on a multi-level folder structure based on the
first x-characters of their sha1.
3. As an aside, not zlib-ing the files saves about 5
seconds/simulation (over a 20 seconds save) but increases the database
size by 4 times. I'll have to check if this is OK.
To use compression or not depends on your needs. If the difference in
time consumed is so stark, I would highly recommend compression.
HTH
AM
Thank you again for your interest.
Andrea.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
<mailto:sqlalchemy+unsubscr...@googlegroups.com>.
To post to this group, send email to sqlalchemy@googlegroups.com
<mailto:sqlalchemy@googlegroups.com>.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
