[julia-users] Re: Saving and Loading data (when JLD is not suitable)

[Páll Haraldsson]

On Wednesday, August 31, 2016 at 4:47:37 AM UTC, Lyndon White wrote:
>
>
> There are 3 problems with using Base,serialize as a data storage format.
>
>1. *It is not stable* -- the format is evolving as the language 
>evolves, and it breaks the ability to load files
>2. *It is only usable from Julia* -- vs JLD which is, in the end fancy 
>HDF5, anything can read it after a little work
>3. *It is not safe from a security perspective*  -- Maliciously 
>crafted ".jsz" files can allow arbitrary code execution to occur during 
> the 
>deserialize step.
>
>
You just reminded me of:

http://prevayler.org/

that has been ported to other than the first language, Java, and I think 
would also be nice for Julia.. They solved a different problem than what 
you have (or the three above), and pointed to XQuery/XML, if I recall, for 
the query/[export out of "db"]serialization


http://www.onjava.com/pub/a/onjava/2005/06/08/prevayler.html

"A *prevalent* system makes use of serialization, and is again useful only 
when an in-memory data set is feasible. A serialized snapshot of a working 
system can be taken at regular intervals as a first-line storage mechanism. 
[..]
Prevayler 1.0 was awarded a JOLT Productivity  
award in 2004. The recent version, 2.0, has many improvements, including a 
simpler API."

https://github.com/jsampson/prevayler

-- 
Palli.

Re: [julia-users] Re: Saving and Loading data (when JLD is not suitable)

[Tim Holy]

https://github.com/JuliaIO/JLD.jl/blob/master/doc/jld.md#custom-serialization

On Wednesday, August 31, 2016 1:56:25 AM CDT Kristoffer Carlsson wrote:
> I am sure you know but I just want to point out that it is possible to
> overload the way JLD saves and loads types. You can write a custom save
> function that transforms the thing you are trying to save into a format
> more suitable for HDF5 and then the loader can transform it back into your
> original format.

[julia-users] Re: Saving and Loading data (when JLD is not suitable)

[Kristoffer Carlsson]

I am sure you know but I just want to point out that it is possible to overload 
the way JLD saves and loads types. You can write a custom save function that 
transforms the thing you are trying to save into a format more suitable for 
HDF5 and then the loader can transform it back into your original format.

[julia-users] Re: Saving and Loading data (when JLD is not suitable)

[Lyndon White]

On Wednesday, 31 August 2016 12:47:37 UTC+8, Lyndon White wrote:
>
>
> So I have one of the models, saved using their custom format (
> https://github.com/sbos/AdaGram.jl/blob/master/src/util.jl#L94)
> It is *6Gb*, and takes ages to load and save.
> I switch to JLD, *13Gb* even worse.
> I used Base.serialize, Its just *13Mb* -- That is a full 3 orders of 
> magnitude different.
> And it is fast to read and write from disk.
>
>
Actually, the figure for Base,Serialize is off there.
Turns out there was a ShareArray that did not serialize to disk.
As so that 13 MB file can't be deserialised.
I thought it was a bit extreme.

But the point stands, that in general there is a large category of types 
for which Base.Serialise is much much faster than JLD.