On Mon, Aug 9, 2010 at 12:41 PM, Marco Neves
<[email protected]> wrote:
> On Monday 09 August 2010 20:59:50 David Nicol wrote:
>> On Sun, Aug 8, 2010 at 4:58 AM, Marco Neves [ModAuthors]
>>
>> <[email protected]> wrote:
>> > Hello,
>> >
>> > In need for a way to transfer a large amount of small
>> > datastructures I created a module that stores and retrieves a stream of
>> > storables in a file.
>>
>> I'm confused -- a stream can't be stored in a file, because it is
>> continually generated.
>>
>> Also, what is different about this freezer/thawer that makes it
>> superior to, for instance, YAML or compressed Data::Dumper dumps?
>>
>> That is, for what general class of use cases would one prefer the
>> Neves data marshalling system to the available others?
>>
>> Also, if you're reusing other things in it -- you say "storables" are
>> you doing something like prefixing length and colon to the output of
>> Storable and jamming a bunch of those together? -- the underlying
>> technology and then "stream" might make sense, especially if it can
>> take an open handle instead of a file name in its constructor. Such a
>> thing could be nice for creating an abstracted pipe between
>> distributed processes for instance.
>
> Hi,
>
> Yes, the module does store a lot of storables on a single file, and
> yes,
> it accepts an open handler as constructor.
>
> I written it to transfer large amounts of "small" data structures
> between
> servers.
>
> I'm thinking to use the name Archive::StorableStream for the module.
> The
> git repository is in:
>
> http://git.magick-source.net/perl5/storablestream
>
> Before you tell it, I know, none of the docs are still done.
FWIW, have you considered how to deal with what happens if your chunks
don't match the size prefix? If I read your protocol correctly, you
are sending the length of the following blob as ASCII digits
terminated by a \0.
$blob = Storable::nfreeze( ... )
$packed = length( $blob ) . "\0$blob"
and to read:
$length = 0 + $packed;
Storable::thaw( substr $packed, length $length )
I dunno your objection to YAML but it has the nice property that it
provides context for helping to recover from mis-transmitted documents
in the stream by virtue of having a nested-ness marker (the
indentation) and both starting and ending document tokens (--- and
...). Further, I would highly discourage you from naming anything
using "Storable" with "Archive" because absolutely no one should ever
consider actually *storing* a Storable blob. If you keep one around
then you'll be tempted to read it in a client version that didn't
match what you created it with. That way lies madness.
Josh
