std.serialization (orange) is now ready to be reviewed.
A couple of notes for the review:
* The most important packages are: orange.serialization and
orange.serialization.archives
* The unit tests are located in its own package, I'm not very happy
about putting the unit tests in the same mod
On 3/24/13, Jacob Carlborg wrote:
> For usage examples, see the github wiki pages:
> https://github.com/jacob-carlborg/orange/wiki/_pages
A small example actually writing the xml file to disk and the reading
back from it would be beneficial.
Btw the library doesn't build with the -w switch:
ora
Just at a glance, a few things strike me...
Phobos doesn't typically use classes, seems to prefer flat functions. Are
we happy with classes in this instance?
Use of caps in the filenames/functions is not very phobos like.
Can I have a post-de-serialise callback to recalculate transient data?
Why
On 2013-03-24 22:41, Andrej Mitrovic wrote:
A small example actually writing the xml file to disk and the reading
back from it would be beneficial.
Ok, so just adding write and read to disk to the usage example on the
github page?
Btw the library doesn't build with the -w switch:
orange\x
On 2013-03-25 02:16, Manu wrote:
Just at a glance, a few things strike me...
Phobos doesn't typically use classes, seems to prefer flat functions.
It's necessary to have a class or struct to pass around. The serializer
is passed to method/functions doing custom serialization. I could create
On 2013-03-24 22:03, Jacob Carlborg wrote:
std.serialization (orange) is now ready to be reviewed.
Just so there is no confusion. If it gets accepted I will replace tabs
with spaces, fix the column limit and change all filenames to lowercase.
--
/Jacob Carlborg
On 2013-03-24 22:41, Andrej Mitrovic wrote:
orange\xml\PhobosXml.d(2536): Error: switch case fallthrough - use
'goto case;' if intended
PhobosXml is a local copy of std.xml with a few small modifications. If
accepted I'll make the changes to std.xml and remove PhobosXml.
--
/Jacob Carlborg
Hello Jacob,
These comments are based on looking into adding Protocol Buffer
as an archive. First some details on the PB format.
https://developers.google.com/protocol-buffers/docs/overview
1) It is a binary format
2) Not all D types can be serialized
3) Serialization is done by message (struc
On 2013-03-30 21:02, Jesse Phillips wrote:
Hello Jacob,
These comments are based on looking into adding Protocol Buffer as an
archive. First some details on the PB format.
https://developers.google.com/protocol-buffers/docs/overview
1) It is a binary format
That shouldn't be a problem. Prefer
On Monday, 25 March 2013 at 08:53:32 UTC, Jacob Carlborg wrote:
With templates:
class Serializer (T)
class XmlArchive
auto archive = new XmlArchive;
auto serializer = new Serializer!(XmlArchive)(archive);
struct Foo
{
void toData (Serializer!(XmlArchive) serializer,
Serializer.Data key);
On Saturday, 30 March 2013 at 20:02:48 UTC, Jesse Phillips wrote:
3) Serialization is done by message (struct) and not by
primitives
PB does serialize by primitives and Archive has archiveStruct
method which is called to serialize struct, I believe. At first
sight orange serializes using buil
On 2013-03-31 12:40, Kagamin wrote:
http://dpaste.dzfl.pl/0f7d8219
So instead Serializer gets a huge API?
--
/Jacob Carlborg
On 2013-03-31 13:23, Kagamin wrote:
PB does serialize by primitives and Archive has archiveStruct method
which is called to serialize struct, I believe. At first sight orange
serializes using built-in grammar (in EXI terms), and since PB uses
schema-informed grammar, you have to provide schema t
On Sunday, 31 March 2013 at 17:33:28 UTC, Jacob Carlborg wrote:
On 2013-03-31 12:40, Kagamin wrote:
http://dpaste.dzfl.pl/0f7d8219
So instead Serializer gets a huge API?
Are a lot of serializers expected to be written? Archives are.
On Sunday, 31 March 2013 at 17:36:12 UTC, Jacob Carlborg wrote:
The actual struct is never passed to the archive in Orange.
It's basically lets the archive know, "the next primitives
belong to a struct".
Knowing the struct type name one may select the matching schema.
In the case of PB the sc
On Sunday, 31 March 2013 at 17:33:28 UTC, Jacob Carlborg wrote:
On 2013-03-31 12:40, Kagamin wrote:
http://dpaste.dzfl.pl/0f7d8219
So instead Serializer gets a huge API?
The point is to have Serializer as an encapsulation point, not
archiver. API can remain the same, it's just calls to the
On Sunday, 31 March 2013 at 17:33:28 UTC, Jacob Carlborg wrote:
On 2013-03-31 12:40, Kagamin wrote:
http://dpaste.dzfl.pl/0f7d8219
So instead Serializer gets a huge API?
As an alternative routing can be handled in the archiver by a
template mixin, which defines virtual methods and routes t
Manual implementation is probably better for PB.
On 2013-03-31 21:02, Kagamin wrote:
Are a lot of serializers expected to be written? Archives are.
Hmm, maybe it could work.
--
/Jacob Carlborg
On 2013-03-31 21:06, Kagamin wrote:
Knowing the struct type name one may select the matching schema. In the
case of PB the schema collection is just int[string][string] - maps type
names to field maps.
Ok. I'm not familiar with Protocol Buffers.
--
/Jacob Carlborg
A "MyArchive" example can be useful too. The basic idea is to
write a minimal archive class with basic test code. All methods
assert(false,"method archive(dchar) not implemented"); the
example complies and runs, but asserts. So people take the
example and fill methods with their own implementat
On Sunday, 31 March 2013 at 21:25:48 UTC, Jacob Carlborg wrote:
Ok. I'm not familiar with Protocol Buffers.
Well, the basic idea of EXI and similar standards is that you can
have 2 types of serialization: built-in when you keep schema in
the serialized message - which value belongs to which f
On Sunday, 31 March 2013 at 11:23:27 UTC, Kagamin wrote:
On Saturday, 30 March 2013 at 20:02:48 UTC, Jesse Phillips
wrote:
3) Serialization is done by message (struct) and not by
primitives
PB does serialize by primitives and Archive has archiveStruct
method which is called to serialize struc
It's a pull parser? Hmm... how reordered fields are supposed to
be handled? When the archiver is requested for a field, it will
probably need to look ahead for the field in the entire message.
Also arrays can be discontinuous both in xml and in pb. Also if
the archiver is requested for a missin
On 2013-03-31 23:40, Kagamin wrote:
A "MyArchive" example can be useful too. The basic idea is to write a
minimal archive class with basic test code. All methods
assert(false,"method archive(dchar) not implemented"); the example
complies and runs, but asserts. So people take the example and fill
On 2013-04-01 07:15, Kagamin wrote:
It's a pull parser? Hmm... how reordered fields are supposed to be
handled? When the archiver is requested for a field, it will probably
need to look ahead for the field in the entire message. Also arrays can
be discontinuous both in xml and in pb. Also if the
On 2013-03-31 23:57, Kagamin wrote:
Well, the basic idea of EXI and similar standards is that you can have 2
types of serialization: built-in when you keep schema in the serialized
message - which value belongs to which field (this way you can read and
write any data structure) or schema-informe
On 2013-04-01 01:39, Jesse Phillips wrote:
I'm not well versed in PB or Orange so I'd need to play around more with
both, but I'm pretty sure Orange would need changes made to be able to
make the claim PB is supported. It should be possible to create a binary
format based on PB.
Isn't PB binar
On Monday, 1 April 2013 at 08:53:51 UTC, Jacob Carlborg wrote:
On 2013-04-01 01:39, Jesse Phillips wrote:
I'm not well versed in PB or Orange so I'd need to play around
more with
both, but I'm pretty sure Orange would need changes made to be
able to
make the claim PB is supported. It should be
On 04/01/2013 01:13 PM, Jesse Phillips wrote:
On Monday, 1 April 2013 at 08:53:51 UTC, Jacob Carlborg wrote:
On 2013-04-01 01:39, Jesse Phillips wrote:
I'm not well versed in PB or Orange so I'd need to play around more with
both, but I'm pretty sure Orange would need changes made to be able t
AFAIK, it's opposite: an array serialized in chunks, and they are
concatenated on deserialization. Useful if you don't know how
much elements you're sending, so you send them in finite chunks
as the data becomes available. Client can also close connection,
so you don't have to see the end of th
Oh, wait, it looks like a (possibly infinite) range rather than
an array.
On 04/01/2013 02:37 PM, Kagamin wrote:
AFAIK, it's opposite: an array serialized in chunks, and they are
concatenated on deserialization. Useful if you don't know how much
elements you're sending, so you send them in finite chunks as the data
becomes available. Client can also close connection, s
On Monday, 1 April 2013 at 18:41:57 UTC, Matt Soucy wrote:
The "packed repeated fields" section explains it and breaks it
down with an example. If the client can close like that, you
probably don't want to use packed.
Why not? If you transfer a result of google search, the client
will be able
So as we see PB doesn't support arrays (and ranges). Hmm...
that's unfortunate.
On 04/01/2013 03:11 PM, Kagamin wrote:
On Monday, 1 April 2013 at 18:41:57 UTC, Matt Soucy wrote:
The "packed repeated fields" section explains it and breaks it down
with an example. If the client can close like that, you probably don't
want to use packed.
Why not? If you transfer a result of
On Monday, 1 April 2013 at 17:24:05 UTC, Matt Soucy wrote:
From what I got from the examples, Repeated fields are done
roughly as following:
auto msg = fields.map!(a=>a.serialize())().reduce!(a,b=>a~b)();
return ((id<<3)|2) ~ msg.length.toVarint() ~ msg;
Where msg is a ubyte[].
-Matt Soucy
I t
On Monday, 1 April 2013 at 19:37:12 UTC, Matt Soucy wrote:
It's not really strange, because of how it actually does the
serialization. A message is recorded as length+serialized
members. Members can happen in any order. Packed repeated
messages would look like...what? How do you know when one
On 04/01/2013 04:38 PM, Jesse Phillips wrote:
On Monday, 1 April 2013 at 17:24:05 UTC, Matt Soucy wrote:
From what I got from the examples, Repeated fields are done roughly as
following:
auto msg = fields.map!(a=>a.serialize())().reduce!(a,b=>a~b)();
return ((id<<3)|2) ~ msg.length.toVarint() ~
On 04/01/2013 04:54 PM, Kagamin wrote:
On Monday, 1 April 2013 at 19:37:12 UTC, Matt Soucy wrote:
It's not really strange, because of how it actually does the
serialization. A message is recorded as length+serialized members.
Members can happen in any order. Packed repeated messages would look
l
On Monday, 1 April 2013 at 21:11:57 UTC, Matt Soucy wrote:
And therefore, it supports arrays just fine (as repeated
fields). Yes. That last sentence was poorly-worded, and should
have said "you'd just end up with the un'packed' data with an
extra header."
It says repeated messages should be m
On 04/02/2013 12:38 AM, Kagamin wrote:
On Monday, 1 April 2013 at 21:11:57 UTC, Matt Soucy wrote:
And therefore, it supports arrays just fine (as repeated fields). Yes.
That last sentence was poorly-worded, and should have said "you'd just
end up with the un'packed' data with an extra header."
On 2013-04-01 19:13, Jesse Phillips wrote:
Let me see if I can describe this.
PB does encoding to binary by type. However it also has a schema in a
.proto file. My first concern is that this file provides the ID to use
for each field, while arbitrary the ID must be what is specified.
The secon
On 2013-03-24 22:03, Jacob Carlborg wrote:
std.serialization (orange) is now ready to be reviewed.
I've been working on a binary archive with the following format:
FileFormat := CompoundArrayOffset Data CompoundArray
CompoundArrayOffset := AbsOffset # Offset of the compound array
AbsOffset :=
On 04/02/2013 03:21 AM, Jacob Carlborg wrote:
On 2013-04-01 19:13, Jesse Phillips wrote:
Let me see if I can describe this.
PB does encoding to binary by type. However it also has a schema in a
.proto file. My first concern is that this file provides the ID to use
for each field, while arbitra
On 2013-04-02 15:38, Matt Soucy wrote:
Unfortunately, only partially correct. Optional isn't an "option", it's
a way of saying that a field may be specified 0 or 1 times. If two
messages with the same ID are read and the ID is considered optional in
the schema, then they are merged.
With "opti
On 04/02/2013 10:52 AM, Jacob Carlborg wrote:
On 2013-04-02 15:38, Matt Soucy wrote:
Unfortunately, only partially correct. Optional isn't an "option", it's
a way of saying that a field may be specified 0 or 1 times. If two
messages with the same ID are read and the ID is considered optional in
On 2013-04-02 18:24, Matt Soucy wrote:
Again, my misunderstanding. I assumed you were talking about taking a
pre-existing struct, not one generated from the .proto
It doesn't really matter where the struct comes from.
You could easily receive 3,1,2 or 2,1,3 or any other such combination,
and
On Sunday, 24 March 2013 at 21:03:59 UTC, Jacob Carlborg wrote:
std.serialization (orange) is now ready to be reviewed.
I'm fundatemtaly agaisnt the integration of Orange into the std
lib.
The basic problem is that there is no flag in the D object model
for the serialization (e.g: the "publis
On 2013-06-15 20:54, Baz wrote:
I'm fundatemtaly agaisnt the integration of Orange into the std lib.
The basic problem is that there is no flag in the D object model for the
serialization (e.g: the "published" attribute in Pascal).
Why does that matter?
--
/Jacob Carlborg
On Saturday, 15 June 2013 at 18:54:35 UTC, Baz wrote:
The basic problem is that there is no flag in the D object
model for the serialization
Yep, and std.serialization adds it, via UDA.
In the same order of idea, the doc about RTI is null. In fact
there's even no RTI usefull for an "academic"
On Tuesday, 2 April 2013 at 18:50:04 UTC, Jacob Carlborg wrote:
It probably makes sense if one sends the data over the network
and the data is mostly value based. I usually have an object
hierarchy with many reference types and objects passed around.
I think that's kinda the sticking point wit
On 2013-06-17 15:31, Francois Chabot wrote:
I think that's kinda the sticking point with Orange for me. Integrating
it in my current project implied bringing in a huge amount of code when
my needs are super-straightforward.
Why is that a problem?
I actually ended up writing myself a super-li
My point is that serializing POD-based structures in D is so
simple that
using a one-size-fit-all serializer made to handle absolutely
everything
feels very wasteful.
You don't have to use it.
But I do use it, quite a bit. Whenever I have any form of
polymorphic serialization, Orange is exc
On 2013-06-17 16:39, Francois Chabot wrote:
I will grant that making Orange part of Phobos will alleviate the
project bloat issue, which is a huge deal. But as it stands, to me, it
only handles a subset of what std.serialization should.
So what would you like it to handle? I assume you want a
On Monday, 17 June 2013 at 20:59:31 UTC, Jacob Carlborg wrote:
On 2013-06-17 16:39, Francois Chabot wrote:
I will grant that making Orange part of Phobos will alleviate
the
project bloat issue, which is a huge deal. But as it stands,
to me, it
only handles a subset of what std.serialization s
On 2013-06-18 00:06, Francois Chabot wrote:
Well, the main thing that I want on my end is a O(1) memory footprint
when dealing with hierarchical data with no cross-references.
That's kind of hard since it creates data. But if you mean except for
the data then it stores some meta data in order
Any news on this?
On 06/10/14 12:24, "Daniele Bondì" " wrote:
Any news on this?
No, I'm currently working on D/Objective-C [1].
[1] http://wiki.dlang.org/DIP43
--
/Jacob Carlborg
59 matches
Mail list logo
