[capnproto] Schema Refactoring and Unique IDs
Hi all,
Suppose I have a simple schema file:
0xabbeabbeabbeabbe;
struct Foo {
val @0 : UInt32;
};
struct Bar {
val @1 : UInt32;
};
I would like to move Bar into a separate schema file. If I understand the
docs correctly, then this
will change Bar's unique ID.
I have two questions about that:
1. Will changing Bar's unique ID cause backwards incompatibility with
old messages that are serialized with the old ID?
2. If so, what can I do to prevent this? I would like my change to have
no side-effects (a pure no-op).
Thanks!
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/aa247efb-ca69-40fa-97a7-415792fd0c1dn%40googlegroups.com.
Re: [capnproto] Schema Refactoring and Unique IDs
Thanks for the quick responses and clarifications!
My actual schema file is compiled in C++ and Java. As such, it has the
following preamble:
using Cxx = import "/capnp/c++.capnp";
$Cxx.namespace("MyNamespace");
using Java = import "/capnp/java.capnp";
$Java.package("com.myorg");
$Java.outerClassname("MyOuterClass");
When I try to run capnp compile, I get the following:
error: Import failed: /capnp/java.capnp
I have tried some of the workarounds in this thread but had no luck:
https://github.com/capnproto/capnproto-java/issues/17
If I comment out the Java bits and just compile in C++ (which works for
me), will this have any effect on the unique IDs for the structs in my
schema file?
Thanks!
On Wednesday, November 25, 2020 at 3:15:57 PM UTC-5 ken...@cloudflare.com
wrote:
> Erin is correct.
>
> For #2, the command-line syntax to have the compiler echo back capnp
> format (with all type IDs defined explicitly) is:
>
> capnp compile -ocapnp foo.capnp
>
> This of course requires that `capnp` and the generator plugin
> `capnpc-capnp` are in your $PATH, which they should be after installing
> Cap'n Proto globally. If you haven't installed it globally, you can do:
>
> path/to/capnp compile -opath/to/capnpc-capnp foo.capnp
>
> In any case, the output will go to the terminal (no files are generated).
>
> -Kenton
>
> On Wed, Nov 25, 2020 at 1:52 PM Erin Shepherd wrote:
>
>> 1. This depends. The only places that the IDs get used "behind the
>> scenes" are (a) those on interfaces are used in RPC calls to identify the
>> interface and (b) when encoding schema annotations
>>
>> On the other hand, someone might be explicitly reading the ID from the
>> schema file or the constant from the generated code
>>
>> 2. capnpc can be asked to generate capnp format output. In that case,
>> it'll emit a copy of the schema with comments stripped and all
>> automatically generated IDs inserted. You can grab the ID (and syntax) from
>> there
>>
>> - Erin
>>
>> On Wed, 25 Nov 2020, at 20:39, Matt Stern wrote:
>>
>> Hi all,
>>
>> Suppose I have a simple schema file:
>>
>> 0xabbeabbeabbeabbe;
>>
>> struct Foo {
>> val @0 : UInt32;
>> };
>> struct Bar {
>> val @1 : UInt32;
>> };
>>
>> I would like to move Bar into a separate schema file. If I understand the
>> docs <https://capnproto.org/language.html#unique-ids> correctly, then
>> this will change Bar's unique ID.
>>
>> I have two questions about that:
>>
>>1. Will changing Bar's unique ID cause backwards incompatibility with
>>old messages that are serialized with the old ID?
>>2. If so, what can I do to prevent this? I would like my change to
>>have no side-effects (a pure no-op).
>>
>> Thanks!
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to capnproto+...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/aa247efb-ca69-40fa-97a7-415792fd0c1dn%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/capnproto/aa247efb-ca69-40fa-97a7-415792fd0c1dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to capnproto+...@googlegroups.com.
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/a98c4968-c814-4c16-a7ed-a299b3d10688%40www.fastmail.com
>>
>> <https://groups.google.com/d/msgid/capnproto/a98c4968-c814-4c16-a7ed-a299b3d10688%40www.fastmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/40e32c13-631f-4f31-85e1-89eb9ef3b49cn%40googlegroups.com.
Re: [capnproto] Schema Refactoring and Unique IDs
Great, thanks everyone. I will get the current unique ID from the capnp
tool and specify it explicitly for my struct before I do my refactoring.
On Wed, Nov 25, 2020, 5:26 PM Ian Denhardt wrote:
> The ID doesn't affect the encoding itself, so the basic things will
> still work.
>
>
> You can avoid changing the id by specifying it explicitly, e.g.
>
> struct Bar 0xfeefefffefeefefe {
> val @0 :UInt32;
> }
>
> You can discover the current id by running:
>
> capnp compile -ocapnp myschema.capnp
>
> Which will output a version of the schema including the ids, as well as
> some other information.
>
> -Ian
>
> Quoting Matt Stern (2020-11-25 14:39:01)
> >Hi all,
> >
> >Suppose I have a simple schema file:
> >
> >0xabbeabbeabbeabbe;
> >
> >struct Foo {
> >
> > val @0 : UInt32;
> >
> >};
> >
> >struct Bar {
> > val @1 : UInt32;
> >};
> >
> >I would like to move Bar into a separate schema file. If I understand
> >the [1]docs correctly, then this will change Bar's unique ID.
> >
> >I have two questions about that:
> >
> > 1. Will changing Bar's unique ID cause backwards incompatibility with
> >old messages that are serialized with the old ID?
> > 2. If so, what can I do to prevent this? I would like my change to
> >have no side-effects (a pure no-op).
> >
> >Thanks!
> >
> >--
> >You received this message because you are subscribed to the Google
> >Groups "Cap'n Proto" group.
> >To unsubscribe from this group and stop receiving emails from it, send
> >an email to [2]capnproto+unsubscr...@googlegroups.com.
> >To view this discussion on the web visit
> >[3]
> https://groups.google.com/d/msgid/capnproto/aa247efb-ca69-40fa-97a7-
> >415792fd0c1dn%40googlegroups.com.
> >
> > Verweise
> >
> >1. https://capnproto.org/language.html#unique-ids
> >2. mailto:capnproto+unsubscr...@googlegroups.com
> >3.
> https://groups.google.com/d/msgid/capnproto/aa247efb-ca69-40fa-97a7-415792fd0c1dn%40googlegroups.com?utm_medium=email&utm_source=footer
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAL717_QtdWwfHQCK4NbEU8%2BHymqgsKg_kQ-i3cO9Yt19fxb0SQ%40mail.gmail.com.
[capnproto] Schemas for "internal data"
Hello!
Suppose team A produces data with the following schema:
struct Data {
id @1 : Uint64;
stuff @2: InternalData;
}
Now, team B consumes this data but doesn't want care about the "stuff"
field, and team B does not want to include the schema files for
InternalData. So in the consumer code, this schema is used instead:
struct Data {
id @1: Uint64;
stuff @2: Void; // Reserved for team A internal data.
}
Is this safe to do? If it's relevant, team A produces with C++ and team B
consumes with Java.
(For the record, InternalData is not privacy sensitive or anything like
that. I know that team B could peek into that struct even without the
schema and reverse engineer).
Thanks!
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/52db7309-5771-4ba6-ad7d-490fb7078a83n%40googlegroups.com.
Re: [capnproto] Schemas for "internal data"
Thanks Ian. Yes, InternalData is a struct. I'll give AnyPointer a look! On Tue, Feb 16, 2021, 2:32 PM Ian Denhardt wrote: > Quoting Matt Stern (2021-02-16 14:21:43) > > > Is this safe to do? If it's relevant, team A produces with C++ and > > team B consumes with Java. > > No; the representations of Void and InternalData are different (I assume > the latter is a struct), so you'll end up with two types that are > wire-incompatible. > > Rather than using `Void`, you can use `AnyPointer`, in which case this > will work provided that `InternalData` is some pointer type (struct, > list, or interface). > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAL717_Rx1J0tLYBuT5fOSgh_Z3BU_Lj1my6Ap9i09R7zzKME0A%40mail.gmail.com.
[capnproto] Schema Mismatch Between Client and Server
Hi Capnp folks,
I have a producer that writes events to a queue. The schema looks like:
struct Event {
union {
foo @0 : Foo;
bar @1 : Bar;
internal @2 : Internal;
}
}
There are downstream consumers of this queue that are meant to ignore the
"internal" field -- in fact, we don't even provide the Internal struct
definition to them.
The downstream consumers have had a lot of log spam lately because they use
this stripped down schema to generate code:
struct Event {
union {
foo @0 : Foo;
bar @1 : Bar;
}
}
and then handle the union with a switch/case like so:
switch (reader.which()) {
case FOO: return handle(reader.getFoo());
case BAR: return handle(reader.getBar());
default: return logError("A new message type exists that we don't know
about!");
}
The default case in the switch makes sense -- if a new message type appears
from the producer, we should surface that in some way so the consumers know
to update their schemas and handle the new message. However, we don't want
to trigger this case for the known Internal type, which consumers don't
need to handle.
I am thinking of giving the consumers a new schema that looks like:
struct Event {
union {
foo @0 : Foo;
bar @1 : Bar;
internal @2: Void;
}
}
so they can explicitly ignore the internal messages but still log errors
for other new messages that may appear.
My question boils down to: Is it safe for the producer to assign one type
(struct Internal) to field 2 but the consumer to assign another type
(Void)? Or would this cause issues?
Thanks!
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/f94fe404-b080-4654-bfc6-c578c411cf55n%40googlegroups.com.
Re: [capnproto] Schema Mismatch Between Client and Server
Awesome, thanks so much!
On Tue, Feb 14, 2023, 8:01 PM Kenton Varda wrote:
> Hi Matt,
>
> Your idea would work initially, but if an @3 field were ever added, it
> could end up incompatible.
>
> Instead, assuming `Internal` is a struct type, you can instead declare
> `internal` to have type `AnyPointer`:
>
> struct Event {
> union {
> foo @0 : Foo;
> bar @1 : Bar;
> internal @2: AnyPointer;
> }
> }
>
> Since `Internal` is a struct, a field of type `Internal` is represented as
> a pointer. Since `AnyPointer` is also a pointer, it'll produce the same
> layout.
>
> At the risk of over-engineering, another option would be to use generics:
>
> struct Event(InternalType) {
> union {
> foo @0 : Foo;
> bar @1 : Bar;
> internal @2: InternalType;
> }
> }
>
> This way, you and your consumers can actually use exactly the same
> definition of `Event`. Your consumers would use `Event<>` (which is
> equivalent to `Event`). In your internal code with
> knowledge of the internal type, you'd use `Event`.
>
> -Kenton
>
> On Tue, Feb 14, 2023 at 1:21 PM Matt Stern wrote:
>
>> Hi Capnp folks,
>>
>> I have a producer that writes events to a queue. The schema looks like:
>>
>> struct Event {
>> union {
>> foo @0 : Foo;
>> bar @1 : Bar;
>> internal @2 : Internal;
>> }
>> }
>>
>> There are downstream consumers of this queue that are meant to ignore the
>> "internal" field -- in fact, we don't even provide the Internal struct
>> definition to them.
>>
>> The downstream consumers have had a lot of log spam lately because they
>> use this stripped down schema to generate code:
>>
>> struct Event {
>> union {
>> foo @0 : Foo;
>> bar @1 : Bar;
>> }
>> }
>>
>> and then handle the union with a switch/case like so:
>>
>> switch (reader.which()) {
>> case FOO: return handle(reader.getFoo());
>> case BAR: return handle(reader.getBar());
>> default: return logError("A new message type exists that we don't know
>> about!");
>> }
>>
>> The default case in the switch makes sense -- if a new message type
>> appears from the producer, we should surface that in some way so the
>> consumers know to update their schemas and handle the new message. However,
>> we don't want to trigger this case for the known Internal type, which
>> consumers don't need to handle.
>>
>> I am thinking of giving the consumers a new schema that looks like:
>>
>> struct Event {
>> union {
>> foo @0 : Foo;
>> bar @1 : Bar;
>> internal @2: Void;
>> }
>> }
>>
>> so they can explicitly ignore the internal messages but still log errors
>> for other new messages that may appear.
>>
>> My question boils down to: Is it safe for the producer to assign one type
>> (struct Internal) to field 2 but the consumer to assign another type
>> (Void)? Or would this cause issues?
>>
>> Thanks!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to capnproto+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/f94fe404-b080-4654-bfc6-c578c411cf55n%40googlegroups.com
>> <https://groups.google.com/d/msgid/capnproto/f94fe404-b080-4654-bfc6-c578c411cf55n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAL717_Rt-%2BRygcn_ewFCLaqhLYQ3MVuVsXbRP1M7jZmsHOkwJw%40mail.gmail.com.
