Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

['Kenton Varda' via Cap'n Proto]

Sounds cool!

Note that the KJ TLS implementation is pretty weird largely to work with
OpenSSL's API being weird. Hopefully the library you're using has a nicer
API and so the implementation will be easier...

-Kenton

On Thu, Dec 9, 2021 at 12:32 PM Jens Alfke  wrote:

> I'm starting to implement support for SecretHandshake
> , a
> "secure-channel based on a a mutually authenticating key agreement
> handshake, with forward secure identity metadata". shs1-c
>  implements the crypto part,
> resulting in a pair of symmetric stream-cipher keys; beyond that I'm going
> to copy and paste and hack the C++ Cap'n Proto TLS code
> ,
> despite being a total newbie at kj.
>
> Basically all I need to do is create a Cap'n Proto RPC connection that
> splices into the TCP I/O and initially does a couple of data exchanges via
> shs1-c, then filters the data streams through the ciphers.
>
> I'm writing in case anyone has knowledge about the kj side of this that
> they'd like to share.
>
> I'll reply here once I've got this working, and I plan to release the code
> as open source.
>
> --Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/fe6b6564-3f08-478e-af5c-2bf461ea0e81n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D0H-Kkdy_oNWmn7t-65kPMguyRT3JYGz5O09t%3DE8g%2Bmg%40mail.gmail.com.

Re: [capnproto] Unsubscribe in pub/sub model

['Kenton Varda' via Cap'n Proto]

What I usually do is something like:

auto impl = kj::heap();
auto& ref = *impl;
Subscriber::Client client = kj::mv(impl);
rq.setSubscriber(client);

Now you can keep a copy of `client` locally, and as long as it still
exists, then `ref` remains valid -- because `client` is itself a strong
reference to the object.

Note this doesn't work if you need to get notification of when the
`SubscriberImpl` is dropped from the remote side, since holding a strong
ref locally prevents the destructor from being called. If this is an issue,
then you need to do something different. What I usually do here is store a
pointer somewhere, and then have the destructor null out that pointer. This
effectively implements a weak reference.

-Kenton

On Thu, Dec 2, 2021 at 2:22 PM Jens Alfke  wrote:

> I'm also implementing pub-sub, so I was glad to see this thread before I
> wasted too much time. I'm implementing this pattern, but having trouble on
> the client side.
>
> In terms of the example interface, I've created my SubscriberImpl class,
> and written the code to send the "subscribe" message. Now I want to store
> the returned Subscription capability reference in my SubscriberImpl, so it
> can own it and drop it when it's done.
>
> However, I can't figure out how to keep a reference to the SubscriberImpl,
> since I have to move it to the Request object (calling setSubscriber) and
> afterwards it's gone, so I can't call it again.
>
> auto rq = remotePublisher.subscribeRequest();
> auto impl = kj::heap();
> rq.setSubscriber(std::move(impl));
> auto promise = rq.send().then([&](auto response) {return
> response.getSubscription();});
> // *somehow convey the promise to the SubscriberImpl...?*
>
> I'm sure this is just due to my incomplete understanding of how
> Promise/Client/Server objects work...
> Thanks,
>
> --Jens
>
> On Monday, November 22, 2021 at 8:53:42 AM UTC-8 ken...@cloudflare.com
> wrote:
>
>> Hi Mitsuo,
>>
>> I recommend designing the interface like this:
>>
>> interface EventPublisher{
>> interface Subscriber {
>> updateEvent @0 (event: Int32) -> ();
>> }
>>
>> interface Subscription {}
>> subscribe @0 (subscriber: Subscriber) -> (result: Int32,
>> subscription: Subscription);
>> # To unsubscribe, drop the returned `subscription`.
>> }
>>
>>
>> Here, subscribe() returns a `subscription` object. This object has no
>> methods. But, when the capability is dropped, then the destructor will run
>> on the server side. Atn that point, you can remove the subscription.
>>
>> A big advantage of this approach is that it handles connection failures
>> gracefully. If the client randomly disconnects, the `subscription`
>> capability is automatically dropped, thus unsubscribing the client. This
>> way you don't end up stuck sending messages to a disconnected subscriber.
>>
>> It also solves your problem because you can remember arbitrary metadata
>> about the subscriber within the `Subscription` object, so you know what to
>> remove when it's destroyed.
>>
>> -Kenton
>>
>> On Mon, Nov 22, 2021 at 12:40 AM mitsuo 
>> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to implement pub/sub like communication model with the
>>> following scheme.
>>> *pubsub.capnp*
>>> *interface EventPublisher{*
>>> *interface Subscriber {*
>>> *updateEvent @0 (event: Int32) -> ();*
>>> *}*
>>> *subscribe @0 (subscriber: Subscriber) -> (result: Int32);*
>>> *unsubscribe @1 (subscriber: Subscriber) -> (result: Int32);*
>>> *}*
>>>
>>> I'm using *kj::Vector m_subscribers
>>> *to store the current subscribers.
>>> When I try to implement unsubscribe and remove the Subscriber from the
>>> Vector, I couldn't find good method to do that.
>>> Could you give me some advice?
>>>
>>> *server implementation*
>>> *class EventPublisherImpl final : public EventPublisher::Server {*
>>> * protected:*
>>> *  ::kj::Promise subscribe(SubscribeContext context) {*
>>> *cout << "subscribe request received" << endl;*
>>> *m_subscribers.add(context.getParams().getSubscriber());*
>>> *return kj::READY_NOW;*
>>> *  }*
>>>
>>> *  ::kj::Promise unsubscribe(UnsubscribeContext context) {*
>>> *cout << "unsubscribe request received" << endl;*
>>> *auto unsub = context.getParams().getSubscriber();*
>>>
>>> *// I want to remove usub from subscribers like
>>> m_subscribers[unsub].erase();*
>>> *// But I couldn't find a method to compare*
>>> *// "EventPublisher::Subscriber::Client" such as == operator or
>>> public method*
>>> *// to distinguish the client.*
>>> *//*
>>> *// One solution is having an additional argument(id) for this
>>> purpose but*
>>> *// that requres additional management of ID.*
>>> *//  subscribe @0 (subscriber: Subscriber, id: Int32) -> (result:
>>> Int32);*
>>> *//  unsubscribe @1 (id: Int32) -> (result: Int32);*
>>>
>>> *// what I can do is erase everything but this is not my goal*
>>> *m_subscribers

Re: [capnproto] Accept connection in new process

['Kenton Varda' via Cap'n Proto]

Yeah, C++ error messages are the worst. :(

You kind of have to get used to knowing that "can't convert" errors are
usually because you forgot to kj::mv...

-Kenton

On Tue, Nov 30, 2021 at 10:08 AM pepijn de vos 
wrote:

> Ah I solved it... by not storing sim in a variable. Or probably kj::mv
> would have done the job too.
> It's kinda unfortunate the error you get just tells you it's the wrong
> type, rather than.. hey you should move the thing.
> Onwards!!
> Pepijn
>
> On Tue, Nov 30, 2021 at 4:50 PM pepijn de vos 
> wrote:
>
>> Hi Kenton,
>>
>> I got back to this problem and tried implementing the fork idea but ran
>> into another silly problem where the C++ templates are smarter than me it
>> seems.
>> You recommended to not use the EzRPC server, but there is a great lack of
>> examples of other ways to go about it.
>>
>> I found a Rust example that uses the low-level API but of course not
>> exactly the C++ one:
>> https://github.com/capnproto/capnproto-rust/blob/master/capnp-rpc/examples/calculator/server.rs
>> I found some code that's copy-pasted all over using github search that
>> seems to use the lower level API:
>> https://github.com/Nickan/nickan.github.io/blob/b0eb31c33a3b6720606974f9576e663f3b7852ca/drawio/etc/sandstorm/server.c%2B%2B#L420-L422
>>
>> However, I cannot quite get it to work.
>>
>> kj::AsyncIoContext ctx = kj::setupAsyncIo();
>> auto stream = ctx.lowLevelProvider->wrapSocketFd(clientFd);
>> auto network = capnp::TwoPartyVatNetwork(*stream, capnp::rpc::twoparty::
>> Side::CLIENT);
>> kj::Own fs = kj::newDiskFilesystem();
>> const kj::Directory &dir = fs->getCurrent();
>>
>> auto sim = kj::heap(dir);
>> auto rpc = capnp::makeRpcServer(network, sim);
>>
>> So I set up asyncio, wrap my unix socket FD into a stream, use that to
>> create a network (client or server??)
>> The problem is the call to makeRpcServer. SimulatorImpl is a ::Server
>> subclass.
>> This seems to match the random copy-pasted example, but gives compile
>> errors that it can't convert sim to whatever is expected.
>> I tried reading the source code, which tells me it expects a restorer or
>> a bootstrap, but the restorer is deprecated, but also what EzRPC seems to
>> be using.
>> I tried looking at the tests but that appears to do the same thing, just
>> pass it an owned instance of the implementation??
>>
>> https://github.com/capnproto/capnproto/blob/e5af75ff0513e6d971fa7b48e8108427766e51f9/c%2B%2B/src/capnp/rpc-test.c%2B%2B#L1273-L1279
>> I am at a loss.
>>
>> Pepijn
>>
>>
>> On Thu, May 6, 2021 at 8:22 PM Kenton Varda 
>> wrote:
>>
>>> I don't recommend using EzRPC. That interface turns out to be too
>>> restrictive.
>>>
>>> To set up a KJ event loop, use kj::setupAsyncIo().
>>>
>>> Then to initiate or accept Cap'n Proto RPC connections, use
>>> capnp::TwoPartyClient / capnp::TwoPartyServer.
>>>
>>> For FD passing, create a capnp server object that overrides the virtual
>>> method `kj::Maybe Capability::Server::getFd()` to return an FD number.
>>> Then on the Client objects pointing to that server, you can call `getFd()`
>>> to get that file descriptor -- even from another process, as long as the
>>> connection between the two is a unix domain socket.
>>>
>>> -Kenton
>>>
>>> On Thu, May 6, 2021 at 1:07 PM pepijn de vos 
>>> wrote:
>>>
 Thanks for the suggestions.
 This is still for the simulation server, so each connection is pretty
 expensive anyway, and it's unlikely there will be more in flight than the
 machine has cores available. It might even make sense to have a pool as you
 suggest, and just reject connections if there are no cores available.
 Concern is that you need to supervise these pools in case they crash. With
 fork you can just launch a new process. Any pointers to useful functions
 for this kinds of hacks would be appreciated. I don't have a clue how I'd
 go about obtaining and sending file descriptors over RPC and making new
 servers out of them. I should probably start by studying EzRPC and
 kj::LowLevelAsyncIoProvider.

 Pepijn

 On Thu, May 6, 2021 at 7:48 PM Kenton Varda 
 wrote:

> Ohhh, sorry, I misread. If you haven't started the KJ event loop until
> after the fork, you are probably OK.
>
> You can definitely construct a KJ socket from a raw socket using
> kj::LowLevelAsyncIoProvider.
>
> But if you're OK with the entire connection being handled in the
> isolated process, there are some other options too:
> - Have a pool of processes that are all waiting to accept from the
> same file descriptor. Each connection will only be accepted by one of the
> processes. Make sure that process doesn't accept a new connection until 
> the
> old connection is done.
> - Cap'n Proto supports sending file descriptors between processes in
> RPC messages. So you could have one process that is accepting connections,
> but each time it does, it sends the file descriptor to

Re: [capnproto] Can't put C++ Client object in std::map

['Kenton Varda' via Cap'n Proto]

This is arguably a bug in the C++ standard library, but the reason it hits
KJ particularly hard is because of KJ's philosophy about constness:

https://github.com/capnproto/capnproto/blob/master/style-guide.md#constness

In particular:
- Constness should be transitive. This implies that copying from a const
value can only be allowed when the copy is a deep copy. Otherwise, making a
copy would discard the transitive constness on the shared backing objects.
Cap'n Proto `Client` objects are copy-by-refcount, so shallow copies, hence
cannot be const copies.
- Constness should imply thread safety. Cap.'n Proto `Client` objects are
refcounted, and the refcounting is not thread-safe (since thread-safe
refcounting is very slow and these objects are tied to a thread-local event
loop anyway).

Unfortunately, the C++ standard library mostly takes the view that `const`
is shallow. To be fair, this is consistent with how built-in pointer and
reference types work, but it is also a much less useful way of using const.

Annoyingly, the C++ standard library containers work just fine with
move-only types, but choke on types that are non-const-copyable. These
containers really ought to default to using move semantics, or
automatically fall back to it when the copy constructor is non-const. I
consider it a bug in the standard library implementation that it errors
instead.

Luckily, you can work around the problem by wrapping the type in a struct
that only has a move constructor, not a copy constructor, which forces the
library to use move semantics only:

struct Wrapper {
  Wrapper(Wrapper&) = delete;
  Wrapper(const Wrapper&) = delete;
  Wrapper(Wrapper&&) = default;
  T value;
}

Now you can use `std::map>`.

Another alternative is to use `kj::HashMap`, which doesn't suffer these
problems, and is faster than `std::unordered_map`.

-Kenton

On Wed, Dec 1, 2021 at 11:59 AM Jens Alfke  wrote:

> Hi! I'm getting started with capnp, using C++ bindings (with C++17
> compiled by Clang 12.) I'm having trouble storing remote object references,
> i.e. xx::Client objects, in STL collections. For example:
>
> std::map foobars;
> ...
> Foobar::Client c = promise.wait(waitScope).getFoobar();
> foobars.insert({name, c});// Compile error
>
> The error is "*the parameter for this explicitly-defaulted copy
> constructor is const, but a member or base requires it to be non-const*."
>
> I eventually worked out that the error is because *Client's copy
> constructor takes a `Client&` parameter, i.e. requires a mutable reference,
> which breaks the std::pair class* because it needs to be able to copy a
> const reference. And if you can't put a client into a std::pair, you can't
> put it into a std::map or std::unordered_map.
>
> I assume there must be a reason for leaving out the standard `const` in
> the copy constructor's parameter; but I can't work out what it would be. I
> know Client objects are wrappers for a pointer to a ref-counted value, so
> copy-constructing one bumps the value's ref-count, but that value is a
> separate object so it shouldn't matter whether the Client reference is
> const or not. (FYI, I've implemented my own ref-counted objects in C++ so
> I'm pretty familiar with the details...)
>
> Is there a good workaround for storing RPC client objects in STL maps?
> Thanks!
>
> --Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/63af6853-ac43-4c75-a161-a4c939199e93n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnFdZ-gxkT_kCbA8%3DQcMeD1-pWf89Y-obx3r3ti7g7B6w%40mail.gmail.com.

Re: [capnproto] Converting AnyStruct to DynamicStruct

['Kenton Varda' via Cap'n Proto]

Hi Vaci,

`anyReader.as(structSchema)` should do it.

-Kenton

On Tue, Nov 30, 2021 at 2:54 PM Vaci  wrote:

> Apologies if there is an obvious answer, but f I have a
> capnp::StructSchema, and a capnp::AnyStruct::Reader, then can I create a
> capnp::DynamicStruct::Reader from them, and if so, how?
>
> Thanks
> Vaci
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/1de9997d-b015-4ce8-bf32-bf08dce167f4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DsP11EOK8tv5DnMuVWL2PW6FRU2kT%3Dk%2BNW%3DGn_XTpWtQ%40mail.gmail.com.

Re: [capnproto] flow limit related deadlock?

['Kenton Varda' via Cap'n Proto]

It sounds like what Go is providing today would be equivalent to a KJ API
that returns two promises: once which waits for backpressure, and one which
waits for the RPC return value. In principle, we could provide a new
version of `send()` in C++ which provides this. But if it only recognizes
backpressure from the first-hop socket, this could cause more harm than
good, causing people to write apps that tend to cause proxy buffer bloat.
Backpressure in Cap'n Proto really needs to be based on return messages to
handle proxies correctly. This is what `-> stream` provides.

(Note we definitely wouldn't want to stall the whole KJ event loop when one
connection has backpressure. Applications may be doing many concurrent
tasks on the same event loop.)

-Kenton

On Wed, Nov 24, 2021 at 6:05 AM Erin Shepherd  wrote:

> On Wed, 24 Nov 2021, at 02:28, Ian Denhardt wrote:
>
> Quoting Kenton Varda (2021-11-23 19:20:50)
> > Cap'n Proto doesn't provide any backpressure from the underlying TCP
> > connection to the app, except through streaming. If you just make a ton
> > of calls all at once without waiting for returns, you'll bloat your
> > memory with unsent messages. And possibly worse: if the capability
> > bounces through a proxy, and you have a fast connection (say, a unix
> > socket) to the proxy, but a slow connection from the proxy to the
> > eventual destination, you'll end up bloating the proxy's memory.
>
> This is not true of the Go implementation, which currently blocks
> until the message has been written to the socket (we don't currently
> treat streaming specially, but presumably we could keep the blocking
> API when we add that; I don't think we even need to treat the annotation
> specially, we should be able to do it for all calls). So I don't think
> this applies to a scenario where both sides of the connection work like
> the Go implementation. But I hadn't thought about the proxy issue
> (where the proxy might be using a different implementation); thank
> you for pointing that out.
>
>
> I guess this is a matter of futures implementation: callback based futures
> (like kj) often do not provides such a backpressure mechanism. Othe
> solutions (like Go fibers or Rust poll-based futures) can easily provide
> such a mechanism, though
>
> If a concurrency limiter were provided at the capnp-server level (i.e.
> allow no more than X RPCs to be in flight at once), I wonder if that would
> help things?
>
> --
>
> As to the original topic: I wonder if a message could be added which
> advises the peer as to limits on the number of questions and/or the maximum
> size of them which may be in flight? The client might then be able to
> thrrottle itself and the sender could then reject (immediately, with an
> error) any calls once that limit was exceeded. This would keep the socket
> unblocked for finish messages
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/19d6b0bb-83d5-4e22-b3d1-eeb2efb10c1b%40www.fastmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmxKav171EdQkf5iP90BRxe2iBmcwgOrG_v4NBUX3y%2BPA%40mail.gmail.com.

Re: [capnproto] flow limit related deadlock?

['Kenton Varda' via Cap'n Proto]

On Tue, Nov 23, 2021 at 5:01 PM Ian Denhardt  wrote:

> Ok, I think I get it, let me know if I have this right:
>
> The correct thing to do is to handle congestion/flow control for
> multiple calls on each object individually, using something like
> the mechanisms provided by the C++ implementiation's streaming
> construct. This is important so that calls on different objects
> originating from the same vat do not cause problems with one
> another (whereas from TCP's standpoint, they're the same stream,
> so it can't help). This also provides backpressure wrt. the
> receiving vat; it avoids queueing too many calls on the connection
> overall.
>

Right.


> Another check of my understanding: am I correct in thinking that in a
> client implementation that's just a single threaded loop calling
> methods on one object in sequence, it would not cause a problem to
> skip in-process flow control and let the TCP connection deal with it,
> since there is only one stream involved anyway? Assuming of course there
> is something like the flow limit provided by the vat on the other side
> of the connection.
>

Cap'n Proto doesn't provide any backpressure from the underlying TCP
connection to the app, except through streaming. If you just make a ton of
calls all at once without waiting for returns, you'll bloat your memory
with unsent messages. And possibly worse: if the capability bounces through
a proxy, and you have a fast connection (say, a unix socket) to the proxy,
but a slow connection from the proxy to the eventual destination, you'll
end up bloating the proxy's memory.

-Kenton


>
> -Ian
>
> Quoting Kenton Varda (2021-11-23 17:32:44)
> >On Tue, Nov 23, 2021 at 3:59 PM Ian Denhardt <[1]i...@zenhack.net>
> >wrote:
> >
> >  What are apps *supposed* to do here? It isn't clear to me where else
> >  the
> >  backpressure is supposed to come from?
> >
> >Apps should cap the number of write()s they have in-flight at once.
> >(`-> stream` helps a lot with this, as it'll automatically figure out
> >how many is a good number of requests to have in flight.)
> >
> >  Most apps are using sandstorm-http-bridge anyway, so they're just
> >  acting
> >  like normal http servers -- which generally write out data to the
> >  response stream as fast as the socket will take it. Skimming
> >  sendRequest() in the bridge's source, it looks like it just copies
> >  that
> >  data directly into the response stream. So I am confused as to what
> >  a
> >  "well written" app would do?
> >
> >sandstorm-http-bridge currently only does one outstanding write RPC at
> >a time. The code is convoluted but look at pumpWrites() -- it waits
> for
> >each send() to complete before performing the next one.
> >Historically there was a time where it didn't implement such a limit
> >and would just pump the whole response as fast as it got it, which led
> >to the need to do some enforcement on the supervisor side.
> >-Kenton
> >
> > Verweise
> >
> >1. mailto:i...@zenhack.net
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk2Y%2BvpN8YUrVCA_S%2B5nuUoPidh%2BsbiQ5ezO5KEpHbOVA%40mail.gmail.com.

Re: [capnproto] flow limit related deadlock?

['Kenton Varda' via Cap'n Proto]

On Tue, Nov 23, 2021 at 3:59 PM Ian Denhardt  wrote:

> What are apps *supposed* to do here? It isn't clear to me where else the
> backpressure is supposed to come from?
>

Apps should cap the number of write()s they have in-flight at once. (`->
stream` helps a lot with this, as it'll automatically figure out how many
is a good number of requests to have in flight.)

Most apps are using sandstorm-http-bridge anyway, so they're just acting
> like normal http servers -- which generally write out data to the
> response stream as fast as the socket will take it. Skimming
> sendRequest() in the bridge's source, it looks like it just copies that
> data directly into the response stream. So I am confused as to what a
> "well written" app would do?
>

sandstorm-http-bridge currently only does one outstanding write RPC at a
time. The code is convoluted but look at pumpWrites() -- it waits for each
send() to complete before performing the next one.

Historically there was a time where it didn't implement such a limit and
would just pump the whole response as fast as it got it, which led to the
need to do some enforcement on the supervisor side.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D9PVexazqg30yqh9Umb83Z5Vnb7%2Bu8A2q%2BDyZiPX68WQ%40mail.gmail.com.

Re: [capnproto] flow limit related deadlock?

['Kenton Varda' via Cap'n Proto]

On Tue, Nov 23, 2021 at 12:41 PM Ian Denhardt  wrote:

> Wouldn't releasing it on return allow the caller to cause runaway memory
> usage by just never sending the finish? the return entry needs to be kept
> around in case calls are pipelined on it, and itself might take up some
> space (arguably it should count against the limit too...).
>

In all honesty, this isn't a very good defense against resource exhaustion
attacks either way. The problem that motivated it was actually
badly-written apps that would stream a whole large HTTP response body
without paying attention to backpressure.

Technically the callee doesn't need to keep the whole response struct in
memory, only any returned capabilities and their paths. The current
implementation does keep the whole struct though.

My general opinion about resource exhaustion is that it's not so much a
security issue as an abuse issue. These attacks create a temporary
nuisance / disruption, but no permanent damage. It's probably impossible to
anticipate every attack. But, since the attacks don't actually get anything
for the attacker, they are much more rare than you might think, and it's
usually fine to respond reactively by revoking capabilities, banning users,
blocking IPs, etc. If the problem recurs then you start investing new
safeguards against the specific scenario.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkNQVASw4QisbtnOvywO5330tWd9muzMpDx31i26piyCQ%40mail.gmail.com.

Re: [capnproto] flow limit related deadlock?

['Kenton Varda' via Cap'n Proto]

Hmm, I think the intention was that the flow limit should be released on
Return, independent of Finish. But I can totally believe I implemented it
wrong. Could we just change it to be based on Return?

FWIW by default there is no flow limit, it's only enabled in the Sandstorm
supervisor to defend against an app sending excessive requests that end up
queued in memory elsewhere in the system.

-Kenton

On Tue, Nov 23, 2021 at 11:51 AM Ian Denhardt  wrote:

> Hey all,
>
> A few days ago one of my co-maintainers (Louis) alerted me to a deadlock
> in the Go implementation. We've pinned down the cause, and while trying
> to figure out how to fix it, I looked into how the C++ implementation
> handles backpressure.
>
> From what I can tell, the only way backpressure is applied is via the
> flow limit, which limits the total size of arguments to in-flight
> incoming calls. The portion of the quota reserved by a call is returned
> to the pool when the call is removed from the questions table, which
> makes sense, since this is when the memory is actually freed.
>
> However, I see two possible deadlocks that could occur because of this.
>
> The one I am less concerned about is one where calls that depend on one
> another go back and forth between vats, until both vats exceed their
> quota and block on oneanother, causing a deadlock. I am less concerned
> about this since it is basically the RPC equivalent of a stack overflow,
> and could be turned from a crash into a thrown exception by adding a
> timeout or such.
>
> The one I'm more worried about comes up in the context of streaming;
> the problematic scenario is as follows:
>
> Alice in vat A is continuously streaming calls to bob in vat B. It
> possible and expected that at some point alice will cause vat B's
> flow limit to be reached, at which point further calls will block
> until some outstanding calls return. Good so far: this backpressure
> is exactly what we want.
>
> The problem arises after the return message arrives at vat A. vat A
> then sends a finish message, but this message is *behind other calls*,
> so it will not reach vat B until vat B reads in all outstanding calls.
> This will never happen, since vat B is waiting on the flow limit.
>
> I don't know how to avoid this problem with the current protocol as
> specified. One way that almost works is for vat A to just send the
> finish message for each streaming call before the next call is sent,
> relying on the -> stream annotations to know what calls it should do
> this for. but this doesn't quite work since vat B is allowed to
> cancel an ongoing call in response to a finish message. Some
> extension to the protocol to allow non-cancelling finish messages
> would solve this.
>
> Is there a solution that I haven't seen? Are there other ways
> of dealing with this in the protocol?
>
> -Ian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/163768976630.4734.18127071831897488161%40localhost.localdomain
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3Dnik3XG3AQ6K%3DOb1Xq-gvrkLDd2Yi--Qz3U%3DmDZDx8Sg%40mail.gmail.com.

Re: [capnproto] Unsubscribe in pub/sub model

['Kenton Varda' via Cap'n Proto]

Hi Mitsuo,

I recommend designing the interface like this:

interface EventPublisher{
interface Subscriber {
updateEvent @0 (event: Int32) -> ();
}
interface Subscription {}
subscribe @0 (subscriber: Subscriber) -> (result: Int32, subscription:
Subscription);
# To unsubscribe, drop the returned `subscription`.
}


Here, subscribe() returns a `subscription` object. This object has no
methods. But, when the capability is dropped, then the destructor will run
on the server side. Atn that point, you can remove the subscription.

A big advantage of this approach is that it handles connection failures
gracefully. If the client randomly disconnects, the `subscription`
capability is automatically dropped, thus unsubscribing the client. This
way you don't end up stuck sending messages to a disconnected subscriber.

It also solves your problem because you can remember arbitrary metadata
about the subscriber within the `Subscription` object, so you know what to
remove when it's destroyed.

-Kenton

On Mon, Nov 22, 2021 at 12:40 AM mitsuo 
wrote:

> Hi,
>
> I'm trying to implement pub/sub like communication model with the
> following scheme.
> *pubsub.capnp*
> *interface EventPublisher{*
> *interface Subscriber {*
> *updateEvent @0 (event: Int32) -> ();*
> *}*
> *subscribe @0 (subscriber: Subscriber) -> (result: Int32);*
> *unsubscribe @1 (subscriber: Subscriber) -> (result: Int32);*
> *}*
>
> I'm using *kj::Vector m_subscribers *to
> store the current subscribers.
> When I try to implement unsubscribe and remove the Subscriber from the
> Vector, I couldn't find good method to do that.
> Could you give me some advice?
>
> *server implementation*
> *class EventPublisherImpl final : public EventPublisher::Server {*
> * protected:*
> *  ::kj::Promise subscribe(SubscribeContext context) {*
> *cout << "subscribe request received" << endl;*
> *m_subscribers.add(context.getParams().getSubscriber());*
> *return kj::READY_NOW;*
> *  }*
>
> *  ::kj::Promise unsubscribe(UnsubscribeContext context) {*
> *cout << "unsubscribe request received" << endl;*
> *auto unsub = context.getParams().getSubscriber();*
>
> *// I want to remove usub from subscribers like
> m_subscribers[unsub].erase();*
> *// But I couldn't find a method to compare*
> *// "EventPublisher::Subscriber::Client" such as == operator or public
> method*
> *// to distinguish the client.*
> *//*
> *// One solution is having an additional argument(id) for this purpose
> but*
> *// that requres additional management of ID.*
> *//  subscribe @0 (subscriber: Subscriber, id: Int32) -> (result:
> Int32);*
> *//  unsubscribe @1 (id: Int32) -> (result: Int32);*
>
> *// what I can do is erase everything but this is not my goal*
> *m_subscribers.clear();*
> *return kj::READY_NOW;*
> *  }*
>
> * private:*
> *  kj::Vector m_subscribers;*
> *};*
> Thank you,
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/26c4964f-21a2-4fe2-a410-7673786d40c9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DvDSdt-sWgDzEFody-5Y7x1XP9BrWVgOD5bwdh8Tunnw%40mail.gmail.com.

Re: [capnproto] Cloning DynamicStructs

['Kenton Varda' via Cap'n Proto]

No, there's no fundamental limitation here. I think we just need an
overload of `clone()` for this that accounts for the subtle API differences
with `DynamicStruct`, e.g. passing the `Schema` in various places.

-Kenton

On Thu, Nov 18, 2021 at 6:31 AM Vaci  wrote:

> Hi,
>
> Should it be possible to use capnp::clone() to copy a DynamicStruct?
>
> This code currently doesn't compile:
>
> #include 
> #include 
> #include "foo.capnp.h"
> int main() {
>   capnp::MallocMessageBuilder mb;
>   auto root = mb.initRoot();
>   auto a = root.asReader();
>   auto a2 = capnp::clone(a); // succeeds
>   auto b = capnp::toDynamic(a);
>   auto b2 = capnp::clone(b); // fails
> }
>
> ...but I'm not clear what it is about DynamicStructs that make them less
> copyable?
>
> Cheers,
> Vaci
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/ce287ada-fee5-4bed-b353-10c438136d40n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkDJ9sDeL_u9YVZxpyNbO1xMeZc%3DSqRT23jGi8hYLtUcg%40mail.gmail.com.

Re: [capnproto] compiling with automake

['Kenton Varda' via Cap'n Proto]

Once the capnp subproject build is done, you should be able to invoke the
`capnp` compiler straight from the subproject location. I think you'd have
to write a raw make rule that invokes the compiler. You can see how Cap'n
Proto's own build does this here:

https://github.com/capnproto/capnproto/blob/master/c++/Makefile.am#L475

-Kenton

On Thu, Oct 21, 2021 at 1:58 PM SR D <
software.research.developm...@gmail.com> wrote:

> Hi Kenton,
>
> I don't have an issue with setting up and building libraries and linking
> my own programs to external libs like capnproto using Autotools. My problem
> is that, I have a test.capnp file and I don't know how to call/run the
> capnp tool via a Makefile.am to produce the *.capnp.h and *.capnp.c++ files.
>
> Or did I miss your point in understanding that this would happen by
> nesting packages?
>
>
> On Thursday, October 21, 2021 at 1:02:05 PM UTC-5 ken...@cloudflare.com
> wrote:
>
>> It should work as a subpackage. See:
>>
>> https://www.gnu.org/software/automake/manual/html_node/Subpackages.html
>>
>> -Kenton
>>
>> On Thu, Oct 21, 2021 at 12:45 PM SR D 
>> wrote:
>>
>>> I would like to make use of capnproto in an autotools project and
>>> wondering if anyone has any references or examples as to how to do basic
>>> compiling with a Makefile.am. Especially w.r.t. the cmake equivalent of
>>> capnp_generate_cpp().
>>>
>>> Any help appreciated.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/1fc1a675-2ed5-41fc-8dd2-d81e23517cfbn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/038394b5-d57d-4997-b77b-63f8ef19bcfcn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkm1q2aD%3Do41Zzk2w%3Db9D7DmBB2AvpS%3D26wT8oyvjAKZA%40mail.gmail.com.

Re: [capnproto] compiling with automake

['Kenton Varda' via Cap'n Proto]

It should work as a subpackage. See:

https://www.gnu.org/software/automake/manual/html_node/Subpackages.html

-Kenton

On Thu, Oct 21, 2021 at 12:45 PM SR D <
software.research.developm...@gmail.com> wrote:

> I would like to make use of capnproto in an autotools project and
> wondering if anyone has any references or examples as to how to do basic
> compiling with a Makefile.am. Especially w.r.t. the cmake equivalent of
> capnp_generate_cpp().
>
> Any help appreciated.
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/1fc1a675-2ed5-41fc-8dd2-d81e23517cfbn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmMw_4U5shofWZy7tzNFhMQyC7-QjbKMxWU2OV6AUSsyg%40mail.gmail.com.

Re: [capnproto] client to client communication

['Kenton Varda' via Cap'n Proto]

Hi Leon,

Abstractly, what you describe should be easy with Cap'n Proto. One client
can send a capability (interface reference) to the server, and the server
can freely send that capability on to some other client. Cap'n Proto will
automatically arrange to proxy messages from one client to the other,
through the server. (Someday, three-party handoff will allow the clients to
form a direct connection to each other... when we get around to
implementing it.)

I am not very familiar with the Python implementation so I'm not sure I can
help debug what specifically is wrong here. Maybe Jacob (cc'd) can help.

-Kenton

On Thu, Oct 14, 2021 at 2:39 AM Leon Wessels  wrote:

> Hi,
>
> I have a use case where multiple clients connect to a server. A call from
> a client to the server might require the server to call another client. For
> some functionality a direct call from one client to another would be best.
> For performance reasons some clients will be written in C++. For simplicity
> the server and other clients will be written in Python. Is this possible?
> In my attempt below the client call never returns. In the example client
> calls manager which calls provider.
>
> poc.capnp
> @0xdb7253e3ef44cbf9;
>
> interface Manager {
> registerService @0 (service :Service) -> ();
> getService @1 () -> (service :Service);
> get @2 () -> (val :UInt16);
>
> interface Service {
> get @0 () -> (val :UInt16);
> }
> }
>
> client.py
> #!/usr/bin/python3
>
> import sys
> import capnp
> import poc_capnp
>
>
> def start_client(host):
> client = capnp.TwoPartyClient(host)
> ms = client.bootstrap().cast_as(poc_capnp.Manager)
> print(ms.get().wait())
> service = ms.getService().wait()
> print('got service')
> print(service.get().wait())
>
>
> start_client(sys.argv[1])
>
> provider.py
> #!/usr/bin/python3
>
> import sys
> import capnp
> import poc_capnp
> import asyncio
>
>
> class Provider(poc_capnp.Manager.Service.Server):
> def __init__(self):
> self.counter = 0
>
> def get(self, **kwargs):
> print(self.counter)
> self.counter = (self.counter + 1) % 1
> return self.counter
>
>
> async def socket_reader(client, reader):
> while True:
> try:
> data = await asyncio.wait_for(reader.read(4096), timeout=1.0)
> client.write(data)
> except asyncio.TimeoutError:
> pass
>
>
> async def socket_writer(client, writer):
> while True:
> try:
> data = await asyncio.wait_for(client.read(4096), timeout=1.0)
> await writer.write(data.tobytes())
> except asyncio.TimeoutError:
> pass
>
>
> async def start_ipc(host):
> client = capnp.TwoPartyClient()
> ms = client.bootstrap().cast_as(poc_capnp.Manager)
>
> reader, writer = await
> asyncio.wait_for(asyncio.open_connection(*host.split(':')), timeout=1.0, )
> coroutines = [socket_reader(client, reader), socket_writer(client,
> writer)]
> asyncio.gather(*coroutines, return_exceptions=True)
>
> await ms.registerService(Provider()).a_wait()
> while True:
> await asyncio.sleep(1)
>
>
> asyncio.run(start_ipc(sys.argv[1]))
>
> manager.py
> #!/usr/bin/python3
>
> import sys
> import capnp
> import poc_capnp
> import asyncio
>
>
> g_service = None
>
>
> class ManagerImpl(poc_capnp.Manager.Server):
> def __init__(self):
> pass
>
> def registerService(self, service, **kwargs):
> global g_service
> g_service = service
> print('service registered')
>
> def getService(self, **kwargs):
> global g_service
> print('service retrieved')
> return g_service
>
> def get(self, _context, **kwargs):
> global g_service
> print('service called')
> return g_service.get().then(
> lambda value: setattr(_context.results, "val", value)
> )
>
>
> # ---
>
>
> class Server:
> async def myreader(self):
> while self.retry:
> try:
> # Must be a wait_for so we don't block on read()
> data = await asyncio.wait_for(
> self.reader.read(4096),
> timeout=0.1
> )
> except asyncio.TimeoutError:
> # print("myreader timeout.")
> continue
> except Exception as err:
> print("Unknown myreader err: %s", err)
> return False
> await self.server.write(data)
> print("myreader done.")
> return True
>
> async def mywriter(self):
> while self.retry:
> try:
> # Must be a wait_for so we don't block on read()
> data = await asyncio.wait_for(
> self.server.read(4096),
> timeout=0.1
> )
> self.writer.write(data.to

[capnproto] Cap'n Proto 0.9.1 release

['Kenton Varda' via Cap'n Proto]

Hi folks,

I just put up release 0.9.1 which fixes a number of small platform-specific
build issues that were present in 0.9.0.

If you had any compile issues with 0.9.0, hopefully they are fixed now.
Also, FreeBSD and OpenBSD should work! (Thanks @sobomax and @ZhanYF.)

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm8njYO6uboR0p1Nm0QdKD7b-pY4xmbqsLZOK6-%2BRgcgw%40mail.gmail.com.

Re: [capnproto] Signing releases

['Kenton Varda' via Cap'n Proto]

Hi Troy,

Assuming you're downloading a specific release, I'd recommend checking the
hash against a known-good hash, with a command like:

echo 'b28054a7a2bfea42bfc392c8d009630d94d72e8ce86a23ad6f18b5e72574064f
 capnproto-c++-0.9.0.tar.gz' | sha256sum -c

Whenever you update to a newer version, you'd update the hash.

I'm not against also signing releases with an asymmetric key, but I don't
think I'll have time to set up the infrastructure for that any time soon,
sorry.

-Kenton

On Thu, Aug 19, 2021 at 12:05 AM Troy Farrell 
wrote:

>
> Hello everyone,
>
> I am using Cap'n Proto in a Sandstorm project.  As part of the build
> process, a script downloads and builds the Cap'n Proto source from
> capnproto.org.  I would like to have a way to verify that the file I've
> downloaded matches what was released.  Would the release manager (Kenton?)
> please consider posting signatures or hashes for the releases?
>
> Thanks for Cap'n Proto (and Sandstorm)!
> Troy
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/5284d2f4-0912-4855-ab09-ddd2eaa5cb4cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnDRc%2B7Bu8R7yUT1JtJfdmZHAgfiK6X%3DnqkXx3fCoahwA%40mail.gmail.com.

Re: [capnproto] Difficulty having capnp read/write from a kj::[Readable]File, misc. discussion

['Kenton Varda' via Cap'n Proto]

Ah, I see, yeah if you're using the atomic creation stuff then you don't
get to use append I guess.

Hmmm, and now that I look at it, the doc comment for kj::newFileAppender()
says that it assumes it is the only writer... but the implementation does
indeed stat() on every write. That's weird, the doc comment to me suggests
that it wouldn't do that, but would instead just remember the file position
from the previous write. Probably the doc comment needs updating.

Anyway, I suppose we really ought to provide a FileOutputStream class that
tracks the current offset as it writes...

> FsNode::mmap()? I'll try it if I ever need to rewrite the file saving
code.

It's ReadableFile::mmap(), but yes.

-Kenton

On Sun, Aug 15, 2021 at 7:23 PM nyanpasu64  wrote:

> To be honest I'm not sure if I've ever used kj::File to load
>> Cap'n-Proto-format data. The file API was sort of built for other things I
>> was doing that happened to be built on KJ.
>>
> That's the impression I got too.
>
>> But if I were using it, I think I'd do:
>> - Reads always using mmap(), not streams. You'll need a reinterpret_cast
>> to get an ArrayPtr, but no shims needed here.
>>
> FsNode::mmap()? I'll try it if I ever need to rewrite the file saving
> code.
>
>> - Writes probably using Directory::appendFile() to open the file for
>> appending. This gives you an OutputStream so you can directly use
>> capnp::writeMessage(), no shims needed.
>>
> I looked into this, but decided to use Directory::replaceFile() because I
> wanted atomic saving. However, Directory::replaceFile() doesn't support
> returning an AppendableFile.
>
>> > And newFileAppender() is a *bad* idea since it calls stat() on every
>> single file write to query the current length
>>
>> I don't understand what you mean here. kj::newFileAppender() is
>> explicitly documented as not doing this, with the caveat that it won't work
>> correctly if there are multiple concurrent appenders. On the other hand, if
>> you use Directory::appendFile() to open the file, then it uses the O_APPEND
>> flag which makes it the operating system's job to ensure all bytes are
>> appended -- no need for stat().
>>
> In the context of Cap'n Proto, concurrent writes result in a corrupted
> file no matter what behavior is used. And capnp's write methods take a 
> kj::[Buffered]OutputStream&,
> and OutputStream is inherited by AppendableFile but not File, and the
> easiest way to get an AppendableFile from a File calls stat() on every
> write (the hard way is to write a subclass of OutputStream... just found
> out I only need to subclass OutputStream, not AppendableFile). I suppose
> each piece of the puzzle is understandable, but the end result is
> unfortunate. Maybe AppendableFile guarantees that interleaved writes within
> or between processes are interleaved and don't overwrite (resulting in 
> newFileAppender()'s
> design), but Cap'n Proto instead expects that no interleaved writes exist
> at all during writing.
>
> If I remember correctly, I decided against writing to fd integers because
> they seemed less supported on Windows than Linux, and required different
> APIs per OS as well, and supporting arbitrary paths on Windows requires
> 16-bit characters while Linux requires 8-bit characters (whereas
> kj::Filesystem handles this using transparent WTF-8 conversion). I could've
> instead used open() and _wopen() (or _wsopen_s() to satisfy Microsoft's
> API deprecations) rather than kj filesystem operations, and on Windows
> converted WTF-8 to UTF-16 like kj does. (Of course it would be easier to
> use narrow characters on Windows, but depending on the codepage it might
> fail to support unpaired UTF-16 surrogates or even Unicode altogether.) I'd
> still need to build a cross-platform atomic-save library, perhaps with
> C++'s std::filesystem, but its char/wchar handling is scary (though perhaps
> std::filesystem::u8path or path(char8_t) would work).
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/a368d1b6-ff73-44ae-8af2-e91bdd862415n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmk0ze-PoTpBfCPSdVFi-Pd4RSLrhwjfLvLNmTTaa--Cw%40mail.gmail.com.

Re: [capnproto] Difficulty having capnp read/write from a kj::[Readable]File, misc. discussion

['Kenton Varda' via Cap'n Proto]

To be honest I'm not sure if I've ever used kj::File to load
Cap'n-Proto-format data. The file API was sort of built for other things I
was doing that happened to be built on KJ.

But if I were using it, I think I'd do:
- Reads always using mmap(), not streams. You'll need a reinterpret_cast to
get an ArrayPtr, but no shims needed here.
- Writes probably using Directory::appendFile() to open the file for
appending. This gives you an OutputStream so you can directly use
capnp::writeMessage(), no shims needed.

> And newFileAppender() is a *bad* idea since it calls stat() on every
single file write to query the current length

I don't understand what you mean here. kj::newFileAppender() is explicitly
documented as not doing this, with the caveat that it won't work correctly
if there are multiple concurrent appenders. On the other hand, if you use
Directory::appendFile() to open the file, then it uses the O_APPEND flag
which makes it the operating system's job to ensure all bytes are appended
-- no need for stat().

-Kenton

On Sun, Aug 15, 2021 at 5:58 AM nyanpasu64  wrote:

> I've built an (unfinished) application around Cap'n Proto and the kj
> filesystem library (which has the nice property of handling valid and
> invalid Unicode paths properly on both Windows and Linux, using WTF-8 or
> raw bytes), but sadly the filesystem library doesn't have online
> documentation, only doc comments which I struggled to understand the "big
> picture" from.
>
> With that in mind, I'm surprised that Cap'n Proto interoperates so poorly
> with kj, in that capnp can't write to a kj::File or read from a
> kj::ReadableFile without a shim I wrote. And newFileAppender() is a *bad*
> idea since it calls stat() on every single file write to query the current
> length, instead of storing the current write position in the object.
>
> Writes:
> https://gitlab.com/exotracker/exotracker-cpp/-/blob/5109bd9411c9baaca837ae349358da3f5c3742bc/src/serialize.cpp#L421
>
> Reads:
> https://gitlab.com/exotracker/exotracker-cpp/-/blob/5109bd9411c9baaca837ae349358da3f5c3742bc/src/serialize.cpp#L1330
>
> It's also unfortunate that it's so much work to convert between
> kj::ArrayPtr and gsl/std::span, kj::StringPtr (null terminated, not
> slicable) and std::string_view (not always null terminated, slicable),
> juggling ConstData and Data::Reader and char and unsigned char/uint8_t,
> etc. I wish it could be done better, maybe with more conversion operators,
> but it's understandable considering kj was designed to avoid the C++
> standard library.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/97117ffc-3567-40eb-acb6-9007c0a2ad73n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm864DYtDBYbzW2pcAerS5T7%3DcxMOewddJU%3DRKduDaaAg%40mail.gmail.com.

[capnproto] Release candidate 0.9.0-rc1

['Kenton Varda' via Cap'n Proto]

Hi all,

Release candidate out. Please let me know if you spot any problems. (But
our CI is pretty thorough these days so it seems unlikely.)

https://capnproto.org/capnproto-c++-0.9.0-rc1.tar.gz
https://capnproto.org/capnproto-c++-win32-0.9.0-rc1.zip

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmW36SvcB3Btqu55zJKN1O0a7G-VSMeo19QnnufCMYGVg%40mail.gmail.com.

Re: [capnproto] generate make dependencies

['Kenton Varda' via Cap'n Proto]

I think this could be implemented as a regular old code generator plugin
that directly produces the deps files. No need to integrate with the C++
plugin nor output an intermediate representation.

Note that `capnp compile` supports passing multiple `-o` flags to run
multiple plugins, so you can run both your C++ generator and your deps
generator in one pass.

-Kenton

On Thu, Aug 5, 2021 at 1:04 PM Ian Denhardt  wrote:

> I've vaguely wanted this in the past, glad someone is interested and
> motivated enough to get their hands dirty :)
>
> Assuming you're interested in doing this only for C++, you'd want to make
> the changes in the capnpc-c++ code generator plugin. Maybe Kenton can
> provide more specific guidance than this, since I'm not familiar with
> that part of the codebase. But at a high level, the main schema compiler
> is going to feed that program a CodeGeneratorRequest as defined in
> schema.capnp. RequestedFile.imports should have the information you need
> to generate dependencies.
>
> A possible alternative wuold be to write a (separate) schema compiler
> plugin that just dumps the input to a file (say foo.capnp.cgr) and also
> generates Makefile stubs for that file; you could then feed in the
> generated file to capnpc-c++ (or other plugins) manually, and add
> rules like:
>
> %.capnp.cgr: %.capnp
> # ...
> %.capnp.h %.capnp.c++: %.capnp.cgr
> # ...
>
> This would be more general, and might be a little easier.
>
> Note that one thing that's a little sad about gcc's -M is that it will
> fail if the file doesn't exist, so you can't use it to gather dependencies
> for the *.capnp.{h,c++} files as easily as you can non-generated files.
>
> -Ian
>
> Quoting Dirk Jagdmann (2021-08-05 11:50:58)
> >Hello Cap'n'Proto experts,
> >
> >I would like to generate dependencies for Cap'n'Proto schema file
> >imports in the syntax understood by (GNU) make. Similar to the gcc
> >compiler's -M command line argument.
> >
> >I'm happy to look into the Cap'n'Proto source code and see if I can
> >implement this myself, but it would help if you can give me some guide
> >which data structures in which files would have the information that I
> >need (currently processed schema file, list of include directories,
> >...)
> >
> >--
> >You received this message because you are subscribed to the Google
> >Groups "Cap'n Proto" group.
> >To unsubscribe from this group and stop receiving emails from it, send
> >an email to [1]capnproto+unsubscr...@googlegroups.com.
> >To view this discussion on the web visit
> >[2]
> https://groups.google.com/d/msgid/capnproto/b5ffcffe-5a2f-45d4-b248-
> >58118947c67dn%40googlegroups.com.
> >
> > Verweise
> >
> >1. mailto:capnproto+unsubscr...@googlegroups.com
> >2.
> https://groups.google.com/d/msgid/capnproto/b5ffcffe-5a2f-45d4-b248-58118947c67dn%40googlegroups.com?utm_medium=email&utm_source=footer
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/162818661944.3948.8788466351021373915%40localhost.localdomain
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmwiQ%3D_ARkgb%2B71bW4RN6ZP7PZAUTgLz-dLgWDX1N28LA%40mail.gmail.com.

Re: [capnproto] Thread pool executor

['Kenton Varda' via Cap'n Proto]

Hi Alessandro,

KJ doesn't currently have any built-in thread pool implementation. The KJ
async / Promise framework is centered around single-threaded event loops,
so probably isn't what you're looking for.

I guess if I were implementing this I'd write a class that creates threads
using kj::Thread, and has some sort of kj::MutexGuarded>. Each
thread would run a loop where it uses MutexGuarded::when() to wait for the
queue to be non-empty, pops the top job off the queue, and runs it.

-Kenton

On Fri, Jul 30, 2021 at 2:51 PM Alessandro Arcangeli <
alessandroarcangeli...@gmail.com> wrote:

> Hi there,
>
> I'm trying to implement a pool thread executor using KJ.
>
> I mean an executor that has a list of threads and sends the job to  the
> "most empty" one.
>
> Or better, when a thread finish his job (or it is blocked by an IO) it
> automatically pick the next job from the executor.
>
> I looked up in kj's public api but found no way (also combining
> executors/threads/TaskSets etc).
>
> Is there a way to do something like this using KJ?.
>
> Thanks
>
> *Sample Java Equivalent:*
>
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors;
>
> public class TestExecutor {
>   public static void main(String[] args) {
> ExecutorService service = Executors.newFixedThreadPool(8);
>
> for (int i = 0; i < 100; i++) {
>   service.execute(() -> {
> System.out.println("Running from " +
> Thread.currentThread().getName());
> try {
>   Thread.sleep(100);
> }
> catch (InterruptedException e) {
> }
>   });
> }
>
> service.shutdown();
>   }
> }
>
>
> *Sample output:*
>
> Running from pool-1-thread-2
> Running from pool-1-thread-8
> Running from pool-1-thread-3
> Running from pool-1-thread-5
> Running from pool-1-thread-1
> Running from pool-1-thread-7
> Running from pool-1-thread-4
> Running from pool-1-thread-6
> Running from pool-1-thread-7
> Running from pool-1-thread-6
> Running from pool-1-thread-3
> Running from pool-1-thread-8
> ...
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/b331a5f4-4162-46b5-a160-ead0ffecdfddn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnoyApjH6inTHCP%2BL151pe3bTsuxtV63AzVZQOVwBsUwA%40mail.gmail.com.

Re: [capnproto] Cap'n Proto test messages

['Kenton Varda' via Cap'n Proto]

Cool stuff!

The C++ implementation has test.capnp that should cover everything, but is
pretty messy, having evolved slowly together with the C++ implementation.
Definitely nice to have a separate test repository that is clean and really
meant to be consumed by multiple implementations.

-Kenton

On Sun, Jul 18, 2021 at 9:47 AM Brian Olsen  wrote:

> A while back I played around with making my own implementation of Cap'n
> Proto in Rust and for that I really wanted a good test suite that covers
> corner cases. And so I started building that.
>
> Because I thought that good test data might also be relevant to other
> implementations I tried making a setup that would be easily usable by other
> implementations so that you can compare your implementation to the
> reference C++ one.
> The result is: https://github.com/griff/capnp-test-suite
>
> It consists of schemas and messages as well as JSON representation of
> those messages as the capnp tool sees them. The idea is that most languages
> have some JSON library that you can use to read the JSON messages and you
> can then compare that to how your implementation reads the binary message.
>
> I would love you hear your input.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/1d3d3010-8636-4aa6-8705-77233b63dcf5n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DcA0RULRihZuu1fToF8J39VqqQ0Mj9gp8NYegfGwp_AA%40mail.gmail.com.

Re: [capnproto] Selecting the correct message type?

['Kenton Varda' via Cap'n Proto]

Sorry, I'm not sure what you mean. `myns::Person` isn't a real type, it's
just an empty class that acts as a namespace containing the `Reader` and
`Builder` types. `Reader` is your "handle to the data", and its getters are
the only way to read the underlying data (the getters read directly from
the raw underlying byte buffer).

-Kenton

On Mon, Jul 19, 2021 at 3:45 PM SR D <
software.research.developm...@gmail.com> wrote:

> Ok, so after some thinking, I can put something together to communicate
> the type. Let's say I now know the type upon receiving the data, I make
> this call ...
>
> auto m = reader.getRoot();
>
> Here, m is of type myns::Person::Reader. So although I know the type, I
> looked at the person.capnp.h file and found no way to get a handle to the
> actual data I care about, the myns::Person object, rather just the getters.
>
> I understand I can rebuild the object by calling the individual fields
> like m.getXXX()'s, but I'm trying to do this in a generic way and not have
> to call the individual fields to get a myns::Person and pass the object to
> another thread for processing. So, my goal is to not have to call any proto
> object methods (getters or setters) and hope to call something from the
> Reader to get some sort of reference to the object since I know it's type.
>
> Is that possible?
>
>
> On Saturday, July 17, 2021 at 12:22:10 PM UTC-5 ken...@cloudflare.com
> wrote:
>
>> There is no way to find out a message's type if you don't know it
>> already. So, you'll need to arrange to communicate the type yourself. The
>> easiest way is to write an "envelope" message that is a union of all your
>> other messages.
>>
>> struct OuterMessage {
>>   union {
>> foo @0 :FooMessage;
>> bar @1 :BarMessage;
>> baz @2 :BazMessage;
>>   }
>> }
>>
>> Then, always send OuterMessage as the top-level message type.
>>
>> -Kenton
>>
>> On Sat, Jul 17, 2021 at 11:55 AM SR D 
>> wrote:
>>
>>> If the sender is sending any number of different data types that have
>>> been serialized across the network, how does one know which type is coming
>>> in so that it can be pulled off using FlatArrayMessageReader's::getRoot<>?
>>>
>>> E.g., say this while loop is constantly reading serialized data from the
>>> network and results in a vector of message buffers.
>>>
>>> for (auto& msg : messages) {
>>> auto arr = kj::ArrayPtr(
>>> reinterpret_cast(msg),
>>> size / sizeof(capnp::word));
>>>
>>> capnp::FlatArrayMessageReader reader(arr);
>>>
>>> // how would one know what type to use here?
>>> auto msg = reader.getRoot();
>>> }
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/c89926fe-3b18-4e0f-9d30-8f8134b78a0an%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/9545bbfc-38a3-44d4-bf5d-2f059e32dfa3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn_vtojuU4eLcg%3DJNDvYj1c0ttbP1njscPnD65tXb-vTQ%40mail.gmail.com.

Re: [capnproto] Selecting the correct message type?

['Kenton Varda' via Cap'n Proto]

There is no way to find out a message's type if you don't know it already.
So, you'll need to arrange to communicate the type yourself. The easiest
way is to write an "envelope" message that is a union of all your other
messages.

struct OuterMessage {
  union {
foo @0 :FooMessage;
bar @1 :BarMessage;
baz @2 :BazMessage;
  }
}

Then, always send OuterMessage as the top-level message type.

-Kenton

On Sat, Jul 17, 2021 at 11:55 AM SR D <
software.research.developm...@gmail.com> wrote:

> If the sender is sending any number of different data types that have been
> serialized across the network, how does one know which type is coming in so
> that it can be pulled off using FlatArrayMessageReader's::getRoot<>?
>
> E.g., say this while loop is constantly reading serialized data from the
> network and results in a vector of message buffers.
>
> for (auto& msg : messages) {
> auto arr = kj::ArrayPtr(
> reinterpret_cast(msg),
> size / sizeof(capnp::word));
>
> capnp::FlatArrayMessageReader reader(arr);
>
> // how would one know what type to use here?
> auto msg = reader.getRoot();
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/c89926fe-3b18-4e0f-9d30-8f8134b78a0an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DdJLEYS9fTpa77DqNGVd7pK6jXtTafyjVN%3D22AnuCdGQ%40mail.gmail.com.

Re: [capnproto] How to align capnproto data to architectures word size?

['Kenton Varda' via Cap'n Proto]

Ideally, you'd update the implementation of `getBuff()` so that it
allocates an aligned buffer. Note that malloc() always returns aligned
buffers, so if you can make it so that the pointer returned by getBuff() is
one that was allocated directly by malloc(), then it should work. Probably,
though, getBuff() is returning a pointer into some larger buffer that was
read off the wire, including some framing bytes that are pushing the data
out of alignment. In this case it may be impractical to get the data to
come out aligned, so you may have no choice but to make a copy. You can do
something like:

auto copy = kj::heapArray(size / sizeof(capnp::word));
memcpy(copy.begin(), buff, size);

The copy will then be aligned.

-Kenton

On Fri, Jul 16, 2021 at 2:39 PM SR D <
software.research.developm...@gmail.com> wrote:

> I am trying to reconstruct received data using Capnproto, but am getting
> the following error.
>
> terminate called after throwing an instance of 'kj::ExceptionImpl'
>   what():  capnp/arena.c++:76: failed: expected
> reinterpret_cast(segment.begin()) % sizeof(void*) == 0 [4 == 0];
> Detected unaligned data in Cap'n Proto message. Messages must be aligned to
> the architecture's word size. Yes, even on x86: Unaligned access is
> undefined behavior under the C/C++ language standard, and compilers can and
> do assume alignment for the purpose of optimizations. Unaligned access may
> lead to crashes or subtle corruption. For example, GCC will use SIMD
> instructions in optimizations, and those instrsuctions require alignment.
> If you really insist on taking your changes with unaligned data, compile
> the Cap'n Proto library with -DCAPNP_ALLOW_UNALIGNED to remove this check.
> stack: 7f4233c40ca5 7f423bbc40cf6 7f42bbccaa27 55f370c02321
> 55f770cc18e3 55f77ac01331 55f77a0b12da 55f7a0cb127b 55fc70c0d1ed
> 55fe70c01137 7f42bffa7de3 7f42b5cc7608 7f42b5aa6292
>
> It is from this snippet where I am receiving the buffer and trying to
> rebuild the data structure.
>
> const void* buff = getBuff();
>
> auto arr = kj::ArrayPtr(
> reinterpret_cast(buff),
> size / sizeof(capnp::word));
>
> capnp::FlatArrayMessageReader msgReader(arr);
>
> auto msg = msgReader.getRoot(); // error occurs at this line
> auto fieldA = msg.getFieldA();
> auto fieldB = msg.getFieldB();
>
>
> The buffer was constructed and sent using:
>
> capnp::MallocMessageBuilder message;
> auto stuff = message.initRoot();
> stuff.setFieldA("A");
> stuff.setFieldB("B");
>
> const auto msgToSend = capnp::messageToFlatArray(message);
> const auto msgAsChars = msgToSend.asChars();
>
> // make const void* and send
>
> How do I align the data to the architectures word size as the error says?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/c6755765-f1f3-4de6-af29-dfdb2bb83881n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkra6KiBTGKZeK7X3HE52V4_gfeXqUJBA5yOLgChgW__Q%40mail.gmail.com.

Re: [capnproto] Use with RTOS-based Embedded Systems

['Kenton Varda' via Cap'n Proto]

Hi Kenny,

Avoiding malloc should be easy. With custom subclasses of MessageReader and
MessageBuilder, you can do your memory management any way you want. Cap'n
Proto's serialization layer is very good about avoiding memory allocation
unless you ask for it.

(The RPC layer, on the other hand, does lots of dynamic allocation, and
there's no way around that. But, it sounds like you're interested in the
serialization only.)

-Kenton

On Wed, Jun 9, 2021 at 3:03 PM Kenny Koller  wrote:

> Thank you Kenton.
>
> I certainly understand that the "big server" use-case is far more common
> (embedded software engineers everywhere are weeping in their Cap'n
> Crunch**) and I appreciate the clarification regarding CAPNP_LITE.
>
> Personally, I find that code space is much less of a concern these days
> within reason (though it shouldn't be completely ignored). Many of these
> microcontrollers have quite a bit of flash. This is obviously not true for
> very inexpensive devices but there are plenty of applications where the
> cost of the processor is negligible relative to the device as a whole.
>
> The more pressing issue is avoiding (possible) memory fragmentation when
> using new or malloc. C++ has been my language for choice for my embedded
> development for more than 20 years. I suspect, if I had time, I could have
> looked into overriding some classes to provide my own chunks of memory.
> Just having say 16 256 byte chunks that I could recycle would probably do
> the trick.
>
> Thanks again,
>
> Kenny
>
> **With crunch berries because we're adults.
>
> On Wed, Jun 9, 2021 at 10:44 AM Kenton Varda 
> wrote:
>
>> Hi Kenny,
>>
>> The code is fairly modular, so I think it should be possible to pull out
>> a subset that has a reasonably small code footprint. However, this is
>> admittedly not a well-traveled path, so you'll be a bit on your own. The
>> maintainers of the C++ implementation (such as myself) mainly use it on big
>> beefy servers, so we haven't had a whole lot of reason to work on
>> optimizing code footprint.
>>
>> CAPNP_LITE might be a good starting point. Note that that option was not
>> really designed to minimize code footprint, but rather was designed to
>> identify a subset of code that could be compiled on MSVC circa 2013. MSVC
>> has since improved, and is now able to compile the whole codebase. Hence,
>> CAPNP_LITE is a bit vestigial now and may have bitrotted somewhat. But, it
>> definitely does cut out a lot of bulkier parts of the library, which you'd
>> also want to remove in an embedded system. So it might be a good guide.
>>
>> Another option that might be less work would be to use c-capnproto, which
>> I think is more explicitly designed for constrained use cases.
>>
>> -Kenton
>>
>> On Mon, Jun 7, 2021 at 10:52 AM Kenny Koller 
>> wrote:
>>
>>> Hi,
>>>
>>> My team and I were hoping to use Cap'n P on our system to pass messages
>>> around via UDP. There is a powerful Linux system running a control system
>>> but the nodes are STM32F7 devices (1 MB flash, 320 kB of RAM, 216 MHz
>>> Cortex-M7 processor) running FreeRTOS with the LWIP networking stack.
>>>
>>> When I tried to build for the embedded system I ran into some errors
>>> related to a dependency on pthreads (in the KJ library I believe).
>>>
>>> I did some searches and saw discussions in this group from 2015-16 that
>>> suggested things like inheriting from MessageBuilder to get around using
>>> malloc or using nanopb instead.
>>>
>>> I don't fully understand what CAPNP_LITE gets me so maybe this would be
>>> helpful.
>>>
>>> What is the thinking today on this topic?
>>>
>>> Thanks,
>>>
>>> Kenny
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/81988491-bcde-4025-b9b8-74f9a06c6eebn%40googlegroups.com
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn3kH_TQd6dNfp867voH-NeJ4ZAPhGoWrHyNQ%2BB3S5t9Q%40mail.gmail.com.

Re: [capnproto] Use with RTOS-based Embedded Systems

['Kenton Varda' via Cap'n Proto]

Hi Kenny,

The code is fairly modular, so I think it should be possible to pull out a
subset that has a reasonably small code footprint. However, this is
admittedly not a well-traveled path, so you'll be a bit on your own. The
maintainers of the C++ implementation (such as myself) mainly use it on big
beefy servers, so we haven't had a whole lot of reason to work on
optimizing code footprint.

CAPNP_LITE might be a good starting point. Note that that option was not
really designed to minimize code footprint, but rather was designed to
identify a subset of code that could be compiled on MSVC circa 2013. MSVC
has since improved, and is now able to compile the whole codebase. Hence,
CAPNP_LITE is a bit vestigial now and may have bitrotted somewhat. But, it
definitely does cut out a lot of bulkier parts of the library, which you'd
also want to remove in an embedded system. So it might be a good guide.

Another option that might be less work would be to use c-capnproto, which I
think is more explicitly designed for constrained use cases.

-Kenton

On Mon, Jun 7, 2021 at 10:52 AM Kenny Koller  wrote:

> Hi,
>
> My team and I were hoping to use Cap'n P on our system to pass messages
> around via UDP. There is a powerful Linux system running a control system
> but the nodes are STM32F7 devices (1 MB flash, 320 kB of RAM, 216 MHz
> Cortex-M7 processor) running FreeRTOS with the LWIP networking stack.
>
> When I tried to build for the embedded system I ran into some errors
> related to a dependency on pthreads (in the KJ library I believe).
>
> I did some searches and saw discussions in this group from 2015-16 that
> suggested things like inheriting from MessageBuilder to get around using
> malloc or using nanopb instead.
>
> I don't fully understand what CAPNP_LITE gets me so maybe this would be
> helpful.
>
> What is the thinking today on this topic?
>
> Thanks,
>
> Kenny
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/81988491-bcde-4025-b9b8-74f9a06c6eebn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DVmRg4_aLQZ-SR4NK89KxryyhHYsKxv%3DpDsGXYjSk%2BMA%40mail.gmail.com.

Re: [capnproto] Waiting with a timeout

['Kenton Varda' via Cap'n Proto]

Yes, I agree there's some missing overview docs regarding Cap'n Proto RPC.
The web page is not very detailed.

FWIW, I recently wrote an overview/tour of KJ:
https://github.com/capnproto/capnproto/blob/master/kjdoc/tour.md

But that doesn't cover Cap'n Proto itself.

> If I use the setupAsyncIo and TwoPartyServer, do I need to use the
> TwoPartyClient, or can that stay an EzClient?

The protocols are the same, so you could use EzClient on the client side
and TwoPartyServer on the server. However, I would recommend avoiding the
EZ interfaces entirely. Over time I've come to regret having created them,
as they are too inflexible for many use cases.

-Kenton

On Tue, Jun 8, 2021 at 1:25 PM Björn Schäpers  wrote:

> Thanks, I will look into that.
>
> Yes I know the code is documented, without that I wouldn't have come this
> far.
> But what is missing, at least for me, is the bigger picture, i.e. how to
> combine
> all these rpc related classes, and what I need to implement and what not.
>
> If I use the setupAsyncIo and TwoPartyServer, do I need to use the
> TwoPartyClient, or can that stay an EzClient?
>
> Regards,
> Björn.
>
> Am 08.06.2021 um 04:38 schrieb Kenton Varda:
> > Hi Björn,
> >
> > Instead of Ez RPC, try using kj::setupAsyncIo() and
> capnp::TwoPartyClient /
> > capnp::TwoPartyServer. These APIs give you more control. In particular,
> > kj::setupAsyncIo() returns an kj::AsyncIoProvider which, among other
> things,
> > provides a kj::Timer API which you can use for timeouts.
> >
> > KJ's and Cap'n Proto's header files are well-commented. We consider the
> header
> > comments to be the API reference documentation.
> >
> > -Kenton
> >
> > On Mon, Jun 7, 2021 at 9:08 PM Björn Schäpers  > > wrote:
> >
> > Hi there,
> >
> > I've started using Cap'n Proto RPC. Until now I only have some tests
> to ensure
> > Cap'n Proto behaves as expected for me.
> >
> > What is missing for me is waiting with a timeout, or I don't see it.
> I want to
> > be able to shut the server down, and NEVER_DONE seems not right for
> that. In my
> > tests I'm using poll with a sleep in a loop.
> >
> > Currently I'm using the Ez RPC, and to be honest the RPC System and
> associates
> > are a bit overwhelming without documentation or good examples.
> >
> > Kind regards,
> > Björn.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "Cap'n Proto" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an
> > email to capnproto+unsubscr...@googlegroups.com
> > .
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/capnproto/d4160a10-c30c-d209-c12f-70b550e5ce0a%40hazardy.de
> .
> >
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmia_ADOJrREK_-FqwPpr25Ke-i5twqCbkU-UCF8XH%3DHw%40mail.gmail.com.

Re: [capnproto] Waiting with a timeout

['Kenton Varda' via Cap'n Proto]

Hi Björn,

Instead of Ez RPC, try using kj::setupAsyncIo() and capnp::TwoPartyClient /
capnp::TwoPartyServer. These APIs give you more control. In particular,
kj::setupAsyncIo() returns an kj::AsyncIoProvider which, among other
things, provides a kj::Timer API which you can use for timeouts.

KJ's and Cap'n Proto's header files are well-commented. We consider the
header comments to be the API reference documentation.

-Kenton

On Mon, Jun 7, 2021 at 9:08 PM Björn Schäpers  wrote:

> Hi there,
>
> I've started using Cap'n Proto RPC. Until now I only have some tests to
> ensure
> Cap'n Proto behaves as expected for me.
>
> What is missing for me is waiting with a timeout, or I don't see it. I
> want to
> be able to shut the server down, and NEVER_DONE seems not right for that.
> In my
> tests I'm using poll with a sleep in a loop.
>
> Currently I'm using the Ez RPC, and to be honest the RPC System and
> associates
> are a bit overwhelming without documentation or good examples.
>
> Kind regards,
> Björn.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/d4160a10-c30c-d209-c12f-70b550e5ce0a%40hazardy.de
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D4LVsQG5SNOu9JQ_LCe3sX_gU4ZFzJXPu80iSTugGYqA%40mail.gmail.com.

Re: [capnproto] Copying existing struct into request params

['Kenton Varda' via Cap'n Proto]

I see. There isn't a great API for this right now. The problem is, once a
struct Builder is already allocated, the struct can't be resized. But if
you're trying to copy the content of another struct into it, it's possible
that other struct was originally created with a newer version of the schema
which had new fields defined, and so it is larger. If the data were copied
from that larger struct into the smaller struct that has been allocated,
the new fields would be silently dropped. I wanted to avoid that situation.

One thing you could do is manually invoke
Capability::Client::typelessReuqest(), manually passing the
appropriate interface and method IDs. Then you get a request where the root
type is AnyPointer, which you can set to an existing struct. This avoids
the problem of dropping unknown fields, since the destination struct isn't
allocated until the source is known, and it is then allocated with the same
size as the source.

Perhaps we should update the code generator to generate RPC client methods
that can accept an existing struct Reader as input, to make this nicer.

-Kenton

On Thu, Jun 3, 2021 at 8:59 AM Vaci  wrote:

> Ah, I figured out that I can use the layered API to decode each item in
> the list individually:
>
> Bar::Client cap;
> kj::ArrayPtr text;
> capnp::MallocMessageBuilder mb;
> capnp::JsonCodec codec;
> kj::ArrayBuilder> promises;
>
> auto value = mb.initRoot();
> codec.decodeRaw(text, value);
>
> for (auto&& entry: value.getArray()) {
>   auto request = cap.fooRequest();
>   codec.decode(entry, request);
>   promises.add(request.send().ignoreResult());
> }
>
> However, I'd still be curious to know how to create a request from an
> existing struct, if that's at all possible!
>
> Many thanks,
> Vaci
>
> On Thursday, 3 June 2021 at 09:10:32 UTC+1 Vaci wrote:
>
>> That's not quite what I want to do. Given that a request is a struct
>> type, I want to copy or assign an existing struct to the whole request.
>>
>> I can use the json decoder to write content into the request struct, but
>> I don't see a way to do that when I'm holding an orphan pointer to a struct
>> that is of the same type as the request.
>>
>> Vaci
>>
>> On Wednesday, 2 June 2021 at 15:36:55 UTC+1 ken...@cloudflare.com wrote:
>>
>>> Hi Vaci,
>>>
>>> An RPC request is always a struct type, not a list, so I assume what you
>>> really mean here is that you want to use the list as a field of the
>>> request? E.g. you have:
>>>
>>> struct Foo {}
>>>
>>> interface Bar {
>>>   foo @0 (foos :List(Foo));
>>> }
>>>
>>>
>>> Then you would do:
>>>
>>> auto req = bar.fooRequest();
>>>
>>> auto orphanage = capnp::Orphanage::getForMessageContaining(req);
>>>
>>> auto orphan = json.decode>(txt, orphanage);
>>>
>>> req.adoptFoos(kj::mv(orphan));
>>>
>>>
>>> -Kenton
>>>
>>> On Wed, Jun 2, 2021 at 6:29 AM Vaci  wrote:
>>>
 I may be missing something obvious, but is it possible to copy the
 content of a struct into the body of a request builder?

 I have an interface of the form:

struct Foo {};

   interface Bar {
  foo @0 Foo;
   };

 I want to import a bunch of Foo structures using the json reader and
 use them to populate the requests. This works fine with a single item,
 because I can just decode directly into the request body:

 auto foo = bar.fooRequest();
 capnp::JsonCodec json;
 json.decode(txt, foo);

 ...but if my input is a list of items, the json decoder will return an
 (orphaned) list of Foo structures, and I can't see a way to use each item
 as request params.

 Vaci







 --
 You received this message because you are subscribed to the Google
 Groups "Cap'n Proto" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to capnproto+...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/capnproto/55a2f540-cece-474f-8795-b65e15f024e1n%40googlegroups.com
 
 .

>>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/206b1c60-4525-487b-82c4-a97db415807cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To

Re: [capnproto] Copying existing struct into request params

['Kenton Varda' via Cap'n Proto]

Hi Vaci,

An RPC request is always a struct type, not a list, so I assume what you
really mean here is that you want to use the list as a field of the
request? E.g. you have:

struct Foo {}

interface Bar {
  foo @0 (foos :List(Foo));
}


Then you would do:

auto req = bar.fooRequest();

auto orphanage = capnp::Orphanage::getForMessageContaining(req);

auto orphan = json.decode>(txt, orphanage);

req.adoptFoos(kj::mv(orphan));


-Kenton

On Wed, Jun 2, 2021 at 6:29 AM Vaci  wrote:

> I may be missing something obvious, but is it possible to copy the content
> of a struct into the body of a request builder?
>
> I have an interface of the form:
>
>struct Foo {};
>
>   interface Bar {
>  foo @0 Foo;
>   };
>
> I want to import a bunch of Foo structures using the json reader and use
> them to populate the requests. This works fine with a single item, because
> I can just decode directly into the request body:
>
> auto foo = bar.fooRequest();
> capnp::JsonCodec json;
> json.decode(txt, foo);
>
> ...but if my input is a list of items, the json decoder will return an
> (orphaned) list of Foo structures, and I can't see a way to use each item
> as request params.
>
> Vaci
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/55a2f540-cece-474f-8795-b65e15f024e1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmPmkbERwaaV3TAzMff5AncWNrnjqXssfMS5fbZ%2BkTiKQ%40mail.gmail.com.

Re: [capnproto] PosixPath error when loading schema

['Kenton Varda' via Cap'n Proto]

Hi Kyle,

I think you need to convert `capnp_path` back into a string before passing
it to `capnp.load()`. Currently it appears you are passing a `Path` object
that you constructed using pathlib, but `capnp.load` doesn't know how to
deal with those, it wants a plain string.

-Kenton



On Tue, Jun 1, 2021 at 9:35 AM Kyle Downey  wrote:

> I have the following simple schema:
>
> @0x8fe87a03768154e9;
>
> enum Side {
> buy @0;
> sell @1;
> }
>
> struct TradeMessage {
> time @0 : Float64;
> tradeId @1 : Int64;
> side @2 : Side;
> size @3 : Float64;
> price @4 : Float64;
> }
>
> struct Level1BookUpdateMessage {
> time @0 : Float64;
> best_bid_qty @1 : Float64;
> best_bid_px @2 : Float64;
> best_ask_qty @3 : Float64;
> best_ask_px @4 : Float64;
> }
>
> which I am loading as follows using pycapnp 1.0.0 on MacOS 11.4 running
> Python 3.8:
>
> import capnp
>
> from pathlib import Path
>
> capnp_path = Path(__file__).parent / '../../../../capnp/serenity-fh.capnp'
> capnp_def = capnp.load(capnp_path)
>
> It fails on load with the following error. Any ideas on what I might be
> doing wrong?
>
> Connected to pydev debugger (build 202.6397.98)
> Traceback (most recent call last):
>   File "", line 991, in _find_and_load
>   File "", line 975, in
> _find_and_load_unlocked
>   File "", line 671, in _load_unlocked
>   File "", line 783, in exec_module
>   File "", line 219, in
> _call_with_frames_removed
>   File
> "/Users/kdowney/dev/shadows/serenity/src/serenity/marketdata/fh/txlog.py",
> line 6, in 
> capnp_def = capnp.load(capnp_path)
>   File "capnp/lib/capnp.pyx", line 4030, in capnp.lib.capnp.load
>   File "capnp/lib/capnp.pyx", line 3276, in
> capnp.lib.capnp.SchemaParser.load
> AttributeError: 'PosixPath' object has no attribute 'endswith'
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/26dda48a-5ca0-4b0c-a42f-930d96b388a4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm9k_WpM%3DGhXinfzm1vv8BDr7%2BHTtWeGpLDkkrcXUGR9w%40mail.gmail.com.

Re: [capnproto] Linker error: struct capnp::_::RawSchema const capnp::schemas

['Kenton Varda' via Cap'n Proto]

Hi Pratik,

When you run the Cap'n Proto code generator, it produces two files: a
header (ending in `.capnp.h`) and a source file (ending in `.capnp.c++`).

Did you make sure to compile the source file into your project?

I'm guessing the missing symbol is from there, since it seems to be
something from capnp::schemas, which is what the `.capnp.c++` file
provides. (Your error message seems to be cut short, by the way. Your error
has the type and namespace of the missing symbol, but not the symbol name,
which should have come next.)

BTW, you probably don't need capnpc.lib -- it contains the capnp compiler
implementation, used for parsing `.capnp` schema definition files. Most
people don't do that at runtime.

-Kenton

On Mon, May 10, 2021 at 6:04 AM Pratik Mankawde 
wrote:

>
> Hi,
> I am getting this linking error on VS2019.
> LNK2001 unresolved external symbol struct capnp::_::RawSchema const
> capnp::schemas
>
> I saw that there are other threads discussing similar error but the
> solution mentioned in them didn't work for me. I am linking against
> following libs in order:
> capnp.lib
> capnpc.lib
> kj.lib
>
> I tried changing order of these libs, didn't work. I am using following
> data format in capnp file. In this test project, I am only serializing and
> de-serializing the data, nothing else.
>
> Following are the VS commands options:
>
> C++: /MP /GS /TP /W4 /Zc:wchar_t /I"D:\ecal\ecal\core\include"
> /I"D:\ecal\_build\complete\ecal\core\include"
> /I"D:\ecal\contrib\ecalproto\include" /I"D:\ecal\thirdparty\protobuf\src"
> /I"D:\capnproto-master\c++\src\capnp\.."
> /I"D:\capnproto-master\c++\src\kj\.." /Zi /Gm- /O2 /Ob2
> /Fd"x64\Release\vc142.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS"
> /D "NDEBUG" /D "_CONSOLE" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope
> /GR /Gd /MD /std:c++14 /Fa"x64\Release\" /EHsc /nologo /Fo"x64\Release\"
> /Fp"x64\Release\eCal5SubscriberTest.pch" /diagnostics:column
>
> Linker: /OUT:"D:\Release\eCal5SubscriberTest.exe" /MANIFEST /NXCOMPAT
> /PDB:"D:\Release\eCal5SubscriberTest.pdb" /DYNAMICBASE
> "D:\capnproto-master\c++\src\capnp\Release\capnp.lib"
> "D:\capnproto-master\c++\src\capnp\Release\capnpc.lib"
> "D:\ecal\_build\complete\ecal\core\Release\ecal_core.lib"
> "D:\capnproto-master\c++\src\kj\Release\kj.lib" "D:\lib\zlib.lib"
> "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib"
> "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib"
> "odbc32.lib" "odbccp32.lib" /MACHINE:X64 /INCREMENTAL:NO
> /PGD:"D:\Release\eCal5SubscriberTest.pgd" /SUBSYSTEM:CONSOLE
> /MANIFESTUAC:"level='asInvoker' uiAccess='false'"
> /ManifestFile:"x64\Release\eCal5SubscriberTest.exe.intermediate.manifest"
> /ERRORREPORT:PROMPT /NOLOGO /TLBID:1
>
>
> I have tried linking against all of these libs together, but I don't think
> I need them all:
> capnp-json.lib
> capnp-rpc.lib
> capnp-websocket.lib
> capnp.lib
> capnpc.lib
> kj-http.lib
> kj-async.lib
> kj-test.lib
> kj.lib
>
> Cap'n Proto Root.sln solution builds fine in VS. All tests pass. I get
> this link error when I use these libs in my test project. I am not getting
> any other error as of now. I am using master branch from github.
> Am I doing something wrong here?
>
> Thanks,
> Pratik
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/1e5ade4b-690f-4bd4-9495-646fa3717850n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmEeJbE0Ym%3DLKGz84btDRBU94CWfwYxA27SsTDMJTHgMQ%40mail.gmail.com.

Re: [capnproto] Accept connection in new process

['Kenton Varda' via Cap'n Proto]

Hi Pepijn,

Yeah, forking won't work well. I don't recommend that. One problem is that
if you fork() with an event loop running, both the parent and child
processes will end up with the same epoll FD and so will receive each
other's events, which will likely lead to chaos.

So I recommend spawning a subprocess and talking to it using RPC over a
unix socket. Yes, it adds some overhead, but really not that much.

-Kenton

On Thu, May 6, 2021 at 2:39 AM pepij...@gmail.com 
wrote:

> Hey,
>
> I'm writing an API for some shared libraries that are written with a
> "binary" use in mind.
> They use a ton of global state and are not thread safe.
> Of course as a server, multiple users could try to use the server at the
> same time, these libraries can't deal with that. What's more they also like
> to segfault sometimes.
>
> So what I'd like to do is have these shared libraries completely in their
> own address space.
> What would be the best way to do that?
>
> One thing I thought could work is have a blocking accept loop, and then
> fork the handling of the connection. Not sure how this would work with kj.
>
> The other solution would be that you launch a process in the method
> handler that runs its own capnp server, and then proxy calls to it. Seems
> like it adds more overhead.
>
> Regards,
> Pepijn
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/99fcf2cf-e6e4-4918-8734-22b7f85cfb85n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmfWx6kk1Ejdr3bsfJQSxAw-aF%2BxA%2BBuAj4QRaj2pZ_uA%40mail.gmail.com.

Re: [capnproto] Generated files .h and .c++

['Kenton Varda' via Cap'n Proto]

The tables are used to support the "schema" and "dynamic" APIs (aka
"reflection"). These APIs are useful for a bunch of things, and in
particular they are used to implement text stringification of messages
(e.g. for debugging) and JSON conversion.

I think some code is also generated in the .c++ file for RPC interfaces.

If you don't get a linker error when you omit the .c++ file, then you must
not be using these features. It's fine to omit the file if you don't see
any errors from doing so. Note, though, that you might find you suddenly
need the .c++ file in the future after some code change, or a future
version of Cap'n Proto might start to require it. I'd generally recommend
making it part of your build, but configuring your linker to drop unused
objects, rather than intentionally omit it.

-Kenton

On Tue, May 4, 2021 at 12:28 PM Nogueira  wrote:

> Hi,
>
> I have been using capnproto to perform serialization and de-serialization
> using mmap. Works pretty well and it's fast!
> I noticed I'm able to compile and use code without the generated .c++
> file, which seems to have some hardcoded tables in it.
> What exactly is the c++ file used for? Am I encountering any problem if I
> use the header but not the c++ file?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/d522df33-da2e-4372-bcb6-a3e760b6871fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D7i_3cnQKTHM8S%3Dk_c_pFg7S_sUh7ZBGVQ07CP2QnaHg%40mail.gmail.com.

Re: [capnproto] How to get server-side implementation from client capability ?

['Kenton Varda' via Cap'n Proto]

I would say that when you are distributing a precompiled binary copy of
your library, you should distribute the `.capnp.h` header along with it, if
you want programs using your library to be able to use that type. Those
programs will need to link against the exact version of the schema that you
compiled into your library, so you might as well provide them with the
matching header. Otherwise, if they try to generate their own copy, they
may run into linker errors due to duplicate symbols -- or worse, subtle
runtime ODR errors if their version didn't exactly match what's in your
library.

With all that said, there's a second layer of this problem, which is that
your library and applications using it need to link against exactly the
same version of Cap'n Proto. Unfortunately, different versions of the Cap'n
Proto library are not ABI-compatible, and headers generated for one version
won't necessarily work with a different version of the library. If you
don't want your library to impose a specific version of Cap'n Proto on the
applications linking with it, then either you need to let each app compile
your library themselves (using the version of capnp they like), or you need
to do symbol-renaming stuff in order to embed a private copy of capnp into
your binary. In the latter case, you can't really distribute the .capnp.h
header since it would not work with any version of Cap'n Proto other than
your private one.

-Kenton

On Tue, Apr 27, 2021 at 2:36 AM Théophile B. 
wrote:

> It works ! Thank you for the expanded explanation, it helped a lot. I
> initially though there would be a "prettier" out-of-the-box solution, but as
> long as it works...
>
> While I have you, I have a more general question. Is it common case to
> distribute Cap'n'Proto genererated headers as part of a library, or
> everything must stay private ?
> Let's say I have defined some enum in the .capnp and I want the users of
> my library to use them.
> For the moment I have a duplicated the definition of this enum in a public
> header, and I do static casts before serialization.
> Le lundi 26 avril 2021 à 17:40:59 UTC+2, ken...@cloudflare.com a écrit :
>
>> You need your `SubscriberImpl` to hold a regular C++ pointer back to the
>> `MediaLibraryImpl` that created it, so that you can access its
>> `publishers_` set through that pointer. You can pass the pointer to
>> `SubscriberImpl`'s constructor.
>>
>> You'll also want to make sure the pointer doesn't become a dangling
>> pointer, so `SubscriberImpl` should also hold a `MediaLibrary::Client`
>> which points to the same `MediaLibraryImpl` -- holding this counts as a
>> strong reference so Cap'n Proto won't destroy the `MediaLibraryImpl`.
>> Inside `addSubscriber()` you can use `thisCap()` to get a
>> `MediaLibrary::Client` pointing to `this`, then you'd pass that to
>> `SubscriberImpl`'s constructor, along with the plain C++ pointer to `this`.
>>
>> -Kenton
>>
>> On Mon, Apr 26, 2021 at 10:25 AM Théophile B. 
>> wrote:
>>
>>> Thank you for your quick answer!
>>>
>>> The `Server` to `Client` "conversion" is done via the bootstrap
>>> capability, but I need the underlying `Server` back when calling
>>> capabilities returned by the bootstrap capability.
>>> Here is a simplified example where I (tried) to implement your advice:
>>>
>>> interface MediaLibrary {
>>>addPublisher @0 (name :Text) -> (pub :Publisher);
>>>addSubscriber @1 (name :Text) -> (sub :Subscriber);
>>>
>>>   interface Publisher {}
>>>
>>>   interface Subscriber {
>>>  listenTo @0 (publisher :Publisher);
>>>}
>>> }
>>>
>>> class MediaLibraryImpl final : public MediaLibrary::Server {
>>> public:
>>> kj::Promise addPublisher(AddPublisherContext context) override {
>>>  my_underlying_library::Publisher* output = ...; // call the
>>> underlying lib
>>> MediaLibrary::Publisher::Client pub = publishers_.add(kj::heap<
>>> PublisherImpl>(output));
>>> context.getResults().setPub(pub);
>>>return kj::READY_NOW;
>>> }
>>>
>>> kj::Promise addSubscriber(AddSubscriberContext context) override {
>>>  my_underlying_library::Subscriber* output = ...; // call the
>>> underlying lib
>>> MediaLibrary::Subscriber::Client sub = subscribers_.add(kj::heap<
>>> SubscriberImpl>(output));
>>> context.getResults().setSub(sub);
>>> return kj::READY_NOW;
>>> }
>>>
>>> private:
>>> capnp::CapabilityServerSet subscribers_;
>>> capnp::CapabilityServerSet publishers_;
>>> };
>>>
>>> class PublisherImpl : public virtual GstDaemon::Publisher::Server {
>>> protected:
>>> PublisherImpl(my_underlying_library::Publisher* pub) : pub_(pub) {}
>>>
>>> private:
>>> my_underlying_library::Publisher* pub_;
>>> }
>>>
>>> class SubscriberImpl : public virtual GstDaemon::Subscriber::Server {
>>> protected:
>>> SubscriberImpl(my_underlying_library::Subscriber* sub) : sub_(sub) {}
>>>
>>> kj::Promise listenTo(ListenToContext context) override {
>>> auto pub = context.getParams().getPublisher(); // GstDaemon::Publisher
>>> ::Client
>>>

Re: [capnproto] How to get server-side implementation from client capability ?

['Kenton Varda' via Cap'n Proto]

You need your `SubscriberImpl` to hold a regular C++ pointer back to the
`MediaLibraryImpl` that created it, so that you can access its
`publishers_` set through that pointer. You can pass the pointer to
`SubscriberImpl`'s constructor.

You'll also want to make sure the pointer doesn't become a dangling
pointer, so `SubscriberImpl` should also hold a `MediaLibrary::Client`
which points to the same `MediaLibraryImpl` -- holding this counts as a
strong reference so Cap'n Proto won't destroy the `MediaLibraryImpl`.
Inside `addSubscriber()` you can use `thisCap()` to get a
`MediaLibrary::Client` pointing to `this`, then you'd pass that to
`SubscriberImpl`'s constructor, along with the plain C++ pointer to `this`.

-Kenton

On Mon, Apr 26, 2021 at 10:25 AM Théophile B. 
wrote:

> Thank you for your quick answer!
>
> The `Server` to `Client` "conversion" is done via the bootstrap
> capability, but I need the underlying `Server` back when calling
> capabilities returned by the bootstrap capability.
> Here is a simplified example where I (tried) to implement your advice:
>
> interface MediaLibrary {
>addPublisher @0 (name :Text) -> (pub :Publisher);
>addSubscriber @1 (name :Text) -> (sub :Subscriber);
>
>   interface Publisher {}
>
>   interface Subscriber {
>  listenTo @0 (publisher :Publisher);
>}
> }
>
> class MediaLibraryImpl final : public MediaLibrary::Server {
> public:
> kj::Promise addPublisher(AddPublisherContext context) override {
>  my_underlying_library::Publisher* output = ...; // call the
> underlying lib
> MediaLibrary::Publisher::Client pub = publishers_.add(kj::heap<
> PublisherImpl>(output));
> context.getResults().setPub(pub);
>return kj::READY_NOW;
> }
>
> kj::Promise addSubscriber(AddSubscriberContext context) override {
>  my_underlying_library::Subscriber* output = ...; // call the
> underlying lib
> MediaLibrary::Subscriber::Client sub = subscribers_.add(kj::heap<
> SubscriberImpl>(output));
> context.getResults().setSub(sub);
> return kj::READY_NOW;
> }
>
> private:
> capnp::CapabilityServerSet subscribers_;
> capnp::CapabilityServerSet publishers_;
> };
>
> class PublisherImpl : public virtual GstDaemon::Publisher::Server {
> protected:
> PublisherImpl(my_underlying_library::Publisher* pub) : pub_(pub) {}
>
> private:
> my_underlying_library::Publisher* pub_;
> }
>
> class SubscriberImpl : public virtual GstDaemon::Subscriber::Server {
> protected:
> SubscriberImpl(my_underlying_library::Subscriber* sub) : sub_(sub) {}
>
> kj::Promise listenTo(ListenToContext context) override {
> auto pub = context.getParams().getPublisher(); // GstDaemon::Publisher
> ::Client
> // No access to CapabilityServerSet !
> // We we would like to do: sub_->listenTo("pub_")
> }
>
> private:
> my_underlying_library::Subscriber* sub_;
> }
>
> If I understand well,  I can now do subscribers_.getLocalServer("GstDaemon
> ::Publisher::Client") from `MediaLibraryImpl`.
> Bu what about SubscriberImpl, where the `listenTo` needs `PublisherImpl` ?
> Is it possible to have access to `MediaLibraryImpl` from `SubscriberImpl`
> ?
> Instead, should I pass references to the `CapabilityServerSet` to `
> PublisherImpl` and `SubscriberImpl` ?
>
> Le lundi 26 avril 2021 à 16:30:55 UTC+2, ken...@cloudflare.com a écrit :
>
>> Hi Théophile,
>>
>> Take a look at CapabilityServerSet:
>>
>>
>> https://github.com/capnproto/capnproto/blob/6b5bcc2c6e954bc6e167ac581eb628e5a462a469/c++/src/capnp/capability.h#L607-L633
>>
>> When you convert your `Server` object to a `Client`, you need to do it
>> using a CapabilityServerSet. Later, you can use the same set to unwrap the
>> `Client` and get the underlying `Server` back -- even if the client has
>> been passed over the network and back in the meantime.
>>
>> -Kenton
>>
>> On Mon, Apr 26, 2021 at 9:25 AM Théophile B. 
>> wrote:
>>
>>> I'm not sure I'm usign the right terminology in my question, but I hope
>>> you will understand my problem with the following example.
>>>
>>> I'm developing a media library where the user can play/pause/stop
>>> different kinds of "pipelines" (source ~ camera, filter ~ processing, sink
>>> ~ display). Depending on their type, these pipeline can listen to each
>>> other, or not (e.g: a source pipeline cannot listen). I'm now working on a
>>> "capnp wrapper" of the library, so that it can be controlled via RPC.
>>>
>>> I have replicated the class inheritance with success:
>>>
>>> interface Publisher {}
>>>
>>> interface Subscriber {
>>> listenTo @0 (publisher :Publisher);
>>> }
>>>
>>> interface Pipeline {
>>> play @0 ();
>>> pause @1 ();
>>> stop @2 ();
>>> }
>>>
>>> interface SourcePipeline extends(Pipeline, Publisher) {}
>>>
>>> interface FilterPipeline extends(Pipeline, Publisher, Subscriber) {}
>>>
>>> interface SinkPipeline extends(Pipeline, Subscriber) {}
>>>
>>> Server-side I have those impl classes, which holds the references to the
>>> underlying objects of the media library (pub_ & sub_).
>>>
>>> class Pu

Re: [capnproto] How to get server-side implementation from client capability ?

['Kenton Varda' via Cap'n Proto]

Hi Théophile,

Take a look at CapabilityServerSet:

https://github.com/capnproto/capnproto/blob/6b5bcc2c6e954bc6e167ac581eb628e5a462a469/c++/src/capnp/capability.h#L607-L633

When you convert your `Server` object to a `Client`, you need to do it
using a CapabilityServerSet. Later, you can use the same set to unwrap the
`Client` and get the underlying `Server` back -- even if the client has
been passed over the network and back in the meantime.

-Kenton

On Mon, Apr 26, 2021 at 9:25 AM Théophile B. 
wrote:

> I'm not sure I'm usign the right terminology in my question, but I hope
> you will understand my problem with the following example.
>
> I'm developing a media library where the user can play/pause/stop
> different kinds of "pipelines" (source ~ camera, filter ~ processing, sink
> ~ display). Depending on their type, these pipeline can listen to each
> other, or not (e.g: a source pipeline cannot listen). I'm now working on a
> "capnp wrapper" of the library, so that it can be controlled via RPC.
>
> I have replicated the class inheritance with success:
>
> interface Publisher {}
>
> interface Subscriber {
> listenTo @0 (publisher :Publisher);
> }
>
> interface Pipeline {
> play @0 ();
> pause @1 ();
> stop @2 ();
> }
>
> interface SourcePipeline extends(Pipeline, Publisher) {}
>
> interface FilterPipeline extends(Pipeline, Publisher, Subscriber) {}
>
> interface SinkPipeline extends(Pipeline, Subscriber) {}
>
> Server-side I have those impl classes, which holds the references to the
> underlying objects of the media library (pub_ & sub_).
>
> class PublisherImpl : public virtual GstDaemon::Publisher::Server {
> protected:
> PublisherImpl(my_underlying_library::Publisher* pub) : pub_(pub) {}
> private:
> my_underlying_library::Publisher* pub_;
> };
>
> class SubscriberImpl : public virtual GstDaemon::Subscriber::Server {
> protected:
> SubscriberImpl(my_underlying_library::Subscriber* sub) : sub_(sub) {}
>
> kj::Promise listenTo(ListenToContext context) override {
> auto pub = context.getParams().getPublisher(); // GstDaemon::Publisher
> ::Client
> // ??? = PublisherImpl::pub_
> sub_->listenTo(???)
> }
>
> private:
> my_underlying_library::Subscriber* sub_;
> };
>
> Now, from the client, I wanted to do something like this:
> //GstDaemon::Subscriber::Client subscriber; // already returned by the
> server
> //GstDaemon::Publisher::Client publisher; // already returned by the
> server
> auto request = subscriber.listenToRequest();
> request.setPublisher(publisher);
> auto promise = request.send();
>
> However, as you can see, on the server it's not possible to get the pub_
> pointer when the listenTo method is called. Is there any way I can get a
> reference to the PublisherImpl here ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/488a63c6-92bc-4bc8-9ab8-e648d3a577bfn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D%3Db_ZwwR-CCSNSDbRYx2eTXGTVZVz5Pr%3DLPaPj%3DZYvCQ%40mail.gmail.com.

Re: [capnproto] Memory issues with captain proto structs?

['Kenton Varda' via Cap'n Proto]

Hi Brando,

It's hard for us to guess what might be the problem without seeing more
code.

-Kenton

On Tue, Mar 30, 2021 at 12:56 PM Brando Miranda 
wrote:

> Hi,
>
> I am doing machine learning with captain proto (because captain proto is
> good at communicating between python and different languages).
>
> The challenge I have is that my data is represented in captain proto
> structs. I load a batch of these proto structs every so often to process
> them with a Neural Network. However, eventually after a certain number of
> iterations it seems I allocated all the system's memory and I get a SIGKILL
> from OOM.
>
> I am unsure why this would happen or where (I've been memory profiling my
> code all day but it is difficult to figure out what part is breaking). I am
> fairly sure it has to do with captain proto because I used to have a
> version of the data set with json files and I didn't have this error but
> now I do. I could directly use the captain proto dataset to create a new
> json file data set to really figure out if that is the case but it seems
> redundant.
>
> I thought it could be that I open a captain proto struct and then I close
> it:
>
>
> file_name = self.convert_to_local_home_path(file_name)
> f = open(file_name)
> ...
> bunch of processing to make it into my python class
> 
> f.close()
> return x, y
>
> I am explicitly closing the file from captain proto so I'd assume that
> isn't the case. But anyway, is there a way to really check if the memory
> errors I am getting are due to captain proto or not?
>
> Thanks, Brando
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/ac8e4eba-11e9-44cc-9095-4313e4b7e544n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkf-x7nHEFb4MTPLhjOC8VU3dTsMipJoyvjmiU6B4KqJA%40mail.gmail.com.

Re: [capnproto] C++: passing around a struct, and Builder<->Reader

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

FWIW someone (not me) wrote this plugin to generate POCS:

https://github.com/nickolasrossi/podgen

I haven't used it nor carefully reviewed it yet, but based on the readme it
looks pretty thorough. Note it uses std types rather than KJ types which
may or may not be to your liking.

-Kenton

On Tue, Mar 30, 2021 at 1:07 PM Topher Cawlfield 
wrote:

> Thanks for the reply. That perspective is very helpful. I've been wanting
> a system like Cap'n Proto for synchronizing a variety of configuration data
> between systems. I was on the fence about just copying the config data back
> and forth to plain old C++ structs, but when I first read the tips and best
> practices section, the *first* bullet point made me think it would be
> great to just pass around the Reader and avoid the extra copying and
> name-mapping code. But since I have an existing code base, not everything
> is going to be read-only. Thus the need for both getFoo and setFoo. My
> structs have about a hundred names each already, so going back and forth
> isn't trivial in terms of hand-written code.
>
> But I'll start with that, and when I can I'll look into what it would take
> to implement generated "C++ structs" and the conversion logic (for where
> benefits of convenience outweigh those of speed).
>
> For my project, I've also been considering FlatBuffers and
> ProtocolBuffers. The two main drawbacks to Cap'n Proto *for me* are this
> whole issue and also the enforcement of camelCase which forces name
> mismatches with existing code. I've also run into some rough weather with
> pycapnp. Still I believe the drawbacks of FlatBuffers and Protobufs are
> even more severe, so I'm still plowing ahead with Cap'n Proto. You have my
> heartfelt thanks for all this, by the way!!
>
> Topher
> On Monday, March 29, 2021 at 8:55:42 PM UTC-6 ken...@cloudflare.com wrote:
>
>> Hi Topher,
>>
>> Unfortunately, Cap'n Proto is not well suited to representing mutable
>> data structures in-memory. Generally, MessageReaders are read-only, and
>> MessageBuilders are write-only. While technically you can modify builders
>> over time, the memory allocation patterns this tends to lead to are not
>> ideal. This is an unavoidable consequence of the memory allocation approach
>> needed to support zero-copy serialization: we need to allocate memory in a
>> contiguous space in order to be able to write it all out at once, but this
>> means memory fragmentation is impossible to avoid when data changes shape
>> over time.
>>
>> This is covered a bit in the "Best practices" section of the docs here:
>> https://capnproto.org/cxx.html#tips-and-best-practices
>>
>> A feature I've long wanted to add to Cap'n Proto is support for
>> generating appropriate "plain old C++ structs" (POCS) that mirror the Cap'n
>> Proto structs, with the ability to copy from a MesasgeReader into a POCS,
>> and from POCS to a MessageBuailder. Using POCS would mean you have to
>> perform at least one copy (not zero-copy anymore), but may be more
>> convenient in some use cases, especially when the structure will be
>> modified in-memory many times.
>>
>> -Kenton
>>
>> On Mon, Mar 29, 2021 at 6:36 PM Topher Cawlfield 
>> wrote:
>>
>>> I'm new to Cap'nProto, but am trying to use it as a replacement for
>>> structs in an existing project. I'm needing to serialize/deserialize
>>> obviously, but also just access the data within various programs. I'm
>>> having a very hard time keeping a MessageBuilder, Builder, and Reader in a
>>> class.
>>>
>>> It would be nice if I could deserialize a CapnProto struct in one
>>> method/function, modify it in another, read it in a third, and serialize in
>>> a fourth. But I'm stuck turning a PackedFdMessadeReader into a
>>> MessageBuilder (I don't mind any few memcpy's needed), and also creating a
>>> Builder instance as a member variable.
>>>
>>> I'm attaching my closest attempt at this. But it fails to print out the
>>> original data value -- I get a null string -- and it fails to write the
>>> modified value out.
>>>
>>> Any suggestions?
>>>
>>> Topher
>>>
>>> mystruct.capnp:
>>> @0xed859a09d409be91;
>>>
>>> struct MyStruct {
>>>   foo @0 :Text;
>>> }
>>>
>>> main.cpp:
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include "mystruct.capnp.h"
>>>
>>> class MyClass {
>>> private:
>>> capnp::MallocMessageBuilder m_message; // Or capnp::MessageBuilder
>>> *m_message maybe
>>>
>>> public:
>>> MyStruct::Builder m_ms;
>>>
>>> MyClass() :
>>> m_message ()
>>> , m_ms (m_message.initRoot()) // sometimes segfaults here but
>>> absolutely required
>>> {
>>> m_ms = m_message.initRoot(); // sometimes helps
>>> }
>>>
>>> void deserializeFrom(std::string filename) {
>>> int fd = 0;
>>> ::capnp::PackedFdMessageReader msg(fd);
>>> auto reader = msg.getRoot();
>>> m_message.setRoot(reader); // This leads to future problems with m_ms
>>> }
>>>
>>> void modify() {
>>> m_ms.setFoo("oh joy");
>>> }
>>>
>>> void serializeTo(std::string filename) {
>>> int fd = 1;
>>> writePackedMessa

Re: [capnproto] C++: passing around a struct, and Builder<->Reader

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Topher,

Unfortunately, Cap'n Proto is not well suited to representing mutable data
structures in-memory. Generally, MessageReaders are read-only, and
MessageBuilders are write-only. While technically you can modify builders
over time, the memory allocation patterns this tends to lead to are not
ideal. This is an unavoidable consequence of the memory allocation approach
needed to support zero-copy serialization: we need to allocate memory in a
contiguous space in order to be able to write it all out at once, but this
means memory fragmentation is impossible to avoid when data changes shape
over time.

This is covered a bit in the "Best practices" section of the docs here:
https://capnproto.org/cxx.html#tips-and-best-practices

A feature I've long wanted to add to Cap'n Proto is support for generating
appropriate "plain old C++ structs" (POCS) that mirror the Cap'n Proto
structs, with the ability to copy from a MesasgeReader into a POCS, and
from POCS to a MessageBuailder. Using POCS would mean you have to perform
at least one copy (not zero-copy anymore), but may be more convenient in
some use cases, especially when the structure will be modified in-memory
many times.

-Kenton

On Mon, Mar 29, 2021 at 6:36 PM Topher Cawlfield 
wrote:

> I'm new to Cap'nProto, but am trying to use it as a replacement for
> structs in an existing project. I'm needing to serialize/deserialize
> obviously, but also just access the data within various programs. I'm
> having a very hard time keeping a MessageBuilder, Builder, and Reader in a
> class.
>
> It would be nice if I could deserialize a CapnProto struct in one
> method/function, modify it in another, read it in a third, and serialize in
> a fourth. But I'm stuck turning a PackedFdMessadeReader into a
> MessageBuilder (I don't mind any few memcpy's needed), and also creating a
> Builder instance as a member variable.
>
> I'm attaching my closest attempt at this. But it fails to print out the
> original data value -- I get a null string -- and it fails to write the
> modified value out.
>
> Any suggestions?
>
> Topher
>
> mystruct.capnp:
> @0xed859a09d409be91;
>
> struct MyStruct {
>   foo @0 :Text;
> }
>
> main.cpp:
> #include 
> #include 
> #include 
> #include 
> #include "mystruct.capnp.h"
>
> class MyClass {
> private:
> capnp::MallocMessageBuilder m_message; // Or capnp::MessageBuilder
> *m_message maybe
>
> public:
> MyStruct::Builder m_ms;
>
> MyClass() :
> m_message ()
> , m_ms (m_message.initRoot()) // sometimes segfaults here but
> absolutely required
> {
> m_ms = m_message.initRoot(); // sometimes helps
> }
>
> void deserializeFrom(std::string filename) {
> int fd = 0;
> ::capnp::PackedFdMessageReader msg(fd);
> auto reader = msg.getRoot();
> m_message.setRoot(reader); // This leads to future problems with m_ms
> }
>
> void modify() {
> m_ms.setFoo("oh joy");
> }
>
> void serializeTo(std::string filename) {
> int fd = 1;
> writePackedMessageToFd(fd, m_message);
> }
> };
>
> int main() {
> MyClass c;
> c.deserializeFrom("some.capnp");
> std::cerr << "foo is " << c.m_ms.getFoo().cStr() << std::endl; // read
> things occasionally
> c.modify();
> c.serializeTo("next.capnp");
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/6ce2d01c-43d4-44a6-9d26-0b3147742f55n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmcwFSOMF%2B4CiJgGbXQaH4zHibSTmPM%2BAn_vwuFxtaT7Q%40mail.gmail.com.

Re: [capnproto] getTimer in RPC & transparent proxy?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Thu, Mar 25, 2021 at 11:38 AM pepij...@gmail.com 
wrote:

>
> Hey,
>
> Another one of these things that's probably good design but somewhat
> puzzling.
> How do I obtain a timer in my RPC server?
> I found you need to get them from the async IO provider.
> The easyrpc server has a method to get it, but then I'm stuck.
> I need to somehow pass the timer to my server implementation, but before I
> create the RPC server I can't access the IO provider, and after I created
> the RPC server my implementation got moved into a private property.
>

The EZ interfaces turn out to be problematic in a lot of ways.

I recommend skipping them and instead using kj::setupAsyncIo() and
capnp::TwoPartyClient/TwoPartyServer to set up your async and RPC
environments more explicitly. Then you'll be able to get the timer before
you construct your server object.

Also note that you can take a (non-owning) reference to your server before
you pass it off to the RPC system. You can assume the server won't be moved
or destroyed while the RPC system is active, so the reference remains valid.


> The second question is: I saw some release notes talking about a proxy not
> having to know the exact message format it's proxying, but is there
> actually a way to make a generic capnproto proxy?
>

You can take a capability received from one RPC connection and pass it over
any other RPC connection. Cap'n Proto will automatically arrange to proxy
requests in this case. In this case, only the caller and callee actually
need to know the schema for the methods being called; the proxy doesn't
need to know. So, for example, you could cast a capability to the type
`capnp::Capability::Client` (which is the base type of all `Client`
objects), then pass it around from machine to machine, and then eventually
use `client.castAs()` to cast back to the object's real type
(or any superclass), and it'll work. You can make calls and they'll pass
through all the proxies to the final object, even if the proxies don't have
the schema for `MyInterface` or only have an older version.


> Similarly I saw a few mentions of HTTP and websocket stuff, but is there
> actually a way to use Capnproto over a websocket? I need none of that right
> now, but it's interesting to know what the options are.
>

Ian mentioned MessageStream. In theory it should be easy to combine that
with KJ HTTP's WebSocket implementation -- at least to the extent that
doing anything with KJ HTTP is "easy". It's like all other KJ interfaces,
weird at first but nice when you get to know it.

-Kenton


>
> Here is my project so far btw https://github.com/NyanCAD/SimServer
> Thanks for all the help so far :)
>
> Cheers,
> Pepijn
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/d9072ac4-2751-4e9e-91f3-a27be38da0dbn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkOGxxGPO-0iVS4izn9YuR9j1%3DgpAc%3DPt1EK2sCY9kuiw%40mail.gmail.com.

Re: [capnproto] Re: CPU-bound RPC command & optional API

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Yeah, you probably need to recompile the python module against the new
version of Cap'n Proto. Unfortunately, it's very hard to remain
ABI-compatible across versions in C++, so KJ and Cap'n Proto don't try.

-Kenton

On Sun, Mar 21, 2021 at 1:13 PM pepijn de vos  wrote:

> I rewrote my queue for newCrossThreadPromiseAndFulfiller, after installing
> Cap'n proto from git, but now Python no longer works :(
> I also installed pycapnp from git, but am getting the following error:
> undefined symbol: _ZN2kj1_24TransformPromiseNodeBase16getInnerForTraceEv
>
> Cheers,
> Pepijn
>
> template
> class AsyncQueue : public kj::AtomicRefcounted
> {
> public:
> AsyncQueue() : paf(kj::newCrossThreadPromiseAndFulfiller()) {}
>
> kj::Promise pop() {
> kj::Locked> queue = mutex.lockExclusive();
> if (queue->empty()) {
> return kj::mv(paf.promise);
> } else {
> T val = queue->front();
> queue->pop();
> return val;
> }
> }
>
> void push(T val) {
> kj::Locked> queue = mutex.lockExclusive();
> if(this->paf.fulfiller->isWaiting()) {
> this->paf.fulfiller->fulfill(T(val));
> paf = kj::newCrossThreadPromiseAndFulfiller();
> } else {
> queue->push(val);
> }
> }
> kj::MutexGuarded> mutex;
> kj::PromiseFulfillerPair paf;
> };
>
> On Fri, Mar 19, 2021 at 7:58 PM pepijn de vos 
> wrote:
>
>> Thanks a lot, I'll look into the level 2 stuff.
>>
>> newCrossThreadPromiseAndFulfiller looks perfect for what I'm trying to do.
>> I actually wanted more of a queue abstraction, which doesn't seem to be
>> there AFAICT, so I'm trying to make one.
>>
>> The basic idea is that when the consumer tries to pop from an empty queue
>> it creates a PromiseAndFulfiller. When the producer pushes a new value it
>> uses an executor to do so on the event loop thread of the consumer, and
>> then checks if the consumer is waiting, if so fulfills the promise, else
>> just pushes onto the queue. It looks like once I get a version with the new
>> cross API, I can use that. For now I have something like this. Happy to
>> make a PR once it works and if you think it's useful.
>>
>> Pepijn
>>
>> template
>> class AsyncQueue
>> {
>> public:
>> AsyncQueue() : paf(kj::newPromiseAndFulfiller()),
>> exec(kj::getCurrentThreadExecutor()) {}
>>
>> kj::Promise pop() {
>> if (queue.empty()) {
>> return kj::mv(paf.promise);
>> } else {
>> T val = queue.front();
>> queue.pop();
>> return val;
>> }
>> }
>>
>> void push(T val) {
>> exec.executeSync([val, this]() {
>> if(this->paf.fulfiller->isWaiting()) {
>> this->paf.fulfiller->fulfill(T(val));
>> paf = kj::newPromiseAndFulfiller();
>> } else {
>> this->queue.push(val);
>> }
>> });
>> }
>> std::queue queue;
>> kj::PromiseFulfillerPair paf;
>> const kj::Executor &exec;
>> };
>>
>> On Fri, Mar 19, 2021 at 7:49 PM Kenton Varda 
>> wrote:
>>
>>> Yes, Executor is a good way to communicate between threads. FWIW you
>>> might also check out newCrossThreadPromiseAndFulfiller(), which was added
>>> very recently. Sometimes it's a better fit than kj::Executor.
>>>
>>> "Level 2" turns out to be something that can't really be built into
>>> libcapnp itself because the design really depends on the execution
>>> environment in which your servers run. A system for saving capabilities and
>>> restoring them later needs to understand how to connect to -- and possibly
>>> even start up -- the appropriate server and storage system. So, for
>>> example, Sandstorm.io has implemented level 2 of the protocol in a way that
>>> is appropriate for it, but there ends up being not much that libcapnp
>>> itself ca ndo to help. You can take a look at persistent.capnp to see a
>>> suggestion for how to structure a level 2 implementation, but it's not much
>>> more than a suggestion.
>>>
>>> Ultimately, it's really up to you to come up with the right way for a
>>> client to request a token representing a particular running simulation, and
>>> then be able to connect back to that simulation later, probably by
>>> presenting the token to another RPC service.
>>>
>>> -Kenton
>>>
>>> On Fri, Mar 19, 2021 at 10:10 AM pepij...@gmail.com <
>>> pepijnde...@gmail.com> wrote:
>>>
 I think I figured out the first two problems.
 Multiple inheritance worked out quite nicely, only downside is each
 simulator needs to be defined explicitly.
 I think I found the correct way to do threading, which seems to be to
 use kj::getCurrentThreadExecutor() to get a way to schedule callbacks
 on the eventloop thread.
 So I guess you'd pass a promise to the thread and then fulfill it using
 the executor.
 https://github.com/capnproto/capnproto/blob/master/kjdoc/tour.md#threads

Re: [capnproto] Re: CPU-bound RPC command & optional API

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Yes, Executor is a good way to communicate between threads. FWIW you might
also check out newCrossThreadPromiseAndFulfiller(), which was added very
recently. Sometimes it's a better fit than kj::Executor.

"Level 2" turns out to be something that can't really be built into
libcapnp itself because the design really depends on the execution
environment in which your servers run. A system for saving capabilities and
restoring them later needs to understand how to connect to -- and possibly
even start up -- the appropriate server and storage system. So, for
example, Sandstorm.io has implemented level 2 of the protocol in a way that
is appropriate for it, but there ends up being not much that libcapnp
itself ca ndo to help. You can take a look at persistent.capnp to see a
suggestion for how to structure a level 2 implementation, but it's not much
more than a suggestion.

Ultimately, it's really up to you to come up with the right way for a
client to request a token representing a particular running simulation, and
then be able to connect back to that simulation later, probably by
presenting the token to another RPC service.

-Kenton

On Fri, Mar 19, 2021 at 10:10 AM pepij...@gmail.com 
wrote:

> I think I figured out the first two problems.
> Multiple inheritance worked out quite nicely, only downside is each
> simulator needs to be defined explicitly.
> I think I found the correct way to do threading, which seems to be to use 
> kj::getCurrentThreadExecutor()
> to get a way to schedule callbacks on the eventloop thread.
> So I guess you'd pass a promise to the thread and then fulfill it using
> the executor.
> https://github.com/capnproto/capnproto/blob/master/kjdoc/tour.md#threads
> As with most of these things, they are quite puzzling at first but
> actually quite nice once you figure them out.
>
> So the only problem left is how to handle disconnects in extremely long
> running processes.
>
> Cheers,
> Pepijn
> On Thursday, 18 March 2021 at 12:18:50 UTC+1 pepij...@gmail.com wrote:
>
>> Hey,
>>
>> I'm designing an RPC server for simulators that can run several different
>> long-running commands, and I have a few design questions.
>>
>> My current design is as follows.
>> Upon loading files to simulate, the simulator returns a list of unions of
>> interfaces, representing all the simulation commands it supports.
>> Upon calling one of the commands, a reader interface is returned that
>> allows streaming simulation results.
>> This currently happens in-thread, so reading big chunks will block the
>> entire server.
>>
>> So first question is, is there a nicer way to represent a thing that can
>> implement a subset of functions? I could just define a mega interface and
>> not implement methods, but then the issue is how to discover which methods
>> the server implements. Or maybe the correct approach is to use multiple
>> inheritance from the smaller interfaces to concrete implementations?
>>
>> The next question is how to offload the simulation to a thread? I assumed
>> this would be a very common task but I can't find much in the docs. I found
>> KJ::Thread or something like that, but it's not clear to me how to tie that
>> into the event loop promise API.
>>
>> Final issue I'm thinking about, for *very* long running simulations, the
>> client disconnecting in the middle of simulation that takes days or weeks
>> becomes a real concern. This is basically level 2 of
>> https://capnproto.org/rpc.html#protocol-features but as far as I
>> understand C++ is only level 1. What would be a good way to go about things
>> here? If level 2 is just around the corner, I can just ignore the issue for
>> a while, but maybe I need to manually store simulator references outside
>> the connection and hand out tokens to it?
>>
>> Regards,
>> Pepijn
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/8fe36a3a-88dd-4761-9fa9-48ba90a1da0fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmwNSUnv91FaCJ%2BrHZmr-nK%2BGbccWh2SCRZTNRqJg9jGg%40mail.gmail.com.

Re: [capnproto] Wikipedia Draft Article

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Tue, Mar 16, 2021 at 2:26 PM Zach Lym  wrote:

> I might say: "As of March 2021, three-party handoff and capability
 equality -- two advanced-but-important features of CapTP -- are not yet
 supported by Cap'n Proto."

>>>
>>> That sounds a bit too massaged.  I think it should flatly state that the
>>> *reference* implementation doesn't implement the full standard.
>>>
>>
>> Fair enough.
>>
>> Maybe: "As of March 2021, Cap'n Proto's reference implementation does not
>> support all of the features that have been specified in its own protocol.
>> In particular, three-party handoff and capability equality -- identified in
>> the spec as level 3 and level 4, respectively -- are not yet supported."
>>
>
> I agree with you that an explanation of the protocols should precede this
> statement.  But if you are self conscious about Cap'n Proto not supporting
> levels 3 & 4 ... it *is* weird and it makes Cap'n Proto feel half baked
> 😟.
>

Eh? No, none of this bothers me in that way. I just thought it was weird to
refer to "level 2" without any explanation of what that means, so I was
trying to come up with ways to explain. If you want to just say "Eight
years after introduction, Cap'n Proto's reference implementation still
doesn't implement all of the protocol as specified." that's fine with me.

I'm not saying this because I want to be mean, I love Cap'n Proto and want
> to see it succeed!  But the reference implementation being in C++ means
> it's not suitable for medium-to-high assurance projects, it lacks features
> that really set it apart from other frameworks (zero-copy IPC, advanced
> OCap functionality, and built in encryption), and it's not available in as
> many languages as competitors.  I *really* don't like giving this type of
> feedback, but hopefully it's constructive.
>

Hmm, it seems like you're worried I'd find something hurtful here, but I
don't, and I'm not even sure what it is that would be hurtful.

I built Cap'n Proto to suit my needs. When there's something missing that I
need, I add it. If there are things missing that other people need, I
invite them to add it. But if they decide to use something else instead,
that doesn't bother me; I want them to use whatever works for them. I have
no particular need or desire for Cap'n Proto to be widely adopted, I only
need it to get the job done in my other projects. I'm way more interested
in adoption of Cloudflare Workers than Cap'n Proto -- the former puts money
in my pocket, the latter does not.

If you think it'd be useful to quote me on that to explain the state of the
project, feel free. :)

-Kenton

>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnL6hvHGN%3DXtKxQcWnLjf%2BFE-RGMPtep%2BFY9A%3Dghpgg8A%40mail.gmail.com.

Re: [capnproto] Wikipedia Draft Article

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Tue, Mar 16, 2021 at 12:13 AM Zach Lym  wrote:

> Is that last sentence CC0 licensed?
>

Sure, feel free to reuse anything I wrote verbatim. CC0 license, or public
domain, whichever you prefer.


> I might say: "As of March 2021, three-party handoff and capability
>> equality -- two advanced-but-important features of CapTP -- are not yet
>> supported by Cap'n Proto."
>>
>
> That sounds a bit too massaged.  I think it should flatly state that the
> *reference* implementation doesn't implement the full standard.
>

Fair enough.

Maybe: "As of March 2021, Cap'n Proto's reference implementation does not
support all of the features that have been specified in its own protocol.
In particular, three-party handoff and capability equality -- identified in
the spec as level 3 and level 4, respectively -- are not yet supported."

Google also uses a variety of other technologies, but I think it's accurate
> to say that Protobufs/gRPC is their "primary" RPC system.  I know you don't
> want to oversell your role in the company, but I don't think we should
> split hairs here.  What I'm trying to get across is that there is a massive
> technology company which adopted it internally and is *actively*
> contributing financial support to the project.  Does Cloudflare pay
> developers to hack on protobufs or Thrift?
>

I don't think you can consider these equivalent. Nearly every project and
engineer at Google uses Protobuf. If you try to choose something else
you'll get pushback. At Cloudflare, each team makes their own decisions
about what to use and doesn't receive any pressure to choose one tech over
the other. I think only three projects use Cap'n Proto, out of dozens.
There are probably more projects using Protobuf than Cap'n Proto (but I
haven't done a survey).

Cloudflare doesn't really pay anyone to hack on Cap'n Proto specifically.
They pay us to build Cloudflare Workers, and in doing that we add what we
need to Cap'n Proto (or, more commonly, KJ). If someone on a Protobuf-using
project decided they needed a change in Protobuf, they could go make that
change. Of course, employing the owner of the project makes it a lot easier
to get changes accepted, so there's that.

In any case, it's accurate to say Cloudflare is using and driving
development of Cap'n Proto, but it's not accurate to say that Cap'n Proto
is Cloudflare's "primary" internal RPC system.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmt%3Dr9-%2By4R6MRLm%2BncSk%2BH0YR9Z%2Byj5EnpDy3YbWti4g%40mail.gmail.com.

Re: [capnproto] Wikipedia Draft Article

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Zach,

Thanks for doing this!

A few nitpicks:

> the former maintainer of Google's popular Protocol Buffers RPC framework

Technically Protocol Buffers is a serialization framework, not RPC. gRPC
and Stubby are RPC frameworks built on Protobuf, but I never worked on
either of them.

> #@n marks order of values in the message.

Technically it marks the order in which fields were added to the message
definition, not literally the order in which they appear in the
serialization. The order in which fields were added to the definition is
important in order to ensure that fields don't change position as new
fields are added. However, newer fields can be placed before older fields,
if there happened to be space available from padding.

> as apposed text encoding used

Typo: opposed

> Cap'n Proto instead tries to represent values in wire-protocol as they
would be when written to disk or even in-memory as much as possible

This sentence implies that on-disk formats and in-memory formats are
typically similar, with network formats being different. In practice,
though, on-disk formats are typically closer to network protocols. Like
protocols, storage formats often need to be position-independent, forwards-
and backwards-compatible, language-independent, architecture-independent,
secure, etc.

I think what this sentence was trying to get at is that Cap'n Proto tries
to make the storage/network protocol appropriate as an in-memory format, so
that no translation step is needed when reading data into memory or writing
data out of memory.

> Unlike other binary serialization protocols such as XMI or Protocol
Buffers, Cap'n Proto considers fine-grained data validation at the RPC
level an anti-feature that limits a protocols ability to evolve. This was
informed by experiences at Google where simply changing a field from
mandatory to optional would cause complex operational failures

For the most part, Protobuf followed the same philosophy from the start.
The only exception was required vs. optional, which is a fairly minor bit
of validation. As you note, "required" was removed from the language in
proto3, making Protobuf and Cap'n Proto basically equivalent in this
regard. But, even before "required" was removed, I think they were pretty
similar.

I don't know anything about XMI.

> As of October 2020, the reference implementation only supports level 2

The level numbers were defined by Cap'n Proto itself, so it's a little
weird to reference "level 2" without defining what that means. I might say:
"As of March 2021, three-party handoff and capability equality -- two
advanced-but-important features of CapTP -- are not yet supported by Cap'n
Proto."

> After Sandstorm.io transitioned from a commercial to community focus

You can just say: "After Sandstorm failed commercially"

> Cloudflare, which subsequently adopted Cap'n Proto as their primary
internal RPC system

This isn't quite accurate.

Cloudflare used Cap'n Proto serialization long before I joined, but it also
uses many other technologies, including Protobuf/gRPC, JSON, etc., in other
places.

I joined and started the Cloudflare Workers project. Workers specifically
makes use of Cap'n Proto in its implementation. However, the rest of
Cloudflare still uses a variety of technologies.

> The primary speedup over Protocol Buffers comes from using arena-style
memory allocation.

I would argue the primary speedup comes from zero-copy
parsing/serialization, not from arena-style allocation. Protobuf actually
supports arena-style allocation, at least in the C++ implementation.

That said, in reality this all depends on the use case. Sometimes arena
allocation might be what makes the difference. Sometimes Protobuf is
actually faster. But often Cap'n Proto is faster because it doesn't need a
translation step.

-Kenton

On Fri, Mar 12, 2021 at 8:51 PM Zach Lym  wrote:

> I have a draft entry for Cap'n Proto
>  awaiting review
> on Wikipedia.  While Wikipedia's conflict-of-interest policy
> 
>  appears
> to prevent contributors to Cap'n Proto
> 
>  from
> making edits directly, just put it in the talk page and tag it with
> `{{Request edit}}` : D
>
> https://en.wikipedia.org/wiki/Draft:Cap%E2%80%99n_Proto
>
> -Zach Lym
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/2fac5f42-8d56-4e06-af47-7e55c6aba890n%40googlegroups.com
> 
> .
>

-- 
You received th

Re: [capnproto] How to write to a file from an RPC server?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hmm when I say "async I/O" I mean the same thing as "non-blocking I/O".

I haven't worked in C#, but have done some stuff with the Win32 C API,
where they call it "overlapped" I/O.

-Kenton

On Wed, Mar 3, 2021 at 3:54 PM John Demme  wrote:

> I think in Windows, async IO is different from non-blocking IO. I haven't
> used Windows async IO, other than briefly trying to use C# async IO,
> getting very confused, and (perhaps prematurely) that C# async is intended
> to be async everywhere (without some really nasty hacks) and you can only
> use async IO in async code.
>
> ~John
>
> On Wed, Mar 3, 2021 at 1:42 PM Kenton Varda  wrote:
>
>> On Wed, Mar 3, 2021 at 2:11 PM John Demme  wrote:
>>
>>> Some operating systems (*cough* Windows *cough*) don't support
>>> non-blocking file (or inter-process pipe) I/O without some unreliable
>>> hacks. AFAICT.
>>>
>>
>> Hate to say it, but my understanding is that Windows has much more robust
>> async filesystem support than Linux does. The comment that Pepijn found is
>> about Linux, not Windows.
>>
>> Linux filesystem implementations are not async (at least for metadata /
>> directory walking), therefore the only way to access them in an async way
>> is to use threads -- either in the kernel, or in userspace. The kernel
>> isn't very excited about spawning kernel threads transparently -- it would
>> rather than you create userspace threads and make blocking calls.
>>
>> That said, this is changing a bit with io_uring, where kernel threads
>> running asynchronously from userspace threads are becoming more of a thing.
>> Still, on the kernel side, there are threads, even if it looks like "async"
>> I/O from the userspace side.
>>
>> Disclaimer: I'm not a kernel developer, this is my second-hand
>> understanding. Also, I know almost nothing about Windows, except that async
>> file I/O has featured much more prominently in the Win32 API going back
>> decades, so I'm guessing it's also baked deeper into the kernel.
>>
>> -Kenton
>>
>>
>>>
>>> ~John
>>>
>>> On Wed, Mar 3, 2021 at 11:02 AM pepijn de vos 
>>> wrote:
>>>
 Oh I just went hunting for the asynch bits and in the header it
 actually say there is no such thing as async file IO

 //
 ===
 // The filesystem API
 //
 // This API is strictly synchronous because, unfortunately, there's no
 such thing as asynchronous
 // filesystem access in practice. The filesystem drivers on Linux are
 written to assume they can
 // block. The AIO API is only actually asynchronous for
 reading/writing the raw file blocks, but if
 // the filesystem needs to be involved (to allocate blocks, update
 metadata, etc.) that will block.
 // It's best to imagine that the filesystem is just another tier of
 memory that happens to be
 // slower than RAM (which is slower than L3 cache, which is slower
 than L2, which is slower than
 // L1). You can't do asynchronous RAM access so why asynchronous
 filesystem? The only way to
 // parallelize these is using threads.
 //
 // All KJ filesystem objects are thread-safe, and so all methods are
 marked "const" (even write
 // methods). Of course, if you concurrently write the same bytes of a
 file from multiple threads,
 // it's unspecified which write will "win".

 On Wed, Mar 3, 2021 at 7:57 PM pepijn de vos 
 wrote:

> Hey Kenton,
>
> Thanks for the answer, and sorry my frustration in trying to find out
> how to do async file IO in cap'n proto came across rude.
>
> What puzzles me is your suggestion that I don't need to use KJ.
> What puzzles me even more is that the KJ file I just obtained seems to
> not support async operations at all.
> Are you suggesting it's actually fine to just do blocking IO in the
> RPC eventloop?
> Is there some KJ thing to do async file IO, or a way to use other
> libraries for async file IO with the KJ eventloop?
>
> Thanks to all the people working on cap'n proto, it seems pretty neat
> :)
>
> Cheers,
> Pepijn
>
> On Wed, Mar 3, 2021 at 4:42 PM Kenton Varda 
> wrote:
>
>> Hi Pepijn,
>>
>> The full comment you refer to says:
>>
>> // DO NOT CALL THIS except at the top level of your program, e.g. in
>> main(). Anywhere else, you
>> // should instead have your caller pass in a Filesystem object, or a
>> specific Directory object,
>> // or whatever it is that your code needs. This ensures that your
>> code supports dependency
>> // injection, which makes it more reusable and testable.
>>
>> As the comment says, the idea is that you construct your Filesystem
>> object at the top level of your program, then you pass it (or specific 
>> File
>> or Directory objects) down to whatever parts of your program need it. 
>> That

Re: [capnproto] How to write to a file from an RPC server?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Wed, Mar 3, 2021 at 2:11 PM John Demme  wrote:

> Some operating systems (*cough* Windows *cough*) don't support
> non-blocking file (or inter-process pipe) I/O without some unreliable
> hacks. AFAICT.
>

Hate to say it, but my understanding is that Windows has much more robust
async filesystem support than Linux does. The comment that Pepijn found is
about Linux, not Windows.

Linux filesystem implementations are not async (at least for metadata /
directory walking), therefore the only way to access them in an async way
is to use threads -- either in the kernel, or in userspace. The kernel
isn't very excited about spawning kernel threads transparently -- it would
rather than you create userspace threads and make blocking calls.

That said, this is changing a bit with io_uring, where kernel threads
running asynchronously from userspace threads are becoming more of a thing.
Still, on the kernel side, there are threads, even if it looks like "async"
I/O from the userspace side.

Disclaimer: I'm not a kernel developer, this is my second-hand
understanding. Also, I know almost nothing about Windows, except that async
file I/O has featured much more prominently in the Win32 API going back
decades, so I'm guessing it's also baked deeper into the kernel.

-Kenton


>
> ~John
>
> On Wed, Mar 3, 2021 at 11:02 AM pepijn de vos 
> wrote:
>
>> Oh I just went hunting for the asynch bits and in the header it actually
>> say there is no such thing as async file IO
>>
>> //
>> ===
>> // The filesystem API
>> //
>> // This API is strictly synchronous because, unfortunately, there's no
>> such thing as asynchronous
>> // filesystem access in practice. The filesystem drivers on Linux are
>> written to assume they can
>> // block. The AIO API is only actually asynchronous for reading/writing
>> the raw file blocks, but if
>> // the filesystem needs to be involved (to allocate blocks, update
>> metadata, etc.) that will block.
>> // It's best to imagine that the filesystem is just another tier of
>> memory that happens to be
>> // slower than RAM (which is slower than L3 cache, which is slower than
>> L2, which is slower than
>> // L1). You can't do asynchronous RAM access so why asynchronous
>> filesystem? The only way to
>> // parallelize these is using threads.
>> //
>> // All KJ filesystem objects are thread-safe, and so all methods are
>> marked "const" (even write
>> // methods). Of course, if you concurrently write the same bytes of a
>> file from multiple threads,
>> // it's unspecified which write will "win".
>>
>> On Wed, Mar 3, 2021 at 7:57 PM pepijn de vos 
>> wrote:
>>
>>> Hey Kenton,
>>>
>>> Thanks for the answer, and sorry my frustration in trying to find out
>>> how to do async file IO in cap'n proto came across rude.
>>>
>>> What puzzles me is your suggestion that I don't need to use KJ.
>>> What puzzles me even more is that the KJ file I just obtained seems to
>>> not support async operations at all.
>>> Are you suggesting it's actually fine to just do blocking IO in the RPC
>>> eventloop?
>>> Is there some KJ thing to do async file IO, or a way to use other
>>> libraries for async file IO with the KJ eventloop?
>>>
>>> Thanks to all the people working on cap'n proto, it seems pretty neat :)
>>>
>>> Cheers,
>>> Pepijn
>>>
>>> On Wed, Mar 3, 2021 at 4:42 PM Kenton Varda 
>>> wrote:
>>>
 Hi Pepijn,

 The full comment you refer to says:

 // DO NOT CALL THIS except at the top level of your program, e.g. in
 main(). Anywhere else, you
 // should instead have your caller pass in a Filesystem object, or a
 specific Directory object,
 // or whatever it is that your code needs. This ensures that your code
 supports dependency
 // injection, which makes it more reusable and testable.

 As the comment says, the idea is that you construct your Filesystem
 object at the top level of your program, then you pass it (or specific File
 or Directory objects) down to whatever parts of your program need it. That
 ensures that those parts of your program can easily be unit-tested against
 a fake in-memory filesystem, by swapping out the objects that you pass
 down. This is trying to help you make your code more flexible, but you can
 ignore it if you want.

 You also don't have to use the KJ filesystem API. In fact, nothing else
 in KJ or Cap'n Proto cares what you use to open files. KJ is a toolkit,
 meaning it provides a bunch of tools, but doesn't force you to use them if
 you don't want to.

 KJ documentation can be found here:
 https://github.com/capnproto/capnproto/tree/master/kjdoc

 Please note that Cap'n Proto and KJ are open source software provided
 for free. I'd like them to be useful but I don't get any actual benefit
 from you using them, and I'm not interested in helping people who are rude.
 So, if you

Re: [capnproto] How to write to a file from an RPC server?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Pepijn,

The full comment you refer to says:

// DO NOT CALL THIS except at the top level of your program, e.g. in
main(). Anywhere else, you
// should instead have your caller pass in a Filesystem object, or a
specific Directory object,
// or whatever it is that your code needs. This ensures that your code
supports dependency
// injection, which makes it more reusable and testable.

As the comment says, the idea is that you construct your Filesystem object
at the top level of your program, then you pass it (or specific File or
Directory objects) down to whatever parts of your program need it. That
ensures that those parts of your program can easily be unit-tested against
a fake in-memory filesystem, by swapping out the objects that you pass
down. This is trying to help you make your code more flexible, but you can
ignore it if you want.

You also don't have to use the KJ filesystem API. In fact, nothing else in
KJ or Cap'n Proto cares what you use to open files. KJ is a toolkit,
meaning it provides a bunch of tools, but doesn't force you to use them if
you don't want to.

KJ documentation can be found here:
https://github.com/capnproto/capnproto/tree/master/kjdoc

Please note that Cap'n Proto and KJ are open source software provided for
free. I'd like them to be useful but I don't get any actual benefit from
you using them, and I'm not interested in helping people who are rude. So,
if you have further questions, please try to tone it down a bit. Thanks.

-Kenton

On Wed, Mar 3, 2021 at 5:54 AM pepij...@gmail.com 
wrote:

> Hey,
>
> I'm just getting started and pretty confused.
> The RPC server is all async and stuff, so no blocking IO in there.
> Then there are some vague mentions that you should use KJ for most things.
> But KJ documentation is nowhere to be found...
>
> All I want to do is just open a file and write to it.
> After resorting to reading the source code, I found
>
> https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/kj/filesystem.h
> So I managed to create a path so far.
> Then I started to look for how to open a file.
> I found some methods on Directory.
> So then I started to look how to make a directory.
> Then I found a Filesystem, which is abstract.
> Then I found newDiskFilesystem which tells me
> DO NOT CALL THIS except in main()
> Well  how am I supposed to open a file inside my server handler then??
>
> Pepijn
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn_%3Dd05PVOYzWFpAKq_31PKhw%3Dm8mZSJEKriVXAP%3Duffw%40mail.gmail.com.

Re: [capnproto] Re: Premature EOF error with List of List of struct implementation under VS2017/x64

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Yojiro,

It would be best if you could reduce the problem to a minimal
self-contained test case. If your program is multiple files then it is
probably too big for us to look at. All I really need is for you to add a
main() function to the code you sent before, so I can see exactly the code
that demonstrates the problem.

-Kenton

On Sun, Feb 21, 2021 at 9:07 AM yojiro@gmail.com <
yojiro.nobuk...@gmail.com> wrote:

> Hi Kenton
>
> I prepared my source for sending to you.
> Can I send it to your email address? or reply to this group with it as
> attachment?
>
> Thank you.
>
>
> 2021年2月21日日曜日 20:09:41 UTC+9 yojiro@gmail.com:
>
>> Hi  Kenton,
>>
>> Glad to see your reply.
>>
>> Yes I am aware of open mode. I do, for writing,
>> return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_CREAT | _O_RDWR |
>> _O_TRUNC), _SH_DENYNO, (_S_IREAD | _S_IWRITE));
>> and for reading,
>> return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_RDWR), (_SH_DENYNO),
>> (_S_IREAD | _S_IWRITE));
>>
>> Using _O_WRONLY for writing, _O_RDONLY for reading did not change the
>> situation.
>> For shared flag,  I have tried SH_DENYNO or SH_DENYRW but made no
>> difference, as I had expected.
>>
>> I can provide my code. I will send it to you when its ready, have to
>> clean it a little I guess.
>>
>> Thank you very much for the message.
>>
>> 2021年2月20日土曜日 1:55:08 UTC+9 ken...@cloudflare.com:
>>
>>> Hi,
>>>
>>> Sorry for the slow reply.
>>>
>>> Just going to throw out a guess here, but seeing that you're using
>>> windows, did you make sure to use O_BINARY when opening the file
>>> descriptor? If you didn't, then the C library will silently corrupt your
>>> data by converting line endings.
>>>
>>> If that's not it, then could you provide a complete self-contained
>>> program (with main() function and all parameters filled in) that
>>> demonstrates the problem?
>>>
>>> -Kenton
>>>
>>> On Tue, Feb 16, 2021 at 7:15 AM yojiro@gmail.com <
>>> yojiro@gmail.com> wrote:
>>>
 Hi

 I could do a little more things.

 Interestingly, capnp.exe built under x64 can do decode message.
 I bet it does different processing than MessageReader's do and simply
 follow message format structure.

 I made a test code using MapViewOfFile  and FlatArrayMessageReader,.
 It can read messages that I couldnt by using StreamFdMessageReader.
 But easily fails with a little large data, say 32 entries in first
 level List and 1 entry in the second level List,
 getting following exception:

 Exception: capnp\serialize.c++:51: failed: expected array.size() >=
 offset + segmentSize; Message ends prematurely in first segment.
 stack: 7ff6a2af4b35 7ff6a2af3577 7ff6a2af14b6 7ff6a2b02323 7ffc1aa97c23
 7ffc1b18d720

 So I think I'm stuck. Any advice on how to proceed will be greatly
 appreciated.

 Thank you in advance.


 2021年2月16日火曜日 16:23:38 UTC+9 yojiro@gmail.com:

> Hi
>
> So, trying to understand what is happening.
>
> using capnp.exe (32bit) in capnproto-tools-win32-0.8.0, message can be
> read correctly.
> capnp.exe decode sparse4d.capnp Dir4D < myoutput.bin
> By looking at hex dump of the message file, it looks like a proper
> message format.
>
> So, writing new message is okay but reading it is not working properly.
> Using capnproto-0.8.0 built with VS2017/x64 , and create a new
> message, save it to a file, and then reading it by my program fails.
> Since all happens when I do message.getRoot(),  this "Premature
> EOF" situation happens inside capnp library.
>
> Is there any way to fix this? or to narrow down what is wrong?
>
> Thank you.
>
> 2021年2月16日火曜日 4:56:44 UTC+9 yojiro@gmail.com:
>
>> Hello
>>
>> I have been learning this wonderful software.
>> So I have been looking for an opportunity to use capnproto for my
>> project,
>> and now I have it but I need to use it with MSVC 2017(recently
>> updated to latest version 15.9.33), x64.
>> I only need serialization and no need for RPC or async.
>> This is my very first time posting a message to this community,
>> please let me know if I'm not doing right here.
>>
>> I have a schema whose basic construct is List of List of struct ( x,y
>> points in 4d expressed by two level lists). Second level will have a few
>> hundreds of entries and first level will have lots of entries.
>> And I am now stuck with a issue. If I put many entries in either of
>> lists (more than a few hundreds) and write that to a file, then when I 
>> read
>> the message, I get exception like this:
>> Any help would be very appreciated.
>>
>> printSparseDir4D: std::exception: kj\io.c++:53: failed: expected n >=
>> minBytes; Premature EOF
>> stack: 7ff672faac46 7ff672fa3fd6 7ff672fa2de5 7ff672fa1308
>> 7ff672fb15d3 7ffc1aa97c23 7ffc1b18d720
>>

Re: [capnproto] Re: Premature EOF error with List of List of struct implementation under VS2017/x64

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi,

Sorry for the slow reply.

Just going to throw out a guess here, but seeing that you're using windows,
did you make sure to use O_BINARY when opening the file descriptor? If you
didn't, then the C library will silently corrupt your data by converting
line endings.

If that's not it, then could you provide a complete self-contained program
(with main() function and all parameters filled in) that demonstrates the
problem?

-Kenton

On Tue, Feb 16, 2021 at 7:15 AM yojiro@gmail.com <
yojiro.nobuk...@gmail.com> wrote:

> Hi
>
> I could do a little more things.
>
> Interestingly, capnp.exe built under x64 can do decode message.
> I bet it does different processing than MessageReader's do and simply
> follow message format structure.
>
> I made a test code using MapViewOfFile  and FlatArrayMessageReader,.
> It can read messages that I couldnt by using StreamFdMessageReader.
> But easily fails with a little large data, say 32 entries in first level
> List and 1 entry in the second level List,
> getting following exception:
>
> Exception: capnp\serialize.c++:51: failed: expected array.size() >= offset
> + segmentSize; Message ends prematurely in first segment.
> stack: 7ff6a2af4b35 7ff6a2af3577 7ff6a2af14b6 7ff6a2b02323 7ffc1aa97c23
> 7ffc1b18d720
>
> So I think I'm stuck. Any advice on how to proceed will be greatly
> appreciated.
>
> Thank you in advance.
>
>
> 2021年2月16日火曜日 16:23:38 UTC+9 yojiro@gmail.com:
>
>> Hi
>>
>> So, trying to understand what is happening.
>>
>> using capnp.exe (32bit) in capnproto-tools-win32-0.8.0, message can be
>> read correctly.
>> capnp.exe decode sparse4d.capnp Dir4D < myoutput.bin
>> By looking at hex dump of the message file, it looks like a proper
>> message format.
>>
>> So, writing new message is okay but reading it is not working properly.
>> Using capnproto-0.8.0 built with VS2017/x64 , and create a new message,
>> save it to a file, and then reading it by my program fails.
>> Since all happens when I do message.getRoot(),  this "Premature
>> EOF" situation happens inside capnp library.
>>
>> Is there any way to fix this? or to narrow down what is wrong?
>>
>> Thank you.
>>
>> 2021年2月16日火曜日 4:56:44 UTC+9 yojiro@gmail.com:
>>
>>> Hello
>>>
>>> I have been learning this wonderful software.
>>> So I have been looking for an opportunity to use capnproto for my
>>> project,
>>> and now I have it but I need to use it with MSVC 2017(recently updated
>>> to latest version 15.9.33), x64.
>>> I only need serialization and no need for RPC or async.
>>> This is my very first time posting a message to this community, please
>>> let me know if I'm not doing right here.
>>>
>>> I have a schema whose basic construct is List of List of struct ( x,y
>>> points in 4d expressed by two level lists). Second level will have a few
>>> hundreds of entries and first level will have lots of entries.
>>> And I am now stuck with a issue. If I put many entries in either of
>>> lists (more than a few hundreds) and write that to a file, then when I read
>>> the message, I get exception like this:
>>> Any help would be very appreciated.
>>>
>>> printSparseDir4D: std::exception: kj\io.c++:53: failed: expected n >=
>>> minBytes; Premature EOF
>>> stack: 7ff672faac46 7ff672fa3fd6 7ff672fa2de5 7ff672fa1308 7ff672fb15d3
>>> 7ffc1aa97c23 7ffc1b18d720
>>>
>>> Now I found that writing 4 entries in first level, two entries in second
>>> level (two x,y points each in 4 frames), I get the same error.
>>>
>>> I'm putting my schema and my read/write code below. I'm using these
>>> codes with capnproto0.8.0. But actually this schema is a little striped
>>> version than the one I want to use. When I used that a little more complex
>>> schema, I faced similar issue with capnproto 0.7.0 and 0.8.0.
>>>
>>> I built capnproto following the steps in the website, had to skip
>>> heavy-tests project and kj-async (0.8.0), hoping the build was ok.
>>>
>>> Is there anything missing or done wrong?
>>> It would be very much appreciated for any help.
>>> Thank you.
>>>
>>>
>>> sparse4d.capnp :
>>>
>>> # unique file ID, generated by `capnp id`
>>> @0xae7dda29a38d355f;
>>>
>>> struct Sparse4DHeader {
>>> tSize @0 :Int32;
>>> zSize @1 :Int32;
>>> ySize @2 :Int32;
>>> xSize @3 :Int32;
>>> }
>>>
>>> struct Dir4D {
>>> header @0 :Sparse4DHeader;
>>> planes @1 :List(SparsePlane);
>>>
>>> struct SparsePlane {
>>> t @0 :UInt16;
>>> z @1 :UInt16;
>>> points @2 :List(Point);
>>> }
>>> }
>>>
>>> struct Point {
>>> x @0 :UInt16;
>>> y @1 :UInt16;
>>> intensity @2 :Float32;
>>> }
>>>
>>>
>>> And read/write code:
>>>
>>> /**
>>>  * Copyright (c) 2021 ASTOM R&D
>>>  * Author: Yojiro Nobukuni
>>>  */
>>> //#include "pch.h"
>>>
>>> #include "../sparse4d.capnp.h"
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>>
>>> #include 
>>> #include 
>>> #include 
>>> //#include 
>>>
>>>  /**
>>>   * g++ writer.cpp photodeco_kernel.capnp.c++ -o writer -lcapnp -lkj
>>>   *

Re: [capnproto] Mixing event loops

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Owen,

KJ is actually designed to allow its event loop to sit on top of some other
event loop, for this very reason. But, it's not easy. You have to implement
a custom subclass of `kj::EventPort` and probably all of the interfaces
(that you care about) in `kj/async-io.h`.

The node-capnp implementation contains an example of this, placing the KJ
event loop on top of libuv. See the first part of this file:

https://github.com/capnproto/node-capnp/blob/master/src/node-capnp/capnp.cc

I've often thought it would be cool if the KJ codebase itself contained
adaptors for various popular async frameworks, but never had time to work
on that...

-Kenton

On Thu, Feb 11, 2021 at 1:29 AM Owen Healy  wrote:

> So, this started out as a practical problem, but now it's more curiosity.
>
> Cap'n Proto uses an event loop for concurrency. I'm using another library
> that has its *own* event loop for concurrency. They refuse to use each
> other's event loops. Although the basic logic is the same between the two,
> the APIs are incompatible.
>
> So, how do I mix two event loops?
>
> My thought was, that if one of the event loops supports a callback when an
> event is queued I can essentially use one event loop to drive the other.
>
> The basic logic would be something like the following (Python syntax, but
> you get the idea):
>
> loop1 = cnp_event_loop()
> loop2 = some_other_lib_event_loop()
>
> @loop1.on_event_added
> def _():
> @loop2.call_soon
> def _():
> loop1.poll()
>
> def adapt_future(f):
> f2 = loop2.create_future()
> if f.done():
> f2.set_result(f.result())
> else:
> @f.add_done_callback
> def _():
> f2.set_result(f.result())
> return f2
>
> async def main():
> res = await adapt_future(func_that_returns_a_cnp_promise())
> print(res)
>
> loop2.run_until_complete(main2())
>
> (The function Cap'n Proto doesn't support is on_event_added() on an
> EventLoop.)
>
> Just curious if anyone has any thoughts on this.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/7b46e93a-a42e-4f69-b5f5-f924a3b81550n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3Dw0%2BD5z_mr2NC03YVQbP%3DVCzGjFamSPhwva%2BgrQsPdFQ%40mail.gmail.com.

Re: [capnproto] Inter process communication used in capnp

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Sorry, I really don't know since I haven't tried it yet. There's a lot of
potential to eliminate copies and syscalls here, but how much impact that
really makes would of course depend on what else your system is doing.

(Note that I go by "Kenton", not "Ken"...)

-Kenton

On Fri, Feb 5, 2021 at 4:39 PM Taylor Dawson  wrote:

> @Ken how much latency can you shave off with shared memory vs unix
> sockets? Cc @Justin if you have any comparisons?
>
> Thanks!
>
> On Wednesday, January 23, 2019 at 1:19:13 PM UTC-8 ken...@cloudflare.com
> wrote:
>
>> I think the way I'd do shared memory Cap'n Proto would be to have a large
>> shared memory space mapped upfront, allocate/write each message within the
>> space, and then signal to the other process "you can find a new message
>> located at position X", "I am done with the message located at position Y",
>> etc.
>>
>> This has always been something I intended to implement, but so far it
>> hasn't come up as a priority. Sending messages over a unix socket is easy
>> and works well, so I'd recommend trying that first, and considering shared
>> memory transport as a possible optimization later on.
>>
>> -Kenton
>>
>> On Sun, Jan 20, 2019 at 11:23 PM Omega Ishendra 
>> wrote:
>>
>>> Hi all,
>>>
>>> I know that communication between two processes can be done mainly using
>>> two methods.
>>>
>>>1. Shared Memory
>>>2. Message passing
>>>
>>>
>>> my initial idea about capnp is, it is using "Message Passing".
>>>
>>> Is my idea correct?
>>> Can we do Inter-Process-Communication using shared memory in capnp?
>>> which is the faster?
>>>
>>> Thanks,
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/capnproto.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/7c4811b6-1468-419a-81e4-4610cd818818n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnnBAddUuiP%3DOHYK7AvQt9yzQuyTAdwfrJLi5mX1xDrpg%40mail.gmail.com.

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Ah, cool.

FWIW, this is still just an issue with the design of this particular test
case. Having socket buffer set large should be no problem in general. I
suppose this test case should probably be rethought but I'll probably wait
for a second complaint before I mess with it.

-Kenton

On Mon, Feb 1, 2021 at 6:58 PM John Wang  wrote:

> Ah, yep that was it. I had net.core.rmem_default and net.core.wmem_default
> set to a rather high value for another application. Reverting them to
> normal fixed the test. Thanks!
>
> John Wang
> Lead Robotics Engineer
> May Mobility
>
>
> On Mon, Feb 1, 2021 at 7:32 PM Kenton Varda  wrote:
>
>> Hmm, well I'm not sure what to say here. Maybe your unix sockets have
>> larger buffers for some reason? Are you running a weird kernel version?
>> What architecture is this?
>>
>> In any case, this error doesn't indicate a problem with Cap'n Proto, only
>> a problem with the test, so you should be safe ignoring it.
>>
>> -Kenton
>>
>> On Mon, Feb 1, 2021 at 5:37 PM John Wang 
>> wrote:
>>
>>> My system also returns 1024.
>>>
>>> John Wang
>>> Lead Robotics Engineer
>>> May Mobility
>>>
>>>
>>> On Mon, Feb 1, 2021 at 5:43 PM Kenton Varda 
>>> wrote:
>>>
 It looks like this error occurs when the number of file descriptors
 sent on a unix socket exceeds `ulimit -n`.

 Admittedly, the test tries to send lots of file descriptors over a unix
 socket, stopping only when the socket buffer seems to be full. So depending
 on the system configuration, it could indeed cause this error.

 On my system, `ulimit -n` returns 1024, while the test manages to send
 278 file descriptors before filling the buffer.

 What does `ulimit -n` show on your system?

 -Kenton

 On Mon, Feb 1, 2021 at 4:14 PM John Wang 
 wrote:

> Ah sorry, missed that bit.
>>
>> kj/async-io-unix.c++:530: failed: ::sendmsg(fd, &msg, 0): Too many
>> references: cannot splice
>> stack: 7fa89b08ceb2 7fa89b07af3b 5bdb1d 7fa89ae084dd
>
>
> John Wang
> Lead Robotics Engineer
> May Mobility
>
>
> On Mon, Feb 1, 2021 at 5:12 PM Kenton Varda 
> wrote:
>
>> Hi John,
>>
>> Was there anything in the log before the fail line? Usually there
>> should be some sort of error message describing what went wrong, and 
>> maybe
>> a stack trace.
>>
>> -Kenton
>>
>> On Mon, Feb 1, 2021 at 3:53 PM John Wang 
>> wrote:
>>
>>> I'm seeing a failure on the following test. Did a quick search of
>>> the group and didn't see any mention of it:
>>>
>>> [ FAIL ] kj/async-io-test.c++:281: legacy test:
>>> AsyncIo/CapabilityPipeBlockedSendStream (52576 μs)
>>>
>>> This result is consistent across 3 runs of `make -j4 check`. My
>>> environment is Ubuntu 16.04 and gcc 5.4.0, and I'm building 
>>> `release-0.8.0`
>>> from git. Let me know if I can provide any other info.
>>>
>>> Best,
>>> John
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to capnproto+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/3820e497-c121-413c-9651-d3af19e068c3n%40googlegroups.com
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkFcuK7i3LTZsn_ZiEDveqQYYfmrypcPM6mxcqaueG6uw%40mail.gmail.com.

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hmm, well I'm not sure what to say here. Maybe your unix sockets have
larger buffers for some reason? Are you running a weird kernel version?
What architecture is this?

In any case, this error doesn't indicate a problem with Cap'n Proto, only a
problem with the test, so you should be safe ignoring it.

-Kenton

On Mon, Feb 1, 2021 at 5:37 PM John Wang  wrote:

> My system also returns 1024.
>
> John Wang
> Lead Robotics Engineer
> May Mobility
>
>
> On Mon, Feb 1, 2021 at 5:43 PM Kenton Varda  wrote:
>
>> It looks like this error occurs when the number of file descriptors sent
>> on a unix socket exceeds `ulimit -n`.
>>
>> Admittedly, the test tries to send lots of file descriptors over a unix
>> socket, stopping only when the socket buffer seems to be full. So depending
>> on the system configuration, it could indeed cause this error.
>>
>> On my system, `ulimit -n` returns 1024, while the test manages to send
>> 278 file descriptors before filling the buffer.
>>
>> What does `ulimit -n` show on your system?
>>
>> -Kenton
>>
>> On Mon, Feb 1, 2021 at 4:14 PM John Wang 
>> wrote:
>>
>>> Ah sorry, missed that bit.

 kj/async-io-unix.c++:530: failed: ::sendmsg(fd, &msg, 0): Too many
 references: cannot splice
 stack: 7fa89b08ceb2 7fa89b07af3b 5bdb1d 7fa89ae084dd
>>>
>>>
>>> John Wang
>>> Lead Robotics Engineer
>>> May Mobility
>>>
>>>
>>> On Mon, Feb 1, 2021 at 5:12 PM Kenton Varda 
>>> wrote:
>>>
 Hi John,

 Was there anything in the log before the fail line? Usually there
 should be some sort of error message describing what went wrong, and maybe
 a stack trace.

 -Kenton

 On Mon, Feb 1, 2021 at 3:53 PM John Wang 
 wrote:

> I'm seeing a failure on the following test. Did a quick search of the
> group and didn't see any mention of it:
>
> [ FAIL ] kj/async-io-test.c++:281: legacy test:
> AsyncIo/CapabilityPipeBlockedSendStream (52576 μs)
>
> This result is consistent across 3 runs of `make -j4 check`. My
> environment is Ubuntu 16.04 and gcc 5.4.0, and I'm building 
> `release-0.8.0`
> from git. Let me know if I can provide any other info.
>
> Best,
> John
>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/3820e497-c121-413c-9651-d3af19e068c3n%40googlegroups.com
> 
> .
>


-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmyKste%2Bm-JZbFBK4T4s9sXKdbwbT2S3D3h%3DCjLiM8-hQ%40mail.gmail.com.

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

It looks like this error occurs when the number of file descriptors sent on
a unix socket exceeds `ulimit -n`.

Admittedly, the test tries to send lots of file descriptors over a unix
socket, stopping only when the socket buffer seems to be full. So depending
on the system configuration, it could indeed cause this error.

On my system, `ulimit -n` returns 1024, while the test manages to send 278
file descriptors before filling the buffer.

What does `ulimit -n` show on your system?

-Kenton

On Mon, Feb 1, 2021 at 4:14 PM John Wang  wrote:

> Ah sorry, missed that bit.
>>
>> kj/async-io-unix.c++:530: failed: ::sendmsg(fd, &msg, 0): Too many
>> references: cannot splice
>> stack: 7fa89b08ceb2 7fa89b07af3b 5bdb1d 7fa89ae084dd
>
>
> John Wang
> Lead Robotics Engineer
> May Mobility
>
>
> On Mon, Feb 1, 2021 at 5:12 PM Kenton Varda  wrote:
>
>> Hi John,
>>
>> Was there anything in the log before the fail line? Usually there should
>> be some sort of error message describing what went wrong, and maybe a stack
>> trace.
>>
>> -Kenton
>>
>> On Mon, Feb 1, 2021 at 3:53 PM John Wang 
>> wrote:
>>
>>> I'm seeing a failure on the following test. Did a quick search of the
>>> group and didn't see any mention of it:
>>>
>>> [ FAIL ] kj/async-io-test.c++:281: legacy test:
>>> AsyncIo/CapabilityPipeBlockedSendStream (52576 μs)
>>>
>>> This result is consistent across 3 runs of `make -j4 check`. My
>>> environment is Ubuntu 16.04 and gcc 5.4.0, and I'm building `release-0.8.0`
>>> from git. Let me know if I can provide any other info.
>>>
>>> Best,
>>> John
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/3820e497-c121-413c-9651-d3af19e068c3n%40googlegroups.com
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm6d4_NcvUt5BOJG%2BKmjAn%2BPO5V2cmng1DBOu%3D5euw8%3Dg%40mail.gmail.com.

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi John,

Was there anything in the log before the fail line? Usually there should be
some sort of error message describing what went wrong, and maybe a stack
trace.

-Kenton

On Mon, Feb 1, 2021 at 3:53 PM John Wang  wrote:

> I'm seeing a failure on the following test. Did a quick search of the
> group and didn't see any mention of it:
>
> [ FAIL ] kj/async-io-test.c++:281: legacy test:
> AsyncIo/CapabilityPipeBlockedSendStream (52576 μs)
>
> This result is consistent across 3 runs of `make -j4 check`. My
> environment is Ubuntu 16.04 and gcc 5.4.0, and I'm building `release-0.8.0`
> from git. Let me know if I can provide any other info.
>
> Best,
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/3820e497-c121-413c-9651-d3af19e068c3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk2196Uido-fHEG%3DVJAZgwnjcwa09f9-cc9Vt3rgDXJTg%40mail.gmail.com.

Re: [capnproto] Capnproto RPC, processing of IPC calls

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

(Dupe message happened because one got stuck in the moderation queue.)

On Mon, Dec 7, 2020 at 10:20 AM Lena Lötter  wrote:

> Thanks for the quick response and direction. The example comes in handy as
> I opted for the longer route.
>
> On Wednesday, December 2, 2020 at 10:56:13 PM UTC+2 ken...@cloudflare.com
> wrote:
>
>> Hi Lena,
>>
>> `WaitScope` has a method called `poll()` which checks for new I/O and
>> handles it without blocking. Maybe you can get away with calling that from
>> time to time.
>>
>> If that doesn't work, then it sounds like you may need to implement your
>> own kj::EventPort to use in place of kj::UnixEventPort. A custom event
>> allows you to integrate with some other event loop. You will end up needing
>> to implement your own AsyncIoStream based on this. For an example of
>> something similar, you could look at how node-capnp implements a custom
>> event port that uses libuv's event loop (in order to integrate with Node);
>> see the first part of this file:
>>
>>
>> https://github.com/capnproto/node-capnp/blob/node10/src/node-capnp/capnp.cc
>>
>> Obviously, that's a fair amount of work, so hopefully the `poll()`
>> approach gets you what you want.
>>
>> -Kenton
>>
>> On Wed, Dec 2, 2020 at 8:48 AM Lena_work Work  wrote:
>>
>>> Hi
>>>
>>> I'm working on a project where rpc calls and other processing must occur
>>> in the same thread. That thread may only be interrupted for short periods.
>>> To achieve this I'm thinking about adding a file descriptor of the IPC to
>>> the poller in our code. When data is available on this file descriptor IPC
>>> calls will be processed until either there are none left or the allowed
>>> time is up.
>>>
>>> The class `EventLoop` has a function `run` that is described to be doing
>>> what seems useful in my case. For this reason it would be helpful to
>>> know how I can set up an rpc server with an eventloop that I can interact
>>> with directly?
>>> Any alternative suggestions would also be greatly appreciated.
>>>
>>> - Lena
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/53fb3f5f-65dc-4f26-9af4-bfc1e4ff206cn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/2f1f83ff-779f-47df-bc5f-278d574aebdfn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkSUYnVs2%3D7TXVbWNvMhbUyBteLo0V2Zt7e88ENLK1fsw%40mail.gmail.com.

Re: [capnproto] Capnproto RPC for Java - progress

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Awesome, I'm very glad to hear you've been paying attention to ordering --
it's tricky to get right but turns out to be very important.

I'm hoping David weighs in at some point and we can decide whether it makes
sense to merge this back into the main repo.

-Kenton

On Tue, Dec 1, 2020 at 7:23 AM Vaci  wrote:

> My main interest is that it's been hard to make a case for Capnproto in a
> multi-language development environment, when the majority language is Java.
> I must confess that I am mostly a C++ developer, and haven't done any
> serious Java work for years, and this looked like a nicely sized problem to
> re-acquaint myself with the language.
>
> No docs yet, but I have a bunch of RPC test cases, and the Calculator
> example could be added to the examples project. I'm at an impass currently,
> deciding whether to dump CompletableFutures altogether and replace with a
> home-grown futures library, which would change the API significantly. My
> implementation doesn't use the concurrent features of CompletableFutures
> (except for socket calls), so there's a bunch of unnecessary thread-safety
> operations going on, and also Capnproto is picky about the order in which
> events complete, whereas CompletableFutures very much take a YOLO approach
> to completion order. That made made, for example, QueuedClient tricky to
> get right and I'm still not entirely confident about it.
>
> Changes to the existing code have, fortunately, been fairly limited. The
> most intrusive change was to keep track of a cap-table context in builders,
> readers and arenas, but I think I've achieved that without affecting the
> existing API. AnyPointers have gained a method or two. Otherwise pretty
> much everything else is new and separate code - the RPC implementation
> itself is a separate library,  as it should be! On the compiler side, it's
> mostly been a case of rendering the interface types and adding pipelines,
> so existing code hasn't been touched outside of the previously
> unimplemented switch branches. That may be less true once generics enter
> the picture.
>
> The new code probably requires a recent (14+?) Java compiler version.
>
> cheers,
> Vaci
>
> On Monday, 30 November 2020 at 22:26:48 UTC ken...@cloudflare.com wrote:
>
>> This is great!
>>
>> Out of curiosity, do you have a specific use case you're building for?
>>
>> Do you have any RPC-specific documentation or examples yet?
>>
>> I notice this is implemented as a fork of the existing capnproto-java.
>> Did you have to make changes to the existing code? Is the API compatible?
>>
>> I agree with Ian that FD passing probably isn't going to be important to
>> Java users, so I wouldn't bother too much with it.
>>
>> +David Renshaw, have you looked at this at all?
>>
>> -Kenton
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/6e4ebe50-1dc5-450d-bf05-6df063fd1a6dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D7ra%2BCd0u-6vJYMg7_FGJrx%2BCxoyMOhjtKxgnS3LuY_g%40mail.gmail.com.

Re: [capnproto] Capnproto RPC, processing of IPC calls

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Lena,

`WaitScope` has a method called `poll()` which checks for new I/O and
handles it without blocking. Maybe you can get away with calling that from
time to time.

If that doesn't work, then it sounds like you may need to implement your
own kj::EventPort to use in place of kj::UnixEventPort. A custom event
allows you to integrate with some other event loop. You will end up needing
to implement your own AsyncIoStream based on this. For an example of
something similar, you could look at how node-capnp implements a custom
event port that uses libuv's event loop (in order to integrate with Node);
see the first part of this file:

https://github.com/capnproto/node-capnp/blob/node10/src/node-capnp/capnp.cc

Obviously, that's a fair amount of work, so hopefully the `poll()` approach
gets you what you want.

-Kenton

On Wed, Dec 2, 2020 at 8:48 AM Lena_work Work  wrote:

> Hi
>
> I'm working on a project where rpc calls and other processing must occur
> in the same thread. That thread may only be interrupted for short periods.
> To achieve this I'm thinking about adding a file descriptor of the IPC to
> the poller in our code. When data is available on this file descriptor IPC
> calls will be processed until either there are none left or the allowed
> time is up.
>
> The class `EventLoop` has a function `run` that is described to be doing
> what seems useful in my case. For this reason it would be helpful to know
> how I can set up an rpc server with an eventloop that I can interact with
> directly?
> Any alternative suggestions would also be greatly appreciated.
>
> - Lena
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/53fb3f5f-65dc-4f26-9af4-bfc1e4ff206cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D74cTP%2BRKTSTRhxq7xc-PXuhkGHtjuvKi12jw_uYT%3DNA%40mail.gmail.com.

Re: [capnproto] Load balancing and metrics

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Yes, it's straightforward to implement a load-balancing proxy server that
redirects each call to a different back-end. To do this in C++, you'd
implement capnp::Capability::Server, implementing the generic
`dispatchCall()` method such that it chooses a back-end and then uses
`context.tailCall()` to forward the call. You'd use that as your
"bootstrap" interface. There's no need for the proxy to know the schema of
this interface; as long as all the back-ends that it talks to use the same
bootstrap schema, any calls will flow through nicely.k

In fact, a proxy implemented this way would "just work" with calls that
return capabilities! Any returned capabilities would automatically become
"sticky", such that calls to them would forward to the specific back-end
that originally returned them. The proxy itself remembers this forwarding
on a per-connection basis; when the client disconnects, the forwarding
tables are discarded. All this logic is pretty much built into the current
implementation.

Heh, if I had more time I'd work on building out a bunch of these little
tools, like a load-balancing proxy, as part of the core project...

-Kenton

On Mon, Nov 30, 2020 at 6:32 PM Kevin Conway 
wrote:

> I appreciate the confirmation.
>
> > Why do you want a "stateless subset of capnproto"?
>
> It's less about implementing the RPC protocol and more about an outlet for
> using familiar deployment and operational practices to get decent work
> distribution among mundane, 12-factor style systems. For example, if I have
> an interface defined that offers CRUD style operations for one or a few
> different structures then I don't really need the stateful portions of the
> capnproto RPC protocol and could potentially benefit from avoiding them.
> Keeping within the "stateless subset" would make it much easier to
> deploy/operate that system using an auto-scaling policy and protocol aware
> reverse proxy to distribute the requests just like most stateless style
> HTTP services that sit behind a load balancer. I don't see a technical
> barrier to that protocol aware proxy acting similar to a membrane by
> ensuring that all interfaces returned from interfaces, recursively, are
> bound to the same underlying instance of the system which would provide
> some support for more stateful transactions or "sticky binding" where it
> makes sense and could provide support for having stateful and stateless
> support side-by-side. If I had a large set of interfaces that all stayed
> within the "stateless subset" then I could potentially build an adapter
> that converts RPC calls to HTTP calls using a standard generation of paths,
> headers, and content bodies (similar to what Twirp does for protobuf). I'd
> be losing basically all of the RPC advantages that capnproto has but I'd be
> able to then buy into a fairly extensive suite of HTTP related tools for
> operations, telemetry collection, access/audit logging, etc.
>
> To be clear, I'm learning capnproto because I have some pet projects that
> fit well into the expected use cases of stateful/distributed systems and
> IPC. The "stateless subset" line of thinking is trying to tease out if
> there's a reasonable path to replace all my current protobuf/gRPC use
> cases. It would be convenient to have only one design language and
> ecosystem to stay up to date on and contribute to. It would be a compelling
> point, to me, if I could say that capnproto + RPC covers all my typical
> cases and also has excellent optimizations for my complex or unique
> projects.
>
>
> On Mon, Nov 30, 2020 at 12:30 PM Kenton Varda 
> wrote:
>
>> Hi Kevin,
>>
>> You are correct that if you avoid passing interface types in your RPC
>> messages, you will never have the opportunity to take advantage of promise
>> pipelining, at least as the protocol is defined today.
>>
>> But I'm not sure I understand the motivation for your question. Why do
>> you want a "stateless subset of capnproto"? Is this about looking for an
>> easier first step to implementing Cap'n Proto RPC in more languages? If so,
>> I think what you're looking for is what we call "Level 0 support" in
>> rpc.capnp. A level 0 implementation does not support passing interface
>> references.
>>
>> -Kenton
>>
>> On Wed, Nov 25, 2020 at 11:14 PM Kevin Conway 
>> wrote:
>>
>>> I apologize for bumping an old thread. I'm curious if your
>>> recommendation has changed any since 2018.
>>>
>>> I have several projects that match the IPC and stateful use cases you
>>> mention that I'm going to try building on capnproto as a learning
>>> experience. I'm also interested, though, in an option to have only one
>>> service definition and schema language, one code generation toolset, etc.
>>> The only hesitance I have in replacing gRPC with capnproto for the
>>> stateless, kubernetes deployed, auto-scaled type of system is this old
>>> thread which seems to be one of the few that mentions this subject.
>>>
>>> My current understanding of how pipelining is implement is that i

Re: [capnproto] Capnproto RPC for Java - progress

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

This is great!

Out of curiosity, do you have a specific use case you're building for?

Do you have any RPC-specific documentation or examples yet?

I notice this is implemented as a fork of the existing capnproto-java. Did
you have to make changes to the existing code? Is the API compatible?

I agree with Ian that FD passing probably isn't going to be important to
Java users, so I wouldn't bother too much with it.

+David Renshaw , have you looked at this at all?

-Kenton

On Thu, Nov 26, 2020 at 10:54 AM Vaci  wrote:

> Hi,
>
> I now have a reasonably complete implementation of Capnproto RPC in pure
> Java.
>
> The major features still missing are:
>
>- file descriptor passing
>- generics
>
> The former is not likely to find support any time soon in Java's
> asynchronous socket library. The latter is hopefully 'just' extensions to
> the compiler.
>
> Streams are recognised, but just fall back to non-streaming
> implementations.
> Cancellation and disconnection still need some work.
>
> I have slight regrets about using Java's CompletableFuture framework -
> eager completion of promises without any guarantee of ordering made for
> some interesting debugging - but it is at least a "standard" API. I would
> like to find a better solution for running the client side message loop as
> just waiting for promises to complete isn't sufficient.
>
> I have yet do do any serious stress or performance testing; so far the
> implementation has been 'good enough' for my requirements.
>
> I suspect there are still plenty of bugs, so if anybody is interested in
> finding them, please check it out:
>
> https://github.com/vaci/capnproto-java-rpc
>
> cheers,
> Vaci
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/e7973e0b-109b-44d3-9e90-0a2753439be3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn7WEytm0BiXWJweKQOUEWUPXSS%2BMb34xrsnu6qmexraA%40mail.gmail.com.

Re: [capnproto] Load balancing and metrics

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Kevin,

You are correct that if you avoid passing interface types in your RPC
messages, you will never have the opportunity to take advantage of promise
pipelining, at least as the protocol is defined today.

But I'm not sure I understand the motivation for your question. Why do you
want a "stateless subset of capnproto"? Is this about looking for an easier
first step to implementing Cap'n Proto RPC in more languages? If so, I
think what you're looking for is what we call "Level 0 support" in
rpc.capnp. A level 0 implementation does not support passing interface
references.

-Kenton

On Wed, Nov 25, 2020 at 11:14 PM Kevin Conway 
wrote:

> I apologize for bumping an old thread. I'm curious if your recommendation
> has changed any since 2018.
>
> I have several projects that match the IPC and stateful use cases you
> mention that I'm going to try building on capnproto as a learning
> experience. I'm also interested, though, in an option to have only one
> service definition and schema language, one code generation toolset, etc.
> The only hesitance I have in replacing gRPC with capnproto for the
> stateless, kubernetes deployed, auto-scaled type of system is this old
> thread which seems to be one of the few that mentions this subject.
>
> My current understanding of how pipelining is implement is that it only
> works for returned interfaces. That is, an interface method defined as `foo
> @1 (bar :String) -> (baz :String)` would not present me the option of
> pipelining that value through subsequent operations. If I wanted to
> "opt-in" to pipelining support then I would need to convert the return
> value to an interface type like what is done in the calculator example of
> the rust RPC implementation. If that's the case then that means I can
> support less stateful interactions by only ever defining methods that
> return concrete types as a way of forcing the RPC implementation to avoid
> pipelining. This is equivalent to only defining unary methods in gRPC or
> using one of the streaming signatures to "opt-in" to a stateful
> interaction. I'm new to capnproto so this description is fishing for
> clarification or disagreement.
>
> Another way to talk about this might be to ask "what would be required to
> support a stateless subset of capnproto even if it meant losing many of the
> advantages of the RPC model?". If it comes down to "use a subset of the
> schema language (e.g. only define unary APIs) and implement a protocol
> aware reverse proxy" then that's a fairly accessible amount of work for a
> contributor to take on and document. After that, the other concerns like
> metrics gathering or integration of growing open standards is a matter of
> building code generation plugins. I'm curious to hear thoughts from folks
> with more capnproto experience.
>
> On Wednesday, September 5, 2018 at 2:47:13 PM UTC-5 ken...@cloudflare.com
> wrote:
>
>> Hi,
>>
>> While Cap'n Proto is certainly capable of replacing gRPC here, obviously
>> Google has a lot more people working on the gRPC ecosystem, and so more
>> infrastructure has been built out there. With Cap'n Proto you will have to
>> do more things yourself. If your needs fit well into the model supported by
>> common gRPC infrastructure, this may not be worth it to you. On the other
>> hand, if you have a more unusual use case, then you might find you have to
>> build custom solutions either way, in which case Cap'n Proto's more
>> fundamental benefits (serialization performance and object capabilities)
>> may make it a better choice.
>>
>> Cap'n Proto is especially powerful for:
>>
>> * State*ful* services, where nodes across a cluster need to control and
>> hold on to the state of other nodes. For example, Sandstorm.io (startup I
>> founded) built a cluster-scaleable version of Sandstorm called "Blackrock"
>> which is itself a container orchestrater designed to run many heterogeneous
>> container instances on behalf of individual end users. This is a
>> fundamentally stateful thing, since each container is serving a specific
>> user with specific state and can't simply be interchanged with others.
>> Using Cap'n Proto as the underlying communications protocol made this a lot
>> easier to manage.
>>
>> * IPC use cases, where services are running on the same machine and can
>> communicate via unix sockets or even shared memory. Cap'n Proto's CPU
>> performance shines here (while its somewhat higher bandwidth usage becomes
>> irrelevant). Running multiple services on the same machine isn't really in
>> style right now, but my current employer, Cloudflare, does a lot of it --
>> every one of the machines in our edge network runs an identical set of
>> services so that any one machine can handle any kind of request on its own.
>>
>> But if you're doing a standard kubernetes microservices thing... probably
>> going with gRPC is going to be a lot easier right now.
>>
>> -Kenton
>>
>> On Wed, Sep 5, 2018 at 7:18 AM, shinzui  wrote:
>>
>>> Hi,
>>>
>>> I just start

Re: [capnproto] Schema Refactoring and Unique IDs

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Wed, Nov 25, 2020 at 3:43 PM Matt Stern  wrote:

> When I try to run capnp compile, I get the following:
>
> error: Import failed: /capnp/java.capnp
>

You will need to specify the same -I flags (import path) that you normally
specify to `capnp compile` when running the Java code generator.

If I comment out the Java bits and just compile in C++ (which works for
> me), will this have any effect on the unique IDs for the structs in my
> schema file?
>

No, the auto-generated IDs do not in any way depend on the contents of
other files. Auto-generated IDs are constructed by concatenating the parent
scope ID and the type name, and then taking a hash of that.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DgWcj8%2BbKitcAn3jSqXYmKbpLsd_wZRRQ8jaPJym7yVQ%40mail.gmail.com.

Re: [capnproto] Schema Refactoring and Unique IDs

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Erin is correct.

For #2, the command-line syntax to have the compiler echo back capnp format
(with all type IDs defined explicitly) is:

capnp compile -ocapnp foo.capnp

This of course requires that `capnp` and the generator plugin
`capnpc-capnp` are in your $PATH, which they should be after installing
Cap'n Proto globally. If you haven't installed it globally, you can do:

path/to/capnp compile -opath/to/capnpc-capnp foo.capnp

In any case, the output will go to the terminal (no files are generated).

-Kenton

On Wed, Nov 25, 2020 at 1:52 PM Erin Shepherd  wrote:

> 1. This depends. The only places that the IDs get used "behind the scenes"
> are (a) those on interfaces are used in RPC calls to identify the interface
> and (b) when encoding schema annotations
>
> On the other hand, someone might be explicitly reading the ID from the
> schema file or the constant from the generated code
>
> 2. capnpc can be asked to generate capnp format output. In that case,
> it'll emit a copy of the schema with comments stripped and all
> automatically generated IDs inserted. You can grab the ID (and syntax) from
> there
>
> - Erin
>
> On Wed, 25 Nov 2020, at 20:39, Matt Stern wrote:
>
> Hi all,
>
> Suppose I have a simple schema file:
>
> 0xabbeabbeabbeabbe;
>
> struct Foo {
>   val @0 : UInt32;
> };
> struct Bar {
>   val @1 : UInt32;
> };
>
> I would like to move Bar into a separate schema file. If I understand the
> docs  correctly, then
> this will change Bar's unique ID.
>
> I have two questions about that:
>
>1. Will changing Bar's unique ID cause backwards incompatibility with
>old messages that are serialized with the old ID?
>2. If so, what can I do to prevent this? I would like my change to
>have no side-effects (a pure no-op).
>
> Thanks!
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/aa247efb-ca69-40fa-97a7-415792fd0c1dn%40googlegroups.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/a98c4968-c814-4c16-a7ed-a299b3d10688%40www.fastmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn6%3DwyXJfRrocTuxUWnkYLV92LknFgkw0qwS6_q8fn8ow%40mail.gmail.com.

Re: [capnproto] Build error in dockcross/linux-arm64 Docker

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Sundar,

Note that the step that is failing is when the build tries to invoke the
capnp binary that it just built -- so it would be invoking an aarch64
binary.

Does this environment use qemu-user to emulate aarch64? I've noticed in the
past that qemu-user has a lot of bugs emulating certain syscalls, including
openat() which capnp uses for all filesystem access. The error message is
pretty nonsensical (./src clearly exists), so my guess is this is a buggy
qemu situation.

To avoid qemu problems when cross-compiling, I recommend compiling and
installing cap'n proto for the host system first. Then, use `./configure
--with-external-capnp`. This will use the installed capnp binary instead of
using the one built in-tree.

If you don't want to install capnp on the host, you can use the environment
variables CAPNP and CAPNPC_CXX to point at the `capnp` and `capnpc-c++`
binaries located in some other location (together with
`--with-external-capnp`).

-Kenton

On Sun, Nov 15, 2020 at 11:18 PM Sundar Palani 
wrote:

> Hi Kenton - Thanks for the reply.
>
> Here are the steps I did:
>
> 1. cd /root
> 2. curl -O https://capnproto.org/capnproto-c++-0.8.0.tar.gz
> 3. cd capnproto-c++-0.8.0
> 4. ./configure (Also tried these options for configure, same
> behaviour: --host aarch64-linux-gnu --build x86_64-linux-gnu
> --enable-static --disable-shared LIBS="-lrt -lpthread -lgcc -lsupc++
> -lstdc++")
> 5. make -j4 check -- failed at this step.
>
> Attached the log file.
>
>
> Regards,
> Sundar
> _
> 
> *SundaraGanesh Palani*
> P   Please consider the environment before printing this e-mail or its
> attachments
>
>
> On Sun, Nov 15, 2020 at 7:26 PM Kenton Varda 
> wrote:
>
>> Hi Sundar,
>>
>> Can you tell us the set of commands that led to this? I.e. what was your
>> command-line for `configure` and `make`? And what directory did you run
>> them in, relative to the source directory?
>>
>> -Kenton
>>
>> On Sun, Nov 15, 2020 at 6:22 PM Sundar Palani 
>> wrote:
>>
>>> I have the following error while building inside dockcross/linux-arm64
>>> docker running on my macOS x86-64.
>>>
>>> /bin/bash ./libtool  --tag=CXX   --mode=link
>>> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-g++
>>> -I./src -I./src -DKJ_HEADER_WARNINGS -DCAPNP_HEADER_WARNINGS
>>> -DCAPNP_INCLUDE_DIR='"/usr/local/include"' -pthread -O2 -DNDEBUG -pthread
>>> -release 0.9-dev -no-undefined  -o libcapnp-json.la -rpath
>>> /usr/local/lib src/capnp/compat/json.lo src/capnp/compat/json.capnp.lo
>>> libcapnp.la libkj.la -lpthread -lpthread
>>> libtool: link:
>>> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-ar
>>> cru .libs/libcapnp-json.a  src/capnp/compat/json.o
>>> src/capnp/compat/json.capnp.o
>>> libtool: link: ranlib .libs/libcapnp-json.a
>>> libtool: link: ( cd ".libs" && rm -f "libcapnp-json.la" && ln -s "../
>>> libcapnp-json.la" "libcapnp-json.la" )
>>> /bin/bash ./libtool  --tag=CXX   --mode=link
>>> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-g++
>>> -I./src -I./src -DKJ_HEADER_WARNINGS -DCAPNP_HEADER_WARNINGS
>>> -DCAPNP_INCLUDE_DIR='"/usr/local/include"' -pthread -O2 -DNDEBUG -pthread
>>> -pthread  -o capnp src/capnp/compiler/module-loader.o
>>> src/capnp/compiler/capnp.o libcapnpc.la libcapnp-json.la libcapnp.la
>>> libkj.la -lpthread -lpthread
>>> libtool: link:
>>> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-g++
>>> -I./src -I./src -DKJ_HEADER_WARNINGS -DCAPNP_HEADER_WARNINGS
>>> -DCAPNP_INCLUDE_DIR=\"/usr/local/include\" -pthread -O2 -DNDEBUG -pthread
>>> -pthread -o capnp src/capnp/compiler/module-loader.o
>>> src/capnp/compiler/capnp.o  ./.libs/libcapnpc.a ./.libs/libcapnp-json.a
>>> /root/capnproto/c++/.libs/libcapnp.a ./.libs/libcapnp.a
>>> /root/capnproto/c++/.libs/libkj.a ./.libs/libkj.a -lpthread -pthread
>>> ./capnp compile --src-prefix=./src -o./capnpc-c++:src -I./src $(for FILE
>>> in src/capnp/test.capnp src/capnp/test-import.capnp
>>> src/capnp/test-import2.capnp src/capnp/compat/json-test.capnp; do echo
>>> ./$FILE; done)
>>> *./capnp compile: --src-prefix=./src: no such directory*
>>> Try './capnp compile --help' for more information.
>>> Makefile:3801: recipe for target 'test_capnpc_middleman' failed
>>> make: *** [test_capnpc_middleman] Error 1
>>>
>>> Has anyone seen this error ? Any thoughts on how to solve this ?
>>>
>>> PS: I was able to successfully compile this on my mac as well as aarch64
>>> native machine (Jetson Nano).
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/a58a4bc0-92d1-499b-b1e6-398f4b8d7e3fn%40googlegroups.com
>>> 

Re: [capnproto] Build error in dockcross/linux-arm64 Docker

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Sundar,

Can you tell us the set of commands that led to this? I.e. what was your
command-line for `configure` and `make`? And what directory did you run
them in, relative to the source directory?

-Kenton

On Sun, Nov 15, 2020 at 6:22 PM Sundar Palani 
wrote:

> I have the following error while building inside dockcross/linux-arm64
> docker running on my macOS x86-64.
>
> /bin/bash ./libtool  --tag=CXX   --mode=link
> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-g++
> -I./src -I./src -DKJ_HEADER_WARNINGS -DCAPNP_HEADER_WARNINGS
> -DCAPNP_INCLUDE_DIR='"/usr/local/include"' -pthread -O2 -DNDEBUG -pthread
> -release 0.9-dev -no-undefined  -o libcapnp-json.la -rpath /usr/local/lib
> src/capnp/compat/json.lo src/capnp/compat/json.capnp.lo libcapnp.la
> libkj.la -lpthread -lpthread
> libtool: link:
> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-ar
> cru .libs/libcapnp-json.a  src/capnp/compat/json.o
> src/capnp/compat/json.capnp.o
> libtool: link: ranlib .libs/libcapnp-json.a
> libtool: link: ( cd ".libs" && rm -f "libcapnp-json.la" && ln -s "../
> libcapnp-json.la" "libcapnp-json.la" )
> /bin/bash ./libtool  --tag=CXX   --mode=link
> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-g++
> -I./src -I./src -DKJ_HEADER_WARNINGS -DCAPNP_HEADER_WARNINGS
> -DCAPNP_INCLUDE_DIR='"/usr/local/include"' -pthread -O2 -DNDEBUG -pthread
> -pthread  -o capnp src/capnp/compiler/module-loader.o
> src/capnp/compiler/capnp.o libcapnpc.la libcapnp-json.la libcapnp.la
> libkj.la -lpthread -lpthread
> libtool: link:
> /usr/xcc/aarch64-unknown-linux-gnueabi/bin/aarch64-unknown-linux-gnueabi-g++
> -I./src -I./src -DKJ_HEADER_WARNINGS -DCAPNP_HEADER_WARNINGS
> -DCAPNP_INCLUDE_DIR=\"/usr/local/include\" -pthread -O2 -DNDEBUG -pthread
> -pthread -o capnp src/capnp/compiler/module-loader.o
> src/capnp/compiler/capnp.o  ./.libs/libcapnpc.a ./.libs/libcapnp-json.a
> /root/capnproto/c++/.libs/libcapnp.a ./.libs/libcapnp.a
> /root/capnproto/c++/.libs/libkj.a ./.libs/libkj.a -lpthread -pthread
> ./capnp compile --src-prefix=./src -o./capnpc-c++:src -I./src $(for FILE
> in src/capnp/test.capnp src/capnp/test-import.capnp
> src/capnp/test-import2.capnp src/capnp/compat/json-test.capnp; do echo
> ./$FILE; done)
> *./capnp compile: --src-prefix=./src: no such directory*
> Try './capnp compile --help' for more information.
> Makefile:3801: recipe for target 'test_capnpc_middleman' failed
> make: *** [test_capnpc_middleman] Error 1
>
> Has anyone seen this error ? Any thoughts on how to solve this ?
>
> PS: I was able to successfully compile this on my mac as well as aarch64
> native machine (Jetson Nano).
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/a58a4bc0-92d1-499b-b1e6-398f4b8d7e3fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmBECUTLsTHF69cb50-FgHXJ-uC1tkb8eUN5eQLJF6fSQ%40mail.gmail.com.

Re: [capnproto] Segfault in on Ubuntu 14.04 32-bit

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Yep, that looks like a smoking gun to me.

-Kenton

On Mon, Nov 9, 2020 at 2:41 PM Zachary Dremann  wrote:

> This looks related:
> https://lists.debian.org/debian-glibc/2014/01/msg00023.html
>
> I rebuilt everything in the same way on an ubuntu 16.04 VM, and I don't
> get the segfault. I don't need to use fibers, so I'm okay to just avoid
> them.
>
> On Friday, November 6, 2020 at 1:20:53 PM UTC-5 ken...@cloudflare.com
> wrote:
>
>> Hi Zachary,
>>
>> It looks like the segfault is happening in glibc's backtrace() function
>> while gathering the stack trace for an exception. An exception is actually
>> expected here (the test case tests throwing an exception). Since it
>> happened specifically in fiber-related tests, my guess is that glibc's
>> backtrace has a bug in which it is confused by the stack manipulation done
>> by fibers.
>>
>> 14.04 is quite old, so I imagine you're using a pretty old glibc here?
>> Maybe this is a glibc bug that was fixed in newer versions?
>>
>> Note that fibers are new and completely optional feature of KJ. If you
>> don't use them then you can probably ignore the test failure.
>>
>> -Kenton
>>
>> On Wed, Nov 4, 2020 at 4:27 PM Zachary Dremann  wrote:
>>
>>> I'm running on Ubuntu 14.04, 32 bit.
>>>
>>> I have a custom g++ 10.1.0 as g++-10 installed in /usr/local, built from
>>> source with no modifications or customizations
>>>
>>> I cloned from master (commit 96936a7446164424d8f3f50430d83ee21f63f6a2),
>>> and ran `autoreconf -fi`, then `./configure --without-openssl
>>> --with-pic LDFLAGS='-L/usr/local/lib -Wl,-rpath,/usr/local/lib'
>>> CXX=/usr/local/bin/g++-10`, followed by `make`, and `make check`.
>>>
>>> `VERBOSE=1 make check` fails with (truncated to show interesting stuff):
>>>
>>> [ TEST ] kj/async-test.c++:860: start a fiber
>>> [ PASS ] kj/async-test.c++:860: start a fiber (112 μs)
>>> [ TEST ] kj/async-test.c++:881: fiber promise chaining
>>> [ PASS ] kj/async-test.c++:881: fiber promise chaining (9 μs)
>>> [ TEST ] kj/async-test.c++:904: throw from a fiber
>>> /bin/bash: line 5: 22656 Segmentation fault  (core dumped) ${dir}$tst
>>> FAIL: capnp-test
>>> Randomly testing backwards-compatibility scenarios with seed: 1604520183
>>> PASS: capnp-evolution-test
>>> PASS: src/capnp/compiler/capnp-test.sh
>>> ===
>>> 1 of 3 tests failed
>>> Please report to capn...@googlegroups.com
>>> ===
>>>
>>> Running `libtool --mode=execute gdb ./capnp-test`, and getting a
>>> backtrace at the time of the segfault gives the following:
>>>
>>> #0  0xb793d71e in x86_fallback_frame_state (context=,
>>> context=, fs=0xb57565b0)
>>> at ./md-unwind-support.h:132
>>> #1  uw_frame_state_for () at ../../../gcc-10.1.0/libgcc/unwind-dw2.c:1271
>>> #2  0xb793ecb1 in _Unwind_Backtrace () at
>>> ../../../gcc-10.1.0/libgcc/unwind.inc:302
>>> #3  0xb7876d75 in __GI___backtrace (array=0xb5756720, size=34) at
>>> ../sysdeps/i386/backtrace.c:126
>>> #4  0xb7bb098a in kj::getStackTrace(kj::ArrayPtr, unsigned int) ()
>>>from /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
>>> #5  0xb7bb0b39 in kj::Exception::extendTrace(unsigned int) () from
>>> /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
>>> #6  0xb7bb0c62 in kj::throwRecoverableException(kj::Exception&&,
>>> unsigned int) ()
>>>from /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
>>> #7  0x0820011c in kj::Promise::wait(kj::WaitScope&) ()
>>> #8  0x081e1620 in kj::_::Fiber>> namespace)::TestCase904::run()::{lambda(kj::WaitScope&)#1}>::runImpl(kj::WaitScope&)
>>> ()
>>> #9  0xb7c36203 in
>>> kj::_::RunnableImpl::run() ()
>>>from /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
>>> #10 0xb7bb0ed2 in kj::_::runCatchingExceptions(kj::_::Runnable&) ()
>>>from /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
>>> #11 0xb7c3b9de in kj::_::FiberBase::run() () from
>>> /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
>>> #12 0xb7c3bdf7 in kj::_::FiberStack::run() () from
>>> /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
>>> #13 0xb7c42c09 in kj::_::FiberStack::StartRoutine::run(int, int) ()
>>>from /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
>>> #14 0xb77bbdeb in makecontext () at
>>> ../sysdeps/unix/sysv/linux/i386/makecontext.S:87
>>> #15 0x8c48 in ?? ()
>>> #16 0x0862 in ?? ()
>>> #17 0x in ?? ()
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/71489f35-286d-4967-aecd-636276f3dfb4n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google G

Re: [capnproto] Segfault in on Ubuntu 14.04 32-bit

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Zachary,

It looks like the segfault is happening in glibc's backtrace() function
while gathering the stack trace for an exception. An exception is actually
expected here (the test case tests throwing an exception). Since it
happened specifically in fiber-related tests, my guess is that glibc's
backtrace has a bug in which it is confused by the stack manipulation done
by fibers.

14.04 is quite old, so I imagine you're using a pretty old glibc here?
Maybe this is a glibc bug that was fixed in newer versions?

Note that fibers are new and completely optional feature of KJ. If you
don't use them then you can probably ignore the test failure.

-Kenton

On Wed, Nov 4, 2020 at 4:27 PM Zachary Dremann  wrote:

> I'm running on Ubuntu 14.04, 32 bit.
>
> I have a custom g++ 10.1.0 as g++-10 installed in /usr/local, built from
> source with no modifications or customizations
>
> I cloned from master (commit 96936a7446164424d8f3f50430d83ee21f63f6a2),
> and ran `autoreconf -fi`, then `./configure --without-openssl --with-pic
> LDFLAGS='-L/usr/local/lib -Wl,-rpath,/usr/local/lib'
> CXX=/usr/local/bin/g++-10`, followed by `make`, and `make check`.
>
> `VERBOSE=1 make check` fails with (truncated to show interesting stuff):
>
> [ TEST ] kj/async-test.c++:860: start a fiber
> [ PASS ] kj/async-test.c++:860: start a fiber (112 μs)
> [ TEST ] kj/async-test.c++:881: fiber promise chaining
> [ PASS ] kj/async-test.c++:881: fiber promise chaining (9 μs)
> [ TEST ] kj/async-test.c++:904: throw from a fiber
> /bin/bash: line 5: 22656 Segmentation fault  (core dumped) ${dir}$tst
> FAIL: capnp-test
> Randomly testing backwards-compatibility scenarios with seed: 1604520183
> PASS: capnp-evolution-test
> PASS: src/capnp/compiler/capnp-test.sh
> ===
> 1 of 3 tests failed
> Please report to capnproto@googlegroups.com
> ===
>
> Running `libtool --mode=execute gdb ./capnp-test`, and getting a
> backtrace at the time of the segfault gives the following:
>
> #0  0xb793d71e in x86_fallback_frame_state (context=,
> context=, fs=0xb57565b0)
> at ./md-unwind-support.h:132
> #1  uw_frame_state_for () at ../../../gcc-10.1.0/libgcc/unwind-dw2.c:1271
> #2  0xb793ecb1 in _Unwind_Backtrace () at
> ../../../gcc-10.1.0/libgcc/unwind.inc:302
> #3  0xb7876d75 in __GI___backtrace (array=0xb5756720, size=34) at
> ../sysdeps/i386/backtrace.c:126
> #4  0xb7bb098a in kj::getStackTrace(kj::ArrayPtr, unsigned int) ()
>from /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
> #5  0xb7bb0b39 in kj::Exception::extendTrace(unsigned int) () from
> /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
> #6  0xb7bb0c62 in kj::throwRecoverableException(kj::Exception&&, unsigned
> int) ()
>from /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
> #7  0x0820011c in kj::Promise::wait(kj::WaitScope&) ()
> #8  0x081e1620 in kj::_::Fiber namespace)::TestCase904::run()::{lambda(kj::WaitScope&)#1}>::runImpl(kj::WaitScope&)
> ()
> #9  0xb7c36203 in
> kj::_::RunnableImpl::run() ()
>from /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
> #10 0xb7bb0ed2 in kj::_::runCatchingExceptions(kj::_::Runnable&) ()
>from /home/build/capnproto/c++/.libs/libkj-0.9-dev.so
> #11 0xb7c3b9de in kj::_::FiberBase::run() () from
> /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
> #12 0xb7c3bdf7 in kj::_::FiberStack::run() () from
> /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
> #13 0xb7c42c09 in kj::_::FiberStack::StartRoutine::run(int, int) ()
>from /home/build/capnproto/c++/.libs/libkj-async-0.9-dev.so
> #14 0xb77bbdeb in makecontext () at
> ../sysdeps/unix/sysv/linux/i386/makecontext.S:87
> #15 0x8c48 in ?? ()
> #16 0x0862 in ?? ()
> #17 0x in ?? ()
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/71489f35-286d-4967-aecd-636276f3dfb4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DsJP5b8E8wY1ksctmCL%3D%3DwThVyjP6LnFdXV3bXRTAR2Q%40mail.gmail.com.

Re: [capnproto] enum brands?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

I probably did it for consistency. It's true that, at least at present,
it's not in any way meaningful for an enum to be parameterized. Moreover,
you cannot declare parameters in an enum declaration. However, an enum
could be nested within a parent scope that is parameterized, and in theory
everything within the scope is subject to those parameters -- but indeed,
since an enum type has no opportunity to refer to any parameter in its
definition, it is meaningless in practice.

-Kenton

On Fri, Oct 30, 2020 at 2:41 PM Ian Denhardt  wrote:

> Hey all,
>
> I was looking at schema.capnp, and noticed that the enum variant of the
> Type struct has a brand field. This doesn't make sense to me: unless I'm
> missing something, enums can't contain pointers, so it doesn't make
> sense to have type parameters on an enum. What is this field for?
>
> -Ian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/160408690654.5900.1088050232156870835%40localhost.localdomain
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DQBc4BFqXQWmK4tuJRZVrHPA0qz1dhN_eVXFOoLQEG7g%40mail.gmail.com.

Re: [capnproto] Middleware & Attaching metadata to calls/returns?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Erin,

In Cloudflare Workers, we've had some success attaching certain kinds of
metadata by having a wrapper object which is called first, like:

interface Endpoint {
  startCall @0 (metadata :Metadata) -> (dispatcher :CallDispatcher);
}

interface CallDispatcher {
  foo @0 (...) -> (...);
  bar @1 (...) -> (...);
  ...
}

Here, the client always first calls startCall() to pass the metadata, then
makes a pipelined call to the specific method. By convention, a
`CallDispatcher` is expected to be used for exactly one call.

Of course, there's a limit to how far this pattern can go. It has worked
well for us so far but it's definitely a bit clunky.

I'm really hesitant to attach a totally untyped int -> any map to every
call. Having built several systems using such a pattern in the past, I know
it can be very useful, but also tends to turn into a mess in certain ways.

For the specific use case of tracing, I can see a strong argument that this
should be built into the system. You really do want trace IDs to pass down
the call stack implicitly, without the business logic having to think about
it (except in cases where the business logic wishes to explicitly declare a
span). To make this work well in Cap'n Proto C++, we'd want to extend KJ's
promise framework to implicitly propagate a trace context along the promise
chain, so that the RPC system can pick it up.

I think authentication is at the opposite end of the spectrum. Implicitly
propagating authentication credentials goes against the capability
philosophy and gives rise to confused deputy problems. Authentication
should be achieved by having the client exchange their credentials for an
authenticated capability upfront, and then use that capability for further
calls. A membrane should revoke the capability if the credentials become
invalid. So I would not want to create a system that might be abused for
authentication.

Lamport timers are interesting. I don't have a lot of intuition here as I
haven't worked with them much. Naively it seems like you want to maintain a
time counter in each vat, send the vector with each call/return, and merge
the vectors received... however, this implies that the timer is ambient
mutable state, which introduces lots of problems. Does it become a side
channel through which it's possible to observe what else is going on within
the vat? This makes me a bit uncomfortable.

Maybe you could implement Lamport clocks with membranes. You could wrap
each group of objects that share a time counter in a single membrane. When
a call exits the membrane, the membrane could first generate a preceding
call to the same target object to some common propagateLamportTime()
method. On the receiving end, the membrane intercepts this call and updates
the receiver's clock. E-order guarantees that the propagateLamportTime()
call is received immediately before the call that caused it. Alternatively,
you could actually wrap outgoing calls in an envelope that include the
timestamp, and have the receiving membrane unwrap the envelope.

-Kenton

On Mon, Oct 12, 2020 at 1:15 PM Erin Shepherd  wrote:

> Many protocols provide some way to attach metadata to requests &
> responses. HTTP and protocols derved from it provide this in the form of
> headers (and occasionally trailers), for example.
>
> There are various reasons why this can be very useful, even in a
> capability oriented system:
>
>- Propagating a request correlation ID from node to node as a call
>passes through multiple parts of a system for logging correlation purposes
>- Even better, propagating an OpenTracing SpanContext or similar to
>enable correlating multiple parts of an action in a distributed system with
>tools like Jaeger 
>- Attaching Lamport timestamps
> to every request and
>reply, to propagate an inter-system causal ordering
>
> These sorts of things are very useful to convey *implicitly*. When every
> node in your system deals in lamport timestamps, then manually annotating
> every call with a timestamp parameter & manually handling that parameter
> and adding it to the response will quickly get tedious. If you want to do
> distributed tracing, then you want that context everywhere.
>
> These kinds of things represent the kind of state/context which is
> implicit from the callstack in traditional single-threaded applications
>
> There are some other places where I think metadata could be useful,
> although they probably are also more matters of taste, like
> identity/authentication (one of my thoughts here is that sometimes it might
> be advantageous to use metadata to pass this kind of information because
> this ensures it is always in a uniform location where intermediate
> proxies/membranes can find it without needing to be aware of involved
> schemas. You could potentially see a system which implements
> authentication/authorisation 

Re: [capnproto] How to implement streaming from server to client?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Dmitriy,

I had a long set of answers typed up for you, but then I came to your last
paragraph:


> PLEASE DO NOT BE LIKE STUPID AMERICANS. Do not answer for any question
> with "Yes". US citizens likes answer for complex question with non related
> answer.
> Please be as more verbose as you can. Not many people creating own
> libraries each weekend. So short answers without code is not answer at all.
> Thanks in advance for patience.
>

Based on this paragraph, I am not interested in answering your questions.
Please do not use Cap'n Proto. Maybe you should try gRPC instead.

Thanks,
-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DJJZCwzGHd-QQk64XGCG3-OXEjVFUopbHxFpdQEt%3Dzdg%40mail.gmail.com.

Re: [capnproto] Assert in exception.c++ triggering only in ASAN

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

I guess this means that under ASAN, the stack is not allocated in the
traditional way. For some reason there's a 1MB offset between where the
ExceptionCallback was allocated and where a local variable inside its
constructor was allocated.

The check there is pretty hacky and probably technically UB. It's meant to
detect and warn about bad usage. Assuming the ExceptionCallback is, in
fact, being allocated on the stack, then the exception is bogus. I suppose
we may need to #ifdef it out when ASAN is active, or find some other way to
verify that the object is stack-allocated...

-Kenton

On Mon, Oct 12, 2020 at 4:35 PM Vitali Lovich  wrote:

> I'm seeing this assert trigger in exception.c++ when run on ASAN on our
> (admittedly old) Linux boxes:
>
>   KJ_ASSERT(offset < 65536 && offset > -65536,
> "ExceptionCallback must be allocated on the stack.");
>
> The value of offset is 1048864. Is this code perhaps relying on some
> construct that's not reliable under ASAN? I would have expected ASAN to
> warn about that but it doesn't. Just the exception triggers.
>
> Unfortunately it's a bit hard to run this under a debugger or I would have
> delved far more on my own (& maybe disassemble things). Any insights would
> be appreciated.
>
> Thanks,
> Vitali
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/de1aaa9b-c798-467d-b1d2-11ac9ae42880n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmPhru_3%2B7XL0ywaQ%2BUu0Ldkxx5dZxnjUFBWzWQKH955Q%40mail.gmail.com.

Re: [capnproto] TSAN faiure

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Tue, Oct 13, 2020 at 5:15 AM Erin Shepherd  wrote:

> > I think that ThreadSanitizer is having trouble recognizing that the
> initialization of `brokenCapFactory` is thread-safe, due to the awkward way
> in which it is initialized. It may end up being initialized by multiple
> threads, but all threads will initialize it to the same value, hence no
> atomics are necessary when reading it later.
>
> This doesn't hold unless every thread reading it also once wrote it.
>

That is the case here.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn5Q%2Bz6Z2B9cuNUX6-tWF5E80EKMhB0wBqFAprkbuVBPw%40mail.gmail.com.

Re: [capnproto] TSAN faiure

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

An atomic read with "relaxed" ordering should be free. Even "acquire"
ordering (if TSAN insists on it) is free on x86. So I'd just have it always
be atomic.

-Kenton

On Fri, Oct 9, 2020 at 4:23 PM Vitali Lovich  wrote:

> Ok. That seems to have made TSAN happy (I still don't understand the exact
> reason it's safe but I'll trust you're higher level reasoning about
> the internals of capnp :D).
>
> Should this atomic read happen only when running under TSAN or just bite
> the bullet & do it regardless? It's unclear to me if this is a hotpath
> that's carefully tuned
>
> On Fri, Oct 9, 2020 at 10:00 AM 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
>> It looks like the writes are done atomically, but the reads aren't. TSAN
>> is right to be suspicious of this. In practice there is no bug here, but
>> showing that requires higher-level reasoning that TSAN wouldn't be expected
>> to figure out.
>>
>> -Kenton
>>
>> On Fri, Oct 9, 2020 at 11:52 AM Vitali Lovich  wrote:
>>
>>> So you're saying just change the read to atomic read?
>>>
>>> Is it possible there's a legitimate read-before-write race condition &
>>> that's what TSAN is complaining about? It doesn't seem to be complaining
>>> about concurrent writes but I also haven't investigated this codepath to
>>> understand yet as I assumed the problem was in my code.
>>>
>>> On Fri, Oct 9, 2020 at 9:48 AM Kenton Varda 
>>> wrote:
>>>
>>>> I think that ThreadSanitizer is having trouble recognizing that the
>>>> initialization of `brokenCapFactory` is thread-safe, due to the awkward way
>>>> in which it is initialized. It may end up being initialized by multiple
>>>> threads, but all threads will initialize it to the same value, hence no
>>>> atomics are necessary when reading it later.
>>>>
>>>> Maybe we should use atomic reads anyway, to make ThreadSanitizer happy.
>>>> Doing so should be free, at least on x86.
>>>>
>>>> (The reason for the weird initialization is that the factory is
>>>> implemented in libcapnp-rpc.so, yet needs to be callable from libcapnp.so
>>>> -- but only if RPC is in use.)
>>>>
>>>> -Kenton
>>>>
>>>> On Fri, Oct 9, 2020 at 8:37 AM Vitali Lovich  wrote:
>>>>
>>>>> I'm trying to do a basic in-process connection of the servers &
>>>>> running that under TSAN but I feel like this is some nuance of the APIs 
>>>>> I'm
>>>>> missing or using the API totally incorrectly outright, so I would
>>>>> appreciate some feedback if possible. I'm sure the bug must be in my code
>>>>> rather than cap'n'proto. I don't have a helpful standalone piece of code 
>>>>> to
>>>>> demonstrate this issue at the moment (but if it's really critical I can
>>>>> work on providing that). I've provided brief snippets of what the threads
>>>>> do (the threads themselves are really that simple, it's just the schema &
>>>>> the threads using various helper internal libraries that make it harder to
>>>>> post a working standalone example).
>>>>>
>>>>> Full TSAN (to avoid spamming the list): https://pastebin.com/hSkVDgsU
>>>>>
>>>>> For context, the main thread does
>>>>> auto ioContext = kj::setupAsyncIo();
>>>>> auto serverThread = ioContext.provider->newPipeThread(...);
>>>>> auto serverPtr = ;
>>>>> capnp::TwoPartyClient rpcClient(*serverThread.pipe);
>>>>> auto client = rpcClient.bootstrap().castAs();
>>>>> auto connectedResponse =
>>>>> client.connectRequest().send().wait(ioContext.waitScope); *<-- this
>>>>> is the problematic TSAN line *
>>>>>
>>>>> T1 (taking AsyncIoProvider&, AsyncIoStream& stream, kj::WaitScope&
>>>>> waitScope):
>>>>> capnp::TwoPartyVatNetwork network(stream,
>>>>> capnp::rpc::twoparty::Side::SERVER);
>>>>> auto myServer = kj::heap();
>>>>> auto myServerPtr = myServer.get()
>>>>> auto rpcServer = capnp::makeRpcServer(network, kj::mv( myServer));
>>>>> 
>>>>> network.onDisconnect().wait(waitScope);* <- problematic TSAN line*
>>>>>
>>>&

Re: [capnproto] TSAN faiure

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

It looks like the writes are done atomically, but the reads aren't. TSAN is
right to be suspicious of this. In practice there is no bug here, but
showing that requires higher-level reasoning that TSAN wouldn't be expected
to figure out.

-Kenton

On Fri, Oct 9, 2020 at 11:52 AM Vitali Lovich  wrote:

> So you're saying just change the read to atomic read?
>
> Is it possible there's a legitimate read-before-write race condition &
> that's what TSAN is complaining about? It doesn't seem to be complaining
> about concurrent writes but I also haven't investigated this codepath to
> understand yet as I assumed the problem was in my code.
>
> On Fri, Oct 9, 2020 at 9:48 AM Kenton Varda  wrote:
>
>> I think that ThreadSanitizer is having trouble recognizing that the
>> initialization of `brokenCapFactory` is thread-safe, due to the awkward way
>> in which it is initialized. It may end up being initialized by multiple
>> threads, but all threads will initialize it to the same value, hence no
>> atomics are necessary when reading it later.
>>
>> Maybe we should use atomic reads anyway, to make ThreadSanitizer happy.
>> Doing so should be free, at least on x86.
>>
>> (The reason for the weird initialization is that the factory is
>> implemented in libcapnp-rpc.so, yet needs to be callable from libcapnp.so
>> -- but only if RPC is in use.)
>>
>> -Kenton
>>
>> On Fri, Oct 9, 2020 at 8:37 AM Vitali Lovich  wrote:
>>
>>> I'm trying to do a basic in-process connection of the servers & running
>>> that under TSAN but I feel like this is some nuance of the APIs I'm missing
>>> or using the API totally incorrectly outright, so I would appreciate some
>>> feedback if possible. I'm sure the bug must be in my code rather than
>>> cap'n'proto. I don't have a helpful standalone piece of code to demonstrate
>>> this issue at the moment (but if it's really critical I can work on
>>> providing that). I've provided brief snippets of what the threads do (the
>>> threads themselves are really that simple, it's just the schema & the
>>> threads using various helper internal libraries that make it harder to post
>>> a working standalone example).
>>>
>>> Full TSAN (to avoid spamming the list): https://pastebin.com/hSkVDgsU
>>>
>>> For context, the main thread does
>>> auto ioContext = kj::setupAsyncIo();
>>> auto serverThread = ioContext.provider->newPipeThread(...);
>>> auto serverPtr = ;
>>> capnp::TwoPartyClient rpcClient(*serverThread.pipe);
>>> auto client = rpcClient.bootstrap().castAs();
>>> auto connectedResponse =
>>> client.connectRequest().send().wait(ioContext.waitScope); *<-- this is
>>> the problematic TSAN line *
>>>
>>> T1 (taking AsyncIoProvider&, AsyncIoStream& stream, kj::WaitScope&
>>> waitScope):
>>> capnp::TwoPartyVatNetwork network(stream,
>>> capnp::rpc::twoparty::Side::SERVER);
>>> auto myServer = kj::heap();
>>> auto myServerPtr = myServer.get()
>>> auto rpcServer = capnp::makeRpcServer(network, kj::mv( myServer));
>>> 
>>> network.onDisconnect().wait(waitScope);* <- problematic TSAN line*
>>>
>>> SUMMARY: ThreadSanitizer: data race
>>> capnproto/c++/src/capnp/layout.c++:2188:5 in
>>> capnp::_::WireHelpers::readCapabilityPointer(capnp::_::SegmentReader*,
>>> capnp::_::CapTableReader*, capnp::_::WirePointer const*, int)
>>>
>>>   Read of size 8 at 0x016673e0 by main thread:
>>> #0
>>> capnp::_::WireHelpers::readCapabilityPointer(capnp::_::SegmentReader*,
>>> capnp::_::CapTableReader*, capnp::_::WirePointer const*, int)
>>> capnproto/c++/src/capnp/layout.c++:2188:5 (ld-2.26.so+0x809e60)
>>> #1 capnp::_::PointerReader::getCapability() const
>>> capnproto/c++/src/capnp/layout.c++:2705 (ld-2.26.so+0x809e60)
>>> #2
>>> capnp::AnyPointer::Reader::getPipelinedCap(kj::ArrayPtr>> const>) const capnproto/c++/src/capnp/any.c++:53:18 (ld-2.26.so
>>> +0x7ff4a7)
>>> #3 capnp::_::(anonymous
>>>
>>>   Previous atomic write of size 8 at 0x016673e0 by thread T1:
>>> #0 __tsan_atomic64_store  (ld-2.26.so+0x6911ee)
>>> #1
>>> capnp::_::setGlobalBrokenCapFactoryForLayoutCpp(capnp::_::BrokenCapFactory&)
>>> capnproto/c++/src/capnp/layout.c++:45:3 (ld-2.26.so+0x800d12)
>>> #2
>>> capnp::ReaderCapabilityTable::ReaderCapabilityTable(kj::Array
>>> > >) capnproto/c++/src/capnp/capability.c++:951:3 (ld-2.26.so+0x6f32cc)
>>> #3 capnp::_::(anonymous
>>> namespace)::RpcConnectionState::RpcCallContext::RpcCallContext(capnp::_::(anonymous
>>> namespace)::RpcConnectionState&, unsigned int,
>>> kj::Own&&,
>>> kj::Array > >,
>>> capnp::AnyPointer::Reader const&, bool, kj::Own
>>> >&&, unsigned long, unsigned short) capnproto/c++/src/capnp/rpc.c++:2144:11
>>> (ld-2.26.so+0x754bea)
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnpr

Re: [capnproto] TSAN faiure

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

I think that ThreadSanitizer is having trouble recognizing that the
initialization of `brokenCapFactory` is thread-safe, due to the awkward way
in which it is initialized. It may end up being initialized by multiple
threads, but all threads will initialize it to the same value, hence no
atomics are necessary when reading it later.

Maybe we should use atomic reads anyway, to make ThreadSanitizer happy.
Doing so should be free, at least on x86.

(The reason for the weird initialization is that the factory is implemented
in libcapnp-rpc.so, yet needs to be callable from libcapnp.so -- but only
if RPC is in use.)

-Kenton

On Fri, Oct 9, 2020 at 8:37 AM Vitali Lovich  wrote:

> I'm trying to do a basic in-process connection of the servers & running
> that under TSAN but I feel like this is some nuance of the APIs I'm missing
> or using the API totally incorrectly outright, so I would appreciate some
> feedback if possible. I'm sure the bug must be in my code rather than
> cap'n'proto. I don't have a helpful standalone piece of code to demonstrate
> this issue at the moment (but if it's really critical I can work on
> providing that). I've provided brief snippets of what the threads do (the
> threads themselves are really that simple, it's just the schema & the
> threads using various helper internal libraries that make it harder to post
> a working standalone example).
>
> Full TSAN (to avoid spamming the list): https://pastebin.com/hSkVDgsU
>
> For context, the main thread does
> auto ioContext = kj::setupAsyncIo();
> auto serverThread = ioContext.provider->newPipeThread(...);
> auto serverPtr = ;
> capnp::TwoPartyClient rpcClient(*serverThread.pipe);
> auto client = rpcClient.bootstrap().castAs();
> auto connectedResponse =
> client.connectRequest().send().wait(ioContext.waitScope); *<-- this is
> the problematic TSAN line *
>
> T1 (taking AsyncIoProvider&, AsyncIoStream& stream, kj::WaitScope&
> waitScope):
> capnp::TwoPartyVatNetwork network(stream,
> capnp::rpc::twoparty::Side::SERVER);
> auto myServer = kj::heap();
> auto myServerPtr = myServer.get()
> auto rpcServer = capnp::makeRpcServer(network, kj::mv( myServer));
> 
> network.onDisconnect().wait(waitScope);* <- problematic TSAN line*
>
> SUMMARY: ThreadSanitizer: data race
> capnproto/c++/src/capnp/layout.c++:2188:5 in
> capnp::_::WireHelpers::readCapabilityPointer(capnp::_::SegmentReader*,
> capnp::_::CapTableReader*, capnp::_::WirePointer const*, int)
>
>   Read of size 8 at 0x016673e0 by main thread:
> #0
> capnp::_::WireHelpers::readCapabilityPointer(capnp::_::SegmentReader*,
> capnp::_::CapTableReader*, capnp::_::WirePointer const*, int)
> capnproto/c++/src/capnp/layout.c++:2188:5 (ld-2.26.so+0x809e60)
> #1 capnp::_::PointerReader::getCapability() const
> capnproto/c++/src/capnp/layout.c++:2705 (ld-2.26.so+0x809e60)
> #2
> capnp::AnyPointer::Reader::getPipelinedCap(kj::ArrayPtr const>) const capnproto/c++/src/capnp/any.c++:53:18 (ld-2.26.so+0x7ff4a7)
> #3 capnp::_::(anonymous
>
>   Previous atomic write of size 8 at 0x016673e0 by thread T1:
> #0 __tsan_atomic64_store  (ld-2.26.so+0x6911ee)
> #1
> capnp::_::setGlobalBrokenCapFactoryForLayoutCpp(capnp::_::BrokenCapFactory&)
> capnproto/c++/src/capnp/layout.c++:45:3 (ld-2.26.so+0x800d12)
> #2
> capnp::ReaderCapabilityTable::ReaderCapabilityTable(kj::Array
> > >) capnproto/c++/src/capnp/capability.c++:951:3 (ld-2.26.so+0x6f32cc)
> #3 capnp::_::(anonymous
> namespace)::RpcConnectionState::RpcCallContext::RpcCallContext(capnp::_::(anonymous
> namespace)::RpcConnectionState&, unsigned int,
> kj::Own&&,
> kj::Array > >,
> capnp::AnyPointer::Reader const&, bool, kj::Own
> >&&, unsigned long, unsigned short) capnproto/c++/src/capnp/rpc.c++:2144:11
> (ld-2.26.so+0x754bea)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/19bac31b-6f11-43d1-886e-93d6bc32557an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmfGxyTG%3Dt8c38aTPJ5JHTJFmn0LHvFZaTPVu9Pr8dy3Q%40mail.gmail.com.

Re: [capnproto] Implementing Level-3 Protocol (in Go)

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Was someone going to send out a meeting invite or something?

On Wed, Sep 30, 2020, 4:11 PM Louis Thibault  wrote:

> Alright, it seems like everyone is eager to meet this Friday, so let’s do
> that. :)
>
> See you all on Friday, as initially planned.
>
> - Louis
>
> On Sep 30, 2020, at 17:08, Kenton Varda  wrote:
>
> I'm still happy to chat this Friday.
>
> -Kenton
>
> On Wed, Sep 30, 2020 at 3:41 PM Louis Thibault  wrote:
>
>> 100% agree — let’s not lose momentum.  I’ll send out another invite when
>> we hear back from Ross.
>>
>> Cheers,
>> Louis
>>
>> On Sep 30, 2020, at 16:27, Ian Denhardt  wrote:
>>
>> I'm okay re-scheduling for the 16th if others are. I think we should
>> just plan to move forward one way or the other from there though;
>> there's some utility maybe in having Kenton work out the kinks first,
>> but ultimately I feel like I understand things well enough to move ahead
>> of the C++ implementation if we need to, and I'd rather not block on it
>> indefinitely.
>>
>> -Ian
>>
>> Quoting Louis Thibault (2020-09-30 15:45:21)
>>
>>   Hi Kenton,
>>
>>   Thanks for your email and congratulations on the launch!  Looks really
>>   cool!
>>
>>   It'll probably take the least time overall if I can manage to take an
>>   initial crack at it first in C++ and just power through things, rather
>>   than have other people try to implement the spec as written and then
>>   have to ask a lot of questions when there's confusion.
>>
>>   Sure, that makes perfect sense.  I'm thinking it's probably best if we
>>   push back the meeting until after you've laid down the foundations for
>>   3PH, at which point we can kill two birds with one stone:  changes to
>>   the spec based on your C++ implementation, and considerations for the
>>   Go implementation.
>>
>>   Kenton, I don't want to pressure you into meeting any sort of deadline,
>>   but I also don't want this project to fall through the cracks.  Is it
>>   ok if we tentatively reschedule for e.g. Friday October 16th at 13:00
>>   PDT and make any adjustments as-needed?  (Ross, Ian, does that date
>>   also work for you?)
>>
>>   Cheers,
>>
>>   Louis
>>
>>   On Sep 30, 2020, at 15:29, Kenton Varda <[1]ken...@cloudflare.com>
>>   wrote:
>>
>>   Sorry for the very slow response this time. We had a big launch on
>>   Monday and I haven't had much time to reply to e-mail.
>>   Incidentally that launch is vaguely relevant here.
>>   [2]https://blog.cloudflare.com/introducing-workers-durable-objects/
>>   We built the infrastructure for this on Cap'n Proto. When you make a
>>   request to a Durable Object, you're actually making Cap'n Proto RPC
>>   requests all they way from the call site to the destination object.
>>   Since Durable Objects will soon be able to migrate around the world,
>>   3-party handoff is likely to become important as old locations will
>>   need to forward requests to new locations.
>>   The Workers Runtime is written in C++, but we do have a fair amount of
>>   Go code at Cloudflare, so it'd be neat if 3PH were supported there too.
>>   In any case, I'd like to start working on this soon, but I also have a
>>   big pile of other things I need to work on so I'm not sure how much
>>   time I'll have to dedicate.
>>   Based on my experience working on the RPC system in the past, I think
>>   it's safe to expect that as we implement 3PH, we're likely to run into
>>   a bunch of little spec issues. It'll probably take the least time
>>   overall if I can manage to take an initial crack at it first in C++ and
>>   just power through things, rather than have other people try to
>>   implement the spec as written and then have to ask a lot of questions
>>   when there's confusion. I'm hoping to get at least a few days to work
>>   on it maybe the week after next.
>>   As it happens I am available at 13:00 PDT (15:00 CDT) Friday if you
>>   still want to chat then.
>>   -Kenton
>>   On Fri, Sep 25, 2020 at 10:59 AM Ian Denhardt <[3]i...@zenhack.net>
>>   wrote:
>>
>> Works for me.
>> Quoting lthibault (2020-09-25 11:04:56)
>>
>>   Friday October 2 at 13:00 PST (16:00 EST for Ian and myself)
>>
>> works for
>>
>>   me.
>>
>>   Ian, is that ok with you?
>>
>>   Kenton, any chance we might count you in as well?
>>
>>   - Louis
>>
>>   On Friday, September 25, 2020 at 10:52:25 AM UTC-4
>>
>> [4]rli...@gmail.com
>>
>>   wrote:
>>
>> I'm available next Friday (Oct 2) from 1p PT onward.
>>
>>   On Friday, September 25, 2020 at 7:36:07 AM UTC-7 lthibault
>>
>> wrote:
>>
>>
>> Sounds good to me � looking forward to it.
>> - Louis
>>
>> On Sep 24, 2020, at 17:17, Ian Denhardt
>>
>> <[5]i...@zenhack.net> wrote:
>>
>>
>> I just opened an issue:
>>
>> [1][6]https://github.com/capnproto/go-capnproto2/issues/160
>>
>> I believe Kenton is in Texas these days. For my part I
>>
>> could
>>
>> manage most
>>
>> afternoons/evenings (eastern time) though Monday & Thursday
>>
>> afternoons
>>
>> are bad. But if Ross 

Re: [capnproto] Standardized schema interchange

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Wed, Sep 23, 2020 at 11:52 PM Ryan Patterson 
wrote:

> Hi Kenton, thanks for weighing in!
>
> > I don't think the generated code would implement the getSchema() RPC
> directly for each type. Instead I think I'd add a new method to the
> `Capability::Server` and `ClientHook` like:
>
> I think we are thinking the same thing here, but to confirm: The
> capability would respond to a standardized method, but the implementation
> of this standard method is provided by the library and not by the generated
> code. Are we in agreement?
>

Right. But I thought you were suggesting the implementation lives in
generated code, whereas I'm suggesting it mostly lives in the RPC system.


> > I'm a bit worried about redundantly transmitting the same schema nodes
> over the wire many times, especially when dealing with many different
> interfaces that implement the same underlying types.
>
> Presumably, a peer would call it on the Bootstrap/Accepted capability,
> which would necessarily pull in every reachable Schema.Node, except for
> through AnyPointer boundaries. So, while the first call might be redundant
> if I did already have the schema, it wouldn't be "many times" in general.
> The worst case is a highly-dynamic interface where every method returns an
> untyped Capability. In such a case, 2 methods (my proposal in the last
> email) makes more sense than a combined method.
>

Technically any capability could implement arbitrary interfaces -- it could
implement a derived class of what it is statically declared to implement,
and that derived class could inherit any other interface as well.

I suppose you could have the calling code explicitly say when it wishes to
query the capability for additional interfaces, rather than assuming the
statically-derived type.


> And via promise pipelining, the 2 method solution shouldn't materially be
> slower than the combined method in the usual case.
>

Not sure if promise pipelining helps here. You'd need to
call supportedInterfaces() first *and wait for it to complete* in order to
find out which type IDs are not in the set you already know about, and only
then could you call schemaForTypes(). So it's still two network round trips.

To do it in one round trip the client would have to tell the server about
types it already knows about. I suppose if the caller already has an
expectation of the capability's type via static typing, it could send that,
so that it only gets a non-empty response if the capability turns out to
implement more than that.


> > There's also the problem that two different servers could have
> conflicting versions of a schema, but could both end up proxying through
> the same third server, meaning you might get different versions of the same
> schema for different capabilities on the same connection. Maybe it's not
> actually feasible to share schemas between capabilities at all?
>
> Let's split this into 2 problems: known (:MyInterface) and unknown
> (:Capability) capabilities. In the first case, a message broker is proxying
> MyInterface between peers which have different underlying versions of the
> same schema. This means that the message broker has a particular schema
> which it supports, and the introspecting peer should treat everything
> coming from the broker as though it were that schema.
>

Cap'n Proto supports passing through messages without knowing their full
schema. This is very important for message brokers/proxies in particular --
you don't want to have to recompile the broker every time you add a new
field to the message. Only the initial producer and final consumer need to
know about a field.


> In the second case, the message broker doesn't have a schema for those
> types, so the introspecting code automatically knows it needs to request
> the schema for the received capability. Because of the case where these
> unknown capabilities might be from different schemas (or different versions
> of one), the introspecting peer shouldn't share schemas between these
> capabilities. The net cost of this is  transferring a schema object
> whenever introspecting an untyped Capability (although that transfer may
> not actually be redundant--we won't know ahead of time).
>

But if you assume the types are disjoint, does that mean that if you
discover the new capability implements interface Foo, you can't actually
use it in a call to the original capability that expects a Foo?

Well, I suppose we could allow it to be used as long as the schemas have
the same type ID, independent of whether the schemas are actually
compatible.

So sure, in that case I buy that when people explicitly query a
capability's dynamic type, we could build a whole new schema graph separate
from the previously-known static type. In most cases it won't be necessary
to query the dynamic type anyway since the static type will have everything
you need.

-Kenton


>
> In conclusion, it feels like attempting to cache/reuse schemas makes the
> project big and possibly intractable, b

Re: [capnproto] How to keep some incoming messages as is and reuse?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Ambrase,

copyToUnchecked() does not write a segment table at the beginning of the
message. That's probably why you're seeing a message 8 bytes shorter.

copyToUnchecked() and readMessageUnchecked() are difficult to use correctly
(and are insecure when used incorrectly). I strongly recommend against
using them.

Instead, consider using capnp::clone(reader) if you just want to clone the
message in memory that you can then pass around with move semantics.

-Kenton

On Tue, Sep 29, 2020 at 9:40 AM Ambrase  wrote:

> Hi
>
> I have an input byte buffer with N messages. There are 2 cases: this byte
> buffer is zero-byte packed or not.
> I read messages like this:
> capnp::PackedMessageReader packedReader(cpnpReader.getAdapter(),
> capnp::ReaderOptions(), cpnpReader.getArray()); (where the adapter is an
> adapter for InputStream on the top of the received packet from UDP/Mcast,
> the array is an unpacked output buffer).
> or with capnp::FlatArrayMessageReader if the message isn't packed.
>
> I'm trying to save some received messages in the way that they will be
> movable, unpacked and at the same time ready to send.
> I get a reader as:
> auto anymessage = packedReader.getRoot();
> then I can copy it to a bytebuffer that can be moved or stored:
> capnp::copyToUnchecked(anymessage, arrayPtr);
>
> The copied buffer is 8 byte less. I assume it is lacking the WirePointer.
> 1)Can I somehow save it with WirePointer at all?
> 2)How to reuse the saved buffer without the WirePointer as a ready message
> for
> MallocMessageBuilder?
>
> I also tried this way:
> capnp::AnyPointer::Reader reader =
> capnp::readMessageUnchecked(reinterpret_cast capnp::word*>(storedBuffer.data()));
> m_builder.setRoot(kj::fwd(reader));
>
> But the builder writes 0. So, I did something wrong.
>
> Thanks for your help.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/a6951785-d283-4460-a3e5-9ef94a2efffen%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DZRR-_2143fWPgok0QKS6Bj77hDzzsR%2BEcbx4Jgp_wYQ%40mail.gmail.com.

Re: [capnproto] Implementing Level-3 Protocol (in Go)

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

I'm still happy to chat this Friday.

-Kenton

On Wed, Sep 30, 2020 at 3:41 PM Louis Thibault  wrote:

> 100% agree — let’s not lose momentum.  I’ll send out another invite when
> we hear back from Ross.
>
> Cheers,
> Louis
>
> On Sep 30, 2020, at 16:27, Ian Denhardt  wrote:
>
> I'm okay re-scheduling for the 16th if others are. I think we should
> just plan to move forward one way or the other from there though;
> there's some utility maybe in having Kenton work out the kinks first,
> but ultimately I feel like I understand things well enough to move ahead
> of the C++ implementation if we need to, and I'd rather not block on it
> indefinitely.
>
> -Ian
>
> Quoting Louis Thibault (2020-09-30 15:45:21)
>
>   Hi Kenton,
>
>   Thanks for your email and congratulations on the launch!  Looks really
>   cool!
>
>   It'll probably take the least time overall if I can manage to take an
>   initial crack at it first in C++ and just power through things, rather
>   than have other people try to implement the spec as written and then
>   have to ask a lot of questions when there's confusion.
>
>   Sure, that makes perfect sense.  I'm thinking it's probably best if we
>   push back the meeting until after you've laid down the foundations for
>   3PH, at which point we can kill two birds with one stone:  changes to
>   the spec based on your C++ implementation, and considerations for the
>   Go implementation.
>
>   Kenton, I don't want to pressure you into meeting any sort of deadline,
>   but I also don't want this project to fall through the cracks.  Is it
>   ok if we tentatively reschedule for e.g. Friday October 16th at 13:00
>   PDT and make any adjustments as-needed?  (Ross, Ian, does that date
>   also work for you?)
>
>   Cheers,
>
>   Louis
>
>   On Sep 30, 2020, at 15:29, Kenton Varda <[1]ken...@cloudflare.com>
>   wrote:
>
>   Sorry for the very slow response this time. We had a big launch on
>   Monday and I haven't had much time to reply to e-mail.
>   Incidentally that launch is vaguely relevant here.
>   [2]https://blog.cloudflare.com/introducing-workers-durable-objects/
>   We built the infrastructure for this on Cap'n Proto. When you make a
>   request to a Durable Object, you're actually making Cap'n Proto RPC
>   requests all they way from the call site to the destination object.
>   Since Durable Objects will soon be able to migrate around the world,
>   3-party handoff is likely to become important as old locations will
>   need to forward requests to new locations.
>   The Workers Runtime is written in C++, but we do have a fair amount of
>   Go code at Cloudflare, so it'd be neat if 3PH were supported there too.
>   In any case, I'd like to start working on this soon, but I also have a
>   big pile of other things I need to work on so I'm not sure how much
>   time I'll have to dedicate.
>   Based on my experience working on the RPC system in the past, I think
>   it's safe to expect that as we implement 3PH, we're likely to run into
>   a bunch of little spec issues. It'll probably take the least time
>   overall if I can manage to take an initial crack at it first in C++ and
>   just power through things, rather than have other people try to
>   implement the spec as written and then have to ask a lot of questions
>   when there's confusion. I'm hoping to get at least a few days to work
>   on it maybe the week after next.
>   As it happens I am available at 13:00 PDT (15:00 CDT) Friday if you
>   still want to chat then.
>   -Kenton
>   On Fri, Sep 25, 2020 at 10:59 AM Ian Denhardt <[3]i...@zenhack.net>
>   wrote:
>
> Works for me.
> Quoting lthibault (2020-09-25 11:04:56)
>
>   Friday October 2 at 13:00 PST (16:00 EST for Ian and myself)
>
> works for
>
>   me.
>
>   Ian, is that ok with you?
>
>   Kenton, any chance we might count you in as well?
>
>   - Louis
>
>   On Friday, September 25, 2020 at 10:52:25 AM UTC-4
>
> [4]rli...@gmail.com
>
>   wrote:
>
> I'm available next Friday (Oct 2) from 1p PT onward.
>
>   On Friday, September 25, 2020 at 7:36:07 AM UTC-7 lthibault
>
> wrote:
>
>
> Sounds good to me � looking forward to it.
> - Louis
>
> On Sep 24, 2020, at 17:17, Ian Denhardt
>
> <[5]i...@zenhack.net> wrote:
>
>
> I just opened an issue:
>
> [1][6]https://github.com/capnproto/go-capnproto2/issues/160
>
> I believe Kenton is in Texas these days. For my part I
>
> could
>
> manage most
>
> afternoons/evenings (eastern time) though Monday & Thursday
>
> afternoons
>
> are bad. But if Ross and/or Kenton are interested in
>
> joining us I
>
> suspect their schedules are much more constrained than
>
> mine.
>
>
> -Ian
>
> Quoting lthibault (2020-09-23 18:31:05)
>
> Are you amenable to Jitsi Meet[1] instead of Zoom?
>
> That's totally fine. I've been meaning to ditch Zoom,
>
> anyway.
>
>
> We should also perhaps reach out to Ross
>
>
> Absolutely. Would you be able to reach out to him?
>
> I think you and I are in the same tim

Re: [capnproto] Implementing Level-3 Protocol (in Go)

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Sorry for the very slow response this time. We had a big launch on Monday
and I haven't had much time to reply to e-mail.

Incidentally that launch is vaguely relevant here.

https://blog.cloudflare.com/introducing-workers-durable-objects/

We built the infrastructure for this on Cap'n Proto. When you make a
request to a Durable Object, you're actually making Cap'n Proto RPC
requests all they way from the call site to the destination object. Since
Durable Objects will soon be able to migrate around the world, 3-party
handoff is likely to become important as old locations will need to forward
requests to new locations.

The Workers Runtime is written in C++, but we do have a fair amount of Go
code at Cloudflare, so it'd be neat if 3PH were supported there too.

In any case, I'd like to start working on this soon, but I also have a big
pile of other things I need to work on so I'm not sure how much time I'll
have to dedicate.

Based on my experience working on the RPC system in the past, I think it's
safe to expect that as we implement 3PH, we're likely to run into a bunch
of little spec issues. It'll probably take the least time overall if I can
manage to take an initial crack at it first in C++ and just power through
things, rather than have other people try to implement the spec as written
and then have to ask a lot of questions when there's confusion. I'm hoping
to get at least a few days to work on it maybe the week after next.

As it happens I am available at 13:00 PDT (15:00 CDT) Friday if you still
want to chat then.

-Kenton

On Fri, Sep 25, 2020 at 10:59 AM Ian Denhardt  wrote:

> Works for me.
>
> Quoting lthibault (2020-09-25 11:04:56)
> >Friday October 2 at 13:00 PST (16:00 EST for Ian and myself) works for
> >me.
> >
> >Ian, is that ok with you?
> >
> >Kenton, any chance we might count you in as well?
> >
> >- Louis
> >
> >On Friday, September 25, 2020 at 10:52:25 AM UTC-4 rli...@gmail.com
> >wrote:
> >
> >  I'm available next Friday (Oct 2) from 1p PT onward.
> >
> >On Friday, September 25, 2020 at 7:36:07 AM UTC-7 lthibault wrote:
> >
> >  Sounds good to me � looking forward to it.
> >  - Louis
> >  > On Sep 24, 2020, at 17:17, Ian Denhardt  wrote:
> >  >
> >  > I just opened an issue:
> >  >
> >  > [1]https://github.com/capnproto/go-capnproto2/issues/160
> >  >
> >  > I believe Kenton is in Texas these days. For my part I could
> >  manage most
> >  > afternoons/evenings (eastern time) though Monday & Thursday
> >  afternoons
> >  > are bad. But if Ross and/or Kenton are interested in joining us I
> >  > suspect their schedules are much more constrained than mine.
> >  >
> >  > -Ian
> >  >
> >  > Quoting lthibault (2020-09-23 18:31:05)
> >  >>> Are you amenable to Jitsi Meet[1] instead of Zoom?
> >  >> That's totally fine. I've been meaning to ditch Zoom, anyway.
> >  >>
> >  >>> We should also perhaps reach out to Ross
> >  >>
> >  >> Absolutely. Would you be able to reach out to him?
> >  >>
> >  >> I think you and I are in the same time zone (I'm in New Haven,
> >  CT), and
> >  >> I believe Kenton and Ross are on the west coast, so maybe early
> >  early
> >  >> afternoon?
> >  >>
> >  >> Any particular preference for the day?
> >  >>
> >  >> Cheers,
> >  >>
> >  >> Louis
> >  >>
> >  >> On Wednesday, September 23, 2020 at 5:36:03 PM UTC-4
> >  i...@zenhack.net
> >  >> wrote:
> >  >>
> >  >> Quoting lthibault (2020-09-23 14:41:13)
> >  >>> How do you suggest we proceed? I'm thinking it might be useful
> >  to
> >  >> get
> >  >>> synchronized over a Zoom call since (a) I could benefit from a
> >  >> guided
> >  >>> tour of v3 and (b) my initial high-level questions about the
> >  Level
> >  >> 3
> >  >>> protocol remain open. Would be cool if Kenton could join us too.
> >  >> :)
> >  >> I'd be up for a video call. Are you amenable to Jitsi Meet[1]
> >  >> instead of
> >  >> Zoom? We've been using it for the weekly Sandstorm office hours
> >  >> calls.
> >  >> We should also perhaps reach out to Ross; I know he doesn't have
> >  >> time to
> >  >> hack on it, but might still be interested in weighing in on
> >  design
> >  >> questions/offering advice. Want to open an issue re: level 3 on
> >  the
> >  >> Go
> >  >> implementation's issue tracker? Maybe we can coordinate the rest
> >  of
> >  >> this there. I know Ross is also on this mailing list, so he may
> >  see
> >  >> this
> >  >> anyway, but he'll definitely see it there.
> >  >> -Ian
> >  >> [1]: [1][2]https://meet.jit.si
> >  >>
> >  >> --
> >  >> You received this message because you are subscribed to the
> >  Google
> >  >> Groups "Cap'n Proto" group.
> >  >> To unsubscribe from this group

Re: [capnproto] Implementing Level-3 Protocol (in Go)

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Louis,

This is somewhat embarrassing, but there is no level 3 implementation of
Cap'n Proto today, in any language. It's likely that when I get around to
implementing it in C++, the protocol spec will change to solve
unanticipated problems. As such, trying to implement the spec as-written in
Go today might be frustrating, as it may be hard to tell the difference
between the spec having a bug vs. the spec doing something weird
intentionally, and there's no reference implementation to test against. :/

As for level 2, yes, we eventually realized that level 2 doesn't really
belong in the core spec at all, so you can pretty much ignore the level 2
interfaces. Maybe I should actually delete them...

-Kenton

On Wed, Sep 23, 2020 at 11:02 AM lthibault  wrote:

> Hello everyone,
>
> I'm currently using the Go implementation
>  of Cap'n Proto, which
> features Level 1 support for the RPC protocol, and I'm discovering that my
> use case would benefit greatly from Level 3 support.  Since Ross is no
> longer actively maintaining the project, I'm investigating the possibility
> of implementing Level 3 support myself, and possibly volunteering to
> maintain the project.
>
> I've read through the Cap'n Proto spec (i.e. the .capnp files in the
> github repository, especially the network interface sketch
> ),
> perused conversations in this Google Group, and even tried to scavenge from
> the E language website/wiki, but I'm still struggling to get a "big
> picture" idea of how the 3-way introduction protocol works.
>
> As such, I'm you kind folks might be able to point me in the right
> direction, starting with two very general questions:
>
>1. How does the 3-way introduction protocol work?  Is it specified
>anywhere (e.g. in an RFC?)
>2. What does the roadmap look like from Leve 1 to Level 3?
>3. My understanding from reading the source is that Level 2 can
>largely be ignored.  Does Level 3 depend on level 2 in any way?  The
>numbering suggests so...
>
> Many thanks in advance,
> Louis
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/e9ad3f6d-a78c-4991-a3de-9a0fc071b906n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnAGjqqv7BTGt8Pi9qCxr%2Bu4n1wiEma-xo3bDbgSzqhjQ%40mail.gmail.com.

Re: [capnproto] Standardized schema interchange

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

This is something I've wanted to do for a while (built into the `capnp`
tool).

I agree that RPC types should be introspectable. I'd design the interface
like this:

getSchema @0 () -> (typeId :UInt64, nodes :List(Schema.Node))

`nodes` would be a list of schema nodes describing the transitive closure
of dependencies. (Each `Node` describes one type, like an interface type, a
struct type, etc. See schema.capnp for details.)

CodeGeneratorRequest isn't quite the right thing to use here, because:
- It's specifically designed to provide information needed to generate
source code, which is a superset of what you need at runtime. Schema.Node
is intended to contain specifically the info that's useful at runtime.
- The CodeGeneratorRequest may include a lot of types that aren't actually
used by the specific interface.
- We don't actually embed the full CodeGeneratorReequest inside the
generated code. We embed each Node separately. I wouldn't want to store a
second copy of all that data, and trying to rebuild the CGR from the
components seems a little weird.

Next thing: I don't think the generated code would implement the
getSchema() RPC directly for each type. Instead I think I'd add a new
method to the `Capability::Server` and `ClientHook` like:

virtual kj::Promise getSchema() = 0;

The generated code for each Server type would implement this virtual
method, which would be a one-liner (`return
capnp::Schema::from();`).

It would be the RPC system's responsibility to actually turn these into RPC
calls, and to maintain a capnp::SchemaLoader used to translate

I think the RPC system would have to be responsible for maintaining a
capnp::SchemaLoader containing all schemas that had been loaded from the
remote server, in order to implement the local version of `getSchema()`.

I'm a bit worried about redundantly transmitting the same schema nodes over
the wire many times, especially when dealing with many different interfaces
that implement the same underlying types. I wonder if we need a more
complicated mechanism to deal with this. Maybe the RPC interface would let
the client specify which schemas they already know about. Or maybe this
would be tracked within the RPC system itself on a per-connection basis?
There's also the problem that two different servers could have conflicting
versions of a schema, but could both end up proxying through the same third
server, meaning you might get different versions of the same schema for
different capabilities on the same connection. Maybe it's not actually
feasible to share schemas between capabilities at all? Ugh.

So... yeah, I really want to see this happen, but there's kind of a lot of
design questions to work through. Big project.

-Kenton

On Tue, Sep 15, 2020 at 6:52 AM Ryan Patterson 
wrote:

> I'm in the process of writing an interactive debugger for capnp. It's easy
> enough to manually provide a schema when connecting to another peer, but I
> think it would be good long-term to support dynamic introspection as a
> feature of the RPC system.
>
> I'm quite new to capnp in general, but after reading over the
> documentation, I think the best way to do this is for capnp to dictate a
> standardized optional interface which returns the original
> CodeGeneratorRequest that was used to implement the peer. Ideally, the code
> generator could automatically implement this interface for every generated
> interface, or perhaps the capnp library could automatically implement for
> on Bootstrap and Accept capabilities. When interacting with services which
> don't support introspection (possibly because of space, security, or other
> concerns), the method call simply fails because it refers to an invalid
> interface ID. The important thing is that such an interface has to have a
> fixed ID to be useful, which is why I think it should be established as a
> core part of capnp.
>
> Here is my straw man version of this interface. I've selected AnyPointer
> so that implementing this interface doesn't require pulling the entire
> capnp schema into your own schema. Since the capnp schema is known to both
> parties (even if it's a different version), this is fine.
>
> interface Introspectable {
>   codeGeneratorRequest @0 () -> (cgr :AnyPointer);
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/131b69c1-8782-4935-97d0-6fd11fa31e06n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups

Re: [capnproto] Can't compile 0.8 release with -Wshadow -Werror

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Jeremy,

The `Which` enums are intentionally not `enum class` because I wanted the
enumerants to appear as constants in the containing class, as this enum
actually represents the union tags for the containing class, so the `Which`
part is sort of superfluous. All other enums, though, are declared as `enum
class`. This is all to say, the API was intentionally designed this way.

In any case, we definitely can't change the way `Which` enums are generated
as doing so would break most users of Cap'n Proto (they'd have to update
their code to match), which is unreasonable.

Note that there is no actual problem created by the shadowing here. The
problem the warning is worried about would be for code inside of
`Type::AnyPointer::Unconstrained` -- if it wanted to refer to e.g.
`Type::STRUCT`, it would have to include the `Type::` prefix, as just
`STRUCT` would refer to `Type::AnyPointer::Unconstrained::STRUCT`. But all
code inside the scope of `Type::AnyPointer::Unconstrained` is generated
code anyway, and the code generator will never get it wrong. External
human-written code that uses the API always has to use qualification
anyway, so also won't get it wrong.

Probably the right answer would be for us to add CAPNP_BEGIN_HEADER
and CAPNP_END_HEADER around generated .capnp.h headers. This would disable
all warnings within the header unless you are compiling Cap'n Proto itself.
All of the hand-written capnp header files do this already, but
schema.capnp.h is generated code. If we update the code generator, then all
.capnp.h generated files will disable warnings, but that's probably
reasonable since if you're not working on Cap'n Proto itself, there's
probably nothing you can do about such warnings anyway, since you can't
modify generated code.

Anyway, I'd accept a PR which modifies the code generator to wrap headers
in CAPNP_BEGIN_HEADER/CAPNP_END_HEADER.

-Kenton

On Mon, Sep 14, 2020 at 4:41 PM Jeremy Steward  wrote:

> Hi Kenton / to whom it may concern!
>
> I’m using capnproto and when I try to use the library I seem to be getting
> the following errors from schema.capnp.h. They seem related to the “Which”
> enum with -Wshadow -Werror.
>
>
> In file included from
> ../../../../../../ext/capnproto/c++/src/capnp/dynamic.h:35:
> In file included from
> ../../../../../../ext/capnproto/c++/src/capnp/schema.h:39:
> ../../../../../../ext/capnproto/c++/src/capnp/schema.capnp.h:488:5: error:
> declaration shadows a static data member of 'capnp::schema::Type'
> [-Werror,-Wshadow]
> STRUCT,
> ^
> ../../../../../../ext/capnproto/c++/src/capnp/schema.capnp.h:379:5: note:
> previous declaration is here
> STRUCT,
> ^
> ../../../../../../ext/capnproto/c++/src/capnp/schema.capnp.h:489:5: error:
> declaration shadows a static data member of 'capnp::schema::Type'
> [-Werror,-Wshadow]
> LIST,
> ^
> ../../../../../../ext/capnproto/c++/src/capnp/schema.capnp.h:377:5: note:
> previous declaration is here
> LIST,
> ^
>
> I think this is because these “Which” enums are declared as “enum” and not
> “enum class.” I think this may be unavoidable as long as I’m on the 0.8
> release version (unless I want to make local edits, which frankly I’m not
> excited to do), so I may have to do -Wno-error=shadow. However, I was
> wondering if in future versions it would be possible to fix this so that
> Type::AnyPointer::Unconstrained::Which can be of type “enum class” instead
> of just a pure “enum.” Would there be any problem with this? Am I
> misunderstanding something? It seems like the project uses C++11/14 in a
> number of places already, so this is probably not the biggest deal?
>
> Cheers,
> —
> Jeremy Steward, M.Sc, E.I.T.
> Senior Software Engineer
> Technical Leader - Calibration
> Occipital Inc.
> +1 (303) 815-4240
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/5D0D8A68-3EB3-4590-96D5-04065FE4E160%40occipital.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3Dt34LZoC%2BruRiEH36NPicSWzB0bEu1O%3DHMuvkRfT203g%40mail.gmail.com.

Re: [capnproto] Re: Use with TCP socket, C++ -> pycapnp, no RPC

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Robert,

> message has too many segments error.

This basically means the data is corrupted somehow. You'll need to track
down why the bytes you're receiving are not the same as the bytes you are
sending. To debug, you might try reading the bytes into a regular byte
buffer and logging them, and also log on the sending end, and see what's
different. Then narrow down the issue.

> kj::OutputStream subclass that sends the data via ASIO.

This sounds tricky, since kj::OutputStream does synchronous I/O but ASIO
does async. How are you handling the case where the write can't complete
immediately?

-Kenton

On Thu, Sep 10, 2020 at 9:51 AM Robert Olivier 
wrote:

> I should have been more clear about how I'm reading the stream on the
> python server side .
>
> I consume the 4 byte length header before calling
> message_capnp.Message.read(socket) as I cannot see how to read the payload
> directly and then offer the bytes to capnp in lieu of having capnp read
> directly from the socket. So those header bytes are not accidentally being
> read by message_capnp.Message.read(socket).
>
> On Thursday, September 10, 2020 at 10:26:24 AM UTC-4 Robert Olivier wrote:
>
>> I've scoured the net for a solid example of using capnp purely as a data
>> interchange format sans RPC, streaming over TCP and dont see it. I like the
>> design concept of capnp vs competitors and I want to use it in place of
>> ASN.1 generated code for a project.
>>
>> However, the documentation is just so scattered and unclear that I can't
>> get a basic test structure sucessfully sent to a python process.
>>
>> I've implemented my own simple framing, consisting of a 4 byte length
>> that c++ sends to python before the message. I compute the payload length
>> as computeSerializedSizeInWords() * sizeof(capnp::word) and then send it,
>> using an kj::OutputStream subclass that sends the data via ASIO.
>>
>> My python process reads the length and payload, but when I try to create
>> the message with
>> message_capnp.Message.read(socket) I get a message has too many segments
>> error.
>>
>> I have read that vague bit in the Encoding | Serialization Over a Stream
>> section of the home page docs and I cannot tell if that is saying I need to
>> implement that on both sides of the wire, or if capnp::writeMessage()
>> implements that.
>>
>> Is there some documentation that I'm missing?  Seems strange that such a
>> common use case would be effectively undocumented!  Is use as a pure
>> data-interchange format discouraged?
>>
>> rjo
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/3fe1dde2-5e30-496b-9469-ab00b946113en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnJP7Qf5QUqgi7sZ5CSRzig1EbHO%3DY9KzXVXa59RL85XA%40mail.gmail.com.

Re: [capnproto] Translate RESTFul to capnp

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Mostafa,

Nope, we don't have that yet. We have JsonCodec to translate JSON to Cap'n
Proto, but no one has put it together into an HTTP service just yet.

It's more complicated for Cap'n Proto compared to gRPC because of
capabilities and promise pipelining. There isn't an obvious way to expose
those over HTTP. We could make a version that just doesn't support them but
that risks encouraging people to design interfaces without using these
important tools.

For debugging, another thing I'd like to do someday is have the `capnp`
tool itself support a REPL and scripting mode for sending RPCs to remote
servers... but, that's a big project.

-Kenton

On Sat, Aug 22, 2020 at 7:21 AM mostafa@gmail.com <
mostafa.sedag...@gmail.com> wrote:

> I am wondering if there is any plugin for capnp similar to gRPC-Gateway
>  which translates a
> RESTful HTTP API into gRPC. This is a very useful tools specially for
> debugging purpose.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/94c6552c-2865-4d16-9bb5-58cba0f4404dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm_z0DVyhKTCM_RbOfnLeTwYmKK-QCwZY-KNTftH_bmVg%40mail.gmail.com.

Re: [capnproto] request for simple interface inheritance?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Right, so, what you're asking for isn't necessarily the same thing as what
other people asking for "inheritance" (even "interface inheritance")
typically want.

A lot of people who ask for inheritance specifically want to be able to do
this:

struct interface Bar { ... }
struct Foo implements Bar { ... }
struct Baz implements Bar { ... }

struct Message {
  value @0 :Bar;
  # May contain a Foo or a Baz.
}

I think if we added a way to declare "struct interfaces" in capnp schema
language, people would be confused to find that they can't actually use the
interface type in their schemas.

It sounds like you're fine with that -- all you want is to be able to write
code in whatever programming language which is polymorphic over struct
types that have the right common subset of fields.

I think the right way to solve your problem really depends on the language:

- In dynamic languages, you just use duck typing.
- In C++, you can use templates (which provide compile-time duck typing).
- In languages with implicit interfaces (e.g. Go and Haskell) you can
declare the interface in code and the capnp types will implicitly
implement it.
- In Java, perhaps you need an annotation which you can use to tell the
code generator to please declare the reader/builder as implementing some
interface:

struct Foo $Java.readerExtends("com.example.Bar") { ... }

This would cause the code generator to add the appropriate "extends"
declaration to your reader type; it's up to you to define the Bar interface
separately.

-Kenton

On Fri, Aug 14, 2020 at 1:21 PM Jon Ross  wrote:

>
> I don't quite understand. You would never see a `bar` on the wire, just
> like you can't instantiate a interface/abstract-class in a OO language. You
> can only send concrete types on the wire. You just have the ability to
> downcast to a `bar` at run-time.
>
> In java lingo I'm really just asking for "implment bar" to be added to the
> class defs of the message types, and to define "bar" as a interface
> someplace. I don't think it has much (any?) meaning outside of oo languages.
>
> On Friday, August 14, 2020 at 8:06:09 AM UTC-7 ken...@cloudflare.com
> wrote:
>
>> Hmm, would `Bar` be allowed to be used as a type in schemas? If so, when
>> you receive an arbitrary `Bar` on the wire, how would you figure out which
>> layout to use?
>>
>> If `Bar` is not allowed on the wire then I can imagine how to implement
>> this, but I think a lot of people would be surprised by such a restriction.
>>
>> -Kenton
>>
>> On Thu, Aug 13, 2020 at 9:55 PM Jon Ross  wrote:
>>
>>> > But the big problem with this is that you can't ever add a new field
>>> to Bar
>>>
>>> Yes you can, if it's just an interface and the fields need to exist
>>> individuals in each message. If you have a field in bar call "theThing",
>>> and it's id is 7 in one message, and 14 in another that's fine, as long as
>>> it exists. simple for the compiler to check that all fields in bar exist in
>>> any message that "extends/inherits" it. at runtime it's just a interface
>>> you can slap on any message that uses it. If you want to add a new field to
>>> bar, you have to add it to all the messages individuals, and their ids will
>>> be specific to that message at least that's how I'd do it
>>>
>>> On Monday, July 20, 2020 at 5:00:31 PM UTC-7 ken...@cloudflare.com
>>> wrote:
>>>
 If you have a struct Foo whose fields are a strict superset of some
 other struct Bar (with matching field numbers and everything), then you can
 convert one to the other like:

 Foo::Reader foo = ...;
 Bar::Reader bar = capnp::AnyStruct::Reader(foo).as();

 (Or vice versa.)

 But the big problem with this is that you can't ever add a new field to
 Bar, because it would force you to renumber all the later fields of Foo,
 which would be a backwards-incompatible change.

 In general I instead recommend that Foo should contain a field of type
 Bar. This seems to suit most use cases just fine.

 BTW, note that RPC interfaces in Cap'n Proto do support inheritance.
 It's only structs that don't.

 -Kenton

 On Sun, Jul 19, 2020 at 11:08 PM  wrote:

>
> I understand real inheritance is a hornets nest or complex weirdness,
> and I don't need that.
>
> but could we add a really simple version?
>
> like all the msgs have a common 5 fields in them. make the common
> fields an interface that all my msgs inherit from?
>
> I don't expect to ever instantiate a concrete version of the "base"
> class. I just want to pass the derived msgs to a function that expects the
> base-type.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to capnproto+...@googlegroups.com.
> To view this discussion on the web

Re: [capnproto] request for simple interface inheritance?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hmm, would `Bar` be allowed to be used as a type in schemas? If so, when
you receive an arbitrary `Bar` on the wire, how would you figure out which
layout to use?

If `Bar` is not allowed on the wire then I can imagine how to implement
this, but I think a lot of people would be surprised by such a restriction.

-Kenton

On Thu, Aug 13, 2020 at 9:55 PM Jon Ross  wrote:

> > But the big problem with this is that you can't ever add a new field to
> Bar
>
> Yes you can, if it's just an interface and the fields need to exist
> individuals in each message. If you have a field in bar call "theThing",
> and it's id is 7 in one message, and 14 in another that's fine, as long as
> it exists. simple for the compiler to check that all fields in bar exist in
> any message that "extends/inherits" it. at runtime it's just a interface
> you can slap on any message that uses it. If you want to add a new field to
> bar, you have to add it to all the messages individuals, and their ids will
> be specific to that message at least that's how I'd do it
>
> On Monday, July 20, 2020 at 5:00:31 PM UTC-7 ken...@cloudflare.com wrote:
>
>> If you have a struct Foo whose fields are a strict superset of some other
>> struct Bar (with matching field numbers and everything), then you can
>> convert one to the other like:
>>
>> Foo::Reader foo = ...;
>> Bar::Reader bar = capnp::AnyStruct::Reader(foo).as();
>>
>> (Or vice versa.)
>>
>> But the big problem with this is that you can't ever add a new field to
>> Bar, because it would force you to renumber all the later fields of Foo,
>> which would be a backwards-incompatible change.
>>
>> In general I instead recommend that Foo should contain a field of type
>> Bar. This seems to suit most use cases just fine.
>>
>> BTW, note that RPC interfaces in Cap'n Proto do support inheritance. It's
>> only structs that don't.
>>
>> -Kenton
>>
>> On Sun, Jul 19, 2020 at 11:08 PM  wrote:
>>
>>>
>>> I understand real inheritance is a hornets nest or complex weirdness,
>>> and I don't need that.
>>>
>>> but could we add a really simple version?
>>>
>>> like all the msgs have a common 5 fields in them. make the common fields
>>> an interface that all my msgs inherit from?
>>>
>>> I don't expect to ever instantiate a concrete version of the "base"
>>> class. I just want to pass the derived msgs to a function that expects the
>>> base-type.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/e08ca0bc-b386-4d04-b77d-569be721009do%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/43bf6eef-1856-405d-8b6e-db638f36c769n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkAwKR-s_9WpyMm-ziAZe0miPBokz%2Bfmx_as%3DmXwROGdQ%40mail.gmail.com.

Re: [capnproto] How to compress/use packing with RPC

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

It's true, currently TwoPartyVatNetwork doesn't feature any flags to enable
these. You could write a custom VatNetwork implementation that does it.

I'd be happy to accept a PR adding the features to TwoPartyVatNetwork.

-Kenton

On Thu, Jul 30, 2020 at 7:57 PM  wrote:

> I can't seem to find out how to use either the gzip compression or capnp
> packing in combination with rpc
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/fb902586-06a5-46ff-8535-03b18ce1b789o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk%2BWKE3mkn-EnL3Wjcgp17WmYqnV_wSMf3-oEOm7CSFaQ%40mail.gmail.com.

Re: [capnproto] Easiest way to connect two kj::AsyncIoStream?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hey, sorry for the delayed response, it's been a really busy time.

And TBH I'm not sure I really have an answer.

The Windows async bindings are definitely very very different from the Unix
ones, since Windows uses overlapped I/O and Unix uses readiness I/O and,
well, they work totally differently. KJ tries to abstract above both but it
doesn't surprise me at all if the behavior is slightly different,
especially in error and cancellation cases.

Unfortunately the Windows code doesn't get anywhere near as much usage as
the Unix code. Like, extremely so... Cloudflare Workers handles millions of
HTTP requests per second using KJ on Linux but the only time I ever run the
Windows code is literally in Cap'n Proto's CI tests. So while I can say the
Unix code is extremely battle-tested, the Windows code could easily have
lots of edge case bugs.

If you think something is behaving wrong and can narrow it down to a
self-contained unit test, I could try debugging.

-Kenton

On Thu, Jul 23, 2020 at 10:25 AM Vitali Lovich  wrote:

> The paf.promise.fork I think was copy-paste error trying to follow what
> TwoPartyServer is doing.
>
> So here's something interesting. When I close one of the remote peers to
> KJ (remote peer for stream2 in this example), none of the error handlers
> even run. If I joinPromises then on Windows the shutdown is never
> processed. On Android there's a kj::Disconnected exception raised which is
> a weird platform difference (either the Windows backend has a bug or it's
> just an inevitable OS difference that can't be abstracted properly).
>
> Here's the KJ loop:
>
> kj::Canceler stillRunning;
>
> auto stream1 = ioContext.lowLevelProvider->wrapSocketFd(
> rawSocket, kj::LowLevelAsyncIoProvider::TAKE_OWNERSHIP);
>
> try {
> stillRunning.wrap(listener->accept().then([&](kj::Own&&
> stream2) mutable {
>   kj::Vector> pumped;
>   pumped.add(stream1->pumpTo(stream2)
> .ignoreResult()
> .then(
>   [] { std::cerr << "stream1 finished pumping" << std::endl; },
>   [] (kj::Exception&& e) { std::cerr << "stream1 pumping error" <<
> std::endl; }));
>   pumped.add(stream2->pumpTo(stream1)
> .ignoreResult()
> .then(
>   [] { std::cerr << "stream2 finished pumping" << std::endl; },
>   [] (kj::Exception&& e) { std::cerr << "stream2 pumping error" <<
> std::endl; }));
>   return kj::joinPromises(pumped.releaseAsArray())
> .attach(kj::mv(stream1), kj::mv(stream2));
> })).wait(waitScope);
> } catch (const kj::Exception& e) {
>   std::cerr << "Bridge failed: " << kj::str(e) << std::endl;
> }
>
> My test code (which runs on the main thread whereas the kj code runs in
> its own dedicated thread) connects 2 blocking sockets manually using
> regular blocking BSD sockets to each socket (i.e. the peers for stream1 and
> stream2). I then send 1 MB via the other endpoint of stream2. Then I
> shutdown that socket with SD_BOTH. Then I recv the 1 MB with MSG_WAITALL on
> the other side of stream1. That seems to work OK. Then I try to recv a
> second time on the expectation that one of the pumps would have detected
> some problem somewhere but I run into the situation outlined initially.
>
> Am I doing something wrong/have incorrect expectations on the error
> handlers for the pump?
>
> I haven't yet tried this code on stock Linux/macos.
>
> Appreciate any insights you may have! I'm thinking in the short term I'll
> just use `exclusiveJoin` because I'm thinking there's no world where any of
> the handlers for pumping between two network sockets would ever end up
> getting called.
>
> On Mon, Jul 13, 2020 at 5:01 PM Kenton Varda 
> wrote:
>
>> Hmm I don't understand what you're doing with paf.promise.fork(), it
>> looks like you're merely attaching it to another promise, not actually
>> waiting on it or anything.
>>
>> I am a little concerned about your use of shutdownWrite() / abortRead()
>> in a way that assumes these functions will cause any concurrent write or
>> read to throw an exception. That actually isn't quite the intent here.
>> These methods are intended to inform the *other side* of the connection
>> that the connection has ended; the weren't designed to cancel operations on
>> your own end. I think that the way they are implemented for native streams
>> today might produce the behavior you want mostly by accident.
>>
>> The "proper" thing to do would be to cancel any concurrent operation
>> first by destroying the associated promise. So before calling abortRead(),
>> make sure to destroy any promises that represent a read() operation.
>>
>> Unfortunately, this brings us back to the problem with joinPromises().
>> What you really want is for joinPromises() to fail fast if either branch
>> fails, but otherwise wait for both to finish.
>>
>> I guess one rather-ugly way you could do that today is to create a
>> PromiseAndFulfiller for cancellation purposes. Use
>> joinPromises().exclusiveJoin(paf.promise). Then reject the fulfiller if you
>> catch an exception on ei

Re: [capnproto] Writing to a boost gzip filter stream

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hi Emyr,

You can use capnp::writeMessage() to write to any kind of stream.

https://github.com/capnproto/capnproto/blob/7e2f70f354e2ca3d9de73ddaf974408ddbd98866/c++/src/capnp/serialize.h#L162

This function writes to a `kj::OutputStream`, which is an abstract
interface that you can implement any way you want. It only has a couple
methods:

https://github.com/capnproto/capnproto/blob/7e2f70f354e2ca3d9de73ddaf974408ddbd98866/c++/src/kj/io.h#L77

(capnp::writeMessage() always calls the second version of write(), which
takes an array of byte arrays, to write the entire message all at once.)

In fact, there's even already an implementation of kj::OutputStream that
gzip-compresses the output:

https://github.com/capnproto/capnproto/blob/7e2f70f354e2ca3d9de73ddaf974408ddbd98866/c++/src/kj/compat/gzip.h#L69

So you don't even need to use boost for that.

-Kenton

On Sun, Jul 19, 2020 at 11:08 PM  wrote:

> Hi,
>
> I have an application that is currently using gzipped tab seperated fields
> file. The fields are a mix of strings and ints (it's the lstat info for
> every file in a filesystem). To store the full path in this file I base64
> encode it so that the TSV format doesn't break when paths contain
> unprintable characters. I load this into a clickhouse database for
> reporting. Having to do base64 decoding in clickhouse is a pain and slows
> down the queries.
> I want to switch my lstap collector to output capnp messages to avoid
> having to use base64 encoding. I currently use the boost gzip filter
> streams to do on the fly compression for my tsv lines. I'd like to do the
> same with capnp messages but it seems the only way to write to a file is
> using file descriptors. How can I write messages to a
> boost::io::filtering_ostream ?
>
> Many thanks,
>
> Emyr
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/7b25d572-414e-481a-a9e8-587b6ed522fco%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnG3NSqJG1%3Dhr56%3DRY%3D9N4p1WDTW-bVk1JqqB9UcvwuUg%40mail.gmail.com.

Re: [capnproto] request for simple interface inheritance?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

If you have a struct Foo whose fields are a strict superset of some other
struct Bar (with matching field numbers and everything), then you can
convert one to the other like:

Foo::Reader foo = ...;
Bar::Reader bar = capnp::AnyStruct::Reader(foo).as();

(Or vice versa.)

But the big problem with this is that you can't ever add a new field to
Bar, because it would force you to renumber all the later fields of Foo,
which would be a backwards-incompatible change.

In general I instead recommend that Foo should contain a field of type Bar.
This seems to suit most use cases just fine.

BTW, note that RPC interfaces in Cap'n Proto do support inheritance. It's
only structs that don't.

-Kenton

On Sun, Jul 19, 2020 at 11:08 PM  wrote:

>
> I understand real inheritance is a hornets nest or complex weirdness, and
> I don't need that.
>
> but could we add a really simple version?
>
> like all the msgs have a common 5 fields in them. make the common fields
> an interface that all my msgs inherit from?
>
> I don't expect to ever instantiate a concrete version of the "base" class.
> I just want to pass the derived msgs to a function that expects the
> base-type.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/e08ca0bc-b386-4d04-b77d-569be721009do%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DxDvfpXZ-WKvQ9EK%3DYwE%2BV%2Bu2QYAeLYWK1B-yL4osXAg%40mail.gmail.com.

Re: [capnproto] Integrating KJ with other systems

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Mon, Jul 6, 2020 at 10:54 AM Vitali Lovich  wrote:

> I think a potential middle ground for this might be to have the stream
> give you the fd and a fulfiller. Any I/O operations on the stream then are
> blocked from executing until after the promise to return the FD is
> fulfilled.
>

That's an interesting idea. Or rather, I'd have it return a "BorrowedFd"
object which gives you access to the FD, and whose destructor unblocks the
stream.

That said, it is already the case today that concurrent operations on a
stream in the same direction (e.g. calling write() again when a previous
write() hasn't finished) is undefined behavior. By the same logic as your
argument, we should instead have the second write() block until the first
one finishes. And I would indeed agree that, in theory, that is better
behavior. But, I don't like the amount of complexity that creates for the
stream implementation. If AsyncStreamFd were the only implementation of
AsyncIoStream, then I'd say, fine, we can handle it. But lots of other
places, including apps, implement AsyncIoStream in various ways, and it
would be a big burden for all of them to handle concurrent operations
gracefully. Meanwhile, this behavior would be of very little benefit to the
vast majority of callers; in practice, streams naturally tend to be
accessed sequentially.

So, I prefer to keep the burden on the caller to make sure they don't issue
concurrent operations. And it seems to me that this policy extends
naturally to borrowing the FD.


> I don't think the WebServer would need an FdObserver because you could
> return the FD in blocking mode & the user could dispatch the I/O to a
> background thread & get back onto the executor to fulfill the promise once
> complete. Not like sendfile and non-blocking interact all that well.
>

I don't agree here. I don't think most use cases would call for spawning
background threads to do blocking I/O. splice(), for example, interacts
with non-blocking IO quite nicely. (And whether or not you consider
sendfile() to play nice with non-blocking IO kind of gets into a
philosophical debate of whether the disk should properly be treated as a
remote device vs. "L5 memory". Not everyone agrees on this topic.)

Also, switching the FD to blocking mode would imply that you *cannot* use
this method while having any async reads or writes in-flight, even if you
don't plan to issue any conflicting reads or writes. E.g. what if you just
want the FD to perform an ioctl() on it? Or you want to do manual writes
but use the regular KJ machinery for reads?

-Kenton

>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk3n0BsUGQG_j5Fn2GEX2uT-uYNnAsyOcyBTLs3T8XhMA%40mail.gmail.com.

Re: [capnproto] Easiest way to connect two kj::AsyncIoStream?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Hmm I don't understand what you're doing with paf.promise.fork(), it looks
like you're merely attaching it to another promise, not actually waiting on
it or anything.

I am a little concerned about your use of shutdownWrite() / abortRead() in
a way that assumes these functions will cause any concurrent write or read
to throw an exception. That actually isn't quite the intent here. These
methods are intended to inform the *other side* of the connection that the
connection has ended; the weren't designed to cancel operations on your own
end. I think that the way they are implemented for native streams today
might produce the behavior you want mostly by accident.

The "proper" thing to do would be to cancel any concurrent operation first
by destroying the associated promise. So before calling abortRead(), make
sure to destroy any promises that represent a read() operation.

Unfortunately, this brings us back to the problem with joinPromises(). What
you really want is for joinPromises() to fail fast if either branch fails,
but otherwise wait for both to finish.

I guess one rather-ugly way you could do that today is to create a
PromiseAndFulfiller for cancellation purposes. Use
joinPromises().exclusiveJoin(paf.promise). Then reject the fulfiller if you
catch an exception on either side. But that's pretty ugly.

We need a joinFailfast(), I think...

-Kenton

On Mon, Jul 6, 2020 at 10:13 AM Vitali Lovich  wrote:

>
>
> On Mon, Jul 6, 2020 at 7:41 AM Vitali Lovich  wrote:
>
>>
>> On Mon, Jul 6, 2020 at 7:26 AM Kenton Varda 
>> wrote:
>>
>>> Almost. Two issues I can think of:
>>>
>>> 1. After each pump completes, you probably want to call shutdownWrite()
>>> to propagate the EOF.
>>> 2. `joinPromises()` always waits for all promises to resolve. I've often
>>> found that this turns out to be the wrong behavior when one of the joined
>>> promises throws an exception. Usually you want the other tasks canceled in
>>> that case. I think that might be the case here -- if you get an I/O error
>>> in one direction, you probably want to kill the whole stream. Then again,
>>> probably that'll happen anyway in most cases. (Whereas, EOF is not an
>>> error, so you do want to wait for the other promise in that case.)
>>>
>>
>> So more like this?
>>
>> return stream1.pumpTo(stream2).ignoreResult().then(
>>   [&] {stream2.shutdownWrite()},
>>   [&](kj::Exception&& e){
>> stream1.shutdownWrite();
>> stream1.abortRead();
>> stream2.shutdownWrite();
>> stream2.abortRead();
>>   })).exclusiveJoin(
>> stream2.pumpTo(stream1).ignoreResult().then(
>>   [&] {stream1.shutdownWrite()},
>>   [&](kj::Exception&& e){
>> stream1.shutdownWrite();
>> stream1.abortRead();
>> stream2.shutdownWrite();
>> stream2.abortRead();
>>   }));
>>
> Actually, I think this is two hand-wavy. Also I think the original
> inclusive join is actually correct because I want to ensure that both sides
> finish any I/O that may be in flight. Otherwise I may end the stream
> prematurely just because 1 end finished (e.g. 1 end sends some data & then
> closes because its done - the peer won't receive all the data).
>
> My current code looks something like:
>
> void completelyClose(kj::AsyncIoStream& stream) {
> stream.shutdownWrite();
> stream.abortRead();
> };
>
> kj::Canceler stillRunning;
> auto stream1 = ioContext.lowLevelProvider->wrapSocketFd(
> rawSocket, kj::LowLevelAsyncIoProvider::TAKE_OWNERSHIP);
>
> stillRunning.wrap(listener->accept().then([&](kj::Own&&
> stream2) mutable {
>   auto paf = kj::newPromiseAndFulfiller();
>   auto unsafeStream2 = stream2.get();
>
>   kj::Vector> pumped;
>   pumped.add(stream1->pumpTo(stream2)
> .ignoreResult()
> .then(
>   [stream2 = stream2.get()] { stream2->shutdownWrite(); },
>   [&stream1, stream2 = stream2.get()] (kj::Exception&& e) {
> completelyClose(*stream1);
> completelyClose(*stream2);
>   });
>   pumped.add(unsafeStream2->pumpTo(stream1)
> .ignoreResult()
> .then(
>   [&stream1] { stream1->shutdownWrite(); },
>   [&stream1, stream2 = unsafeStream2] (kj::Exception&& e) {
> completelyClose(*stream1);
>   completelyClose(*stream2);
>   }));
>   return kj::joinPromises(pumped.releaseAsArray())
> .attach(
>   paf.promise.fork(),
>   // AcceptedConnection simply fulfills on destruction.
>   kj::heap(kj::mv(stream2), kj::mv(paf.fulfiller)),
> );
> })).wait(waitScope);
>
> The fulfiller stuff is another place I'm pretty sure I haven't done right.
> I was just going off of what's happening under the hood when you wait on
> the promise that TwoPartyServer returns when it listens.
>
>
>>
>>> On another note, a warning: I'm likely to change the AsyncOutputStream
>>> interface significantly in the future, in order to replace
>>> `shutdownWrite()` with something that returns a promise, and to make it so
>>> that if you don't explicitly shut down a stream

Re: [capnproto] Integrating KJ with other systems

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Mon, Jul 6, 2020 at 10:22 AM Vitali Lovich  wrote:

> I don't feel great about the duplication approach (or allowing retrieving
> the raw handle directly) as it can be subtly tricky to actually use
> correctly. For example, if you dup & make the dup'ed FD blocking, that will
> impact correct behavior within cap'n'proto. So as a user you'd end up
> wanting to always close the other FD after retrieving it. Since all usages
> have you invalidating the stream & consuming the FD, putting it in the
> provider is nice: it's symmetrical, you clearly have ownership of the
> stream, and it's a natural spot to have flags to automatically make the
> socket/FD blocking again.
>

I think there are a lot of use cases where you wouldn't want to destroy the
AsyncIoStream object, though. For example, if I have a web server that's
serving a file from disk, it might call getFd() so that it can use
sendfile() to push bytes out without copying through userspace. But the
stream can be reused for another HTTP request after that, so it can't be
destroyed in the meantime.

Hmm, in fact, for this use case it seems like the web server has to be able
to get the FD *and* the UnixEventPort::FdObserver object together, so that
it can use the latter to observe when the socket is writable. That's a bit
of a problem since FdObserver is currently specific to UnixEventPort; when
using something like the libuv compatibility layer in node-capnp, this
wouldn't work. Maybe the answer is to define an FdObserver abstract
interface. At least the way sockets work is pretty consistent across unix
variants. (On Windows, of course, something entirely different is needed...
but that was already true.)

I don't like requiring an AsyncIoProvider to unwrap because it's generally
uncommon to pass the AsyncIoProvider down into libraries. I think these
kinds of optimizations should be possible to implement without requiring
the library to take whole new, wide interfaces.

-Kenton

>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DOU4FRRcn%2BzA9BQDBZ5FO-dcZiaKj8z1GAUdGsRueHzA%40mail.gmail.com.

Re: [capnproto] Integrating KJ with other systems

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

On Mon, Jul 6, 2020 at 9:28 AM Vitali Lovich  wrote:

>  Yeah, that's what I figured. The KJ API is s much more user friendly
>>> though :). It would be cool to be able to consume the raw file
>>> descriptor/socket out of the pipes the LowlevelIoProvider constructs to
>>> simplify code/error handling.
>>>
>>
>> Yeah, I think adding `kj::Maybe getFd();` (and `kj::Maybe
>> getHandle();` on Windows) to the `AsyncIoStream` interface is probably
>> something we should do. I resisted this for a long time since it breaks the
>> abstraction a bit, but there's just too many places where it would be
>> useful, especially for interoperability and optimizations.
>>
>
> And similarly transfer ownership out by having AsyncIoProvider have an
> unwrap methd that take in an Own and return an OwnFd?
>

I think that could again be a method on the stream itself; doesn't need to
be on AsyncIoProvider.

OTOH, it might not be necessary to support ownership transfer. Instead, you
could call getFd() and then dup() it (or DuplicateHandle() on windows).
Then you can destroy the AsyncIoStream, which will close its copy of the
handle, but you still have the duplicate. But if implementing a
`Maybe releaseFd();` method seems easy then I'm fine with that.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk5tM6XwVyyFED%2BV2fvxH%3Dw%2B-438wmLVG%3D%3DD%2BhcGEupvQ%40mail.gmail.com.

Re: [capnproto] Easiest way to connect two kj::AsyncIoStream?

[&#x27;Kenton Varda&#x27; via Cap'n Proto]

Almost. Two issues I can think of:

1. After each pump completes, you probably want to call shutdownWrite() to
propagate the EOF.
2. `joinPromises()` always waits for all promises to resolve. I've often
found that this turns out to be the wrong behavior when one of the joined
promises throws an exception. Usually you want the other tasks canceled in
that case. I think that might be the case here -- if you get an I/O error
in one direction, you probably want to kill the whole stream. Then again,
probably that'll happen anyway in most cases. (Whereas, EOF is not an
error, so you do want to wait for the other promise in that case.)

On another note, a warning: I'm likely to change the AsyncOutputStream
interface significantly in the future, in order to replace
`shutdownWrite()` with something that returns a promise, and to make it so
that if you don't explicitly shut down a stream, then it's treated as an
error. Currently, AsyncOutputStream's destructor implicitly sends a clean
EOF, but that's the wrong thing to do when the sender terminated
prematurely due to an exception. So, your code will need some updating when
that happens.

-Kenton

On Sun, Jul 5, 2020 at 8:13 PM Vitali Lovich  wrote:

> I was wondering what would be the best way to bridge 2 kj::AsyncIoStreams
> to each other (read to write/write to read) so that they act as
> pass-through? I'm assuming something like:
>
> auto pumped = kj::ArrayBuilder>(2);
> pumped.add(stream1.pumpTo(stream2).ignoreResult());
> pumped.add(stream2.pumpTo(stream1).ignoreResult());
> return kj::joinPromises(pumped.finish()).ignoreResult();
>
> Thanks,
> Vitali
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/CAF8PYMh%3DKr9Yzmz9on4Cxprb0irNOGpV0MUtBxdGitbOgkjiEg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnQT2EeTcZt5XrJU%3Djc%3D%3DHk5mO497d79pVA-Q-F1%2BKiGA%40mail.gmail.com.