Re: D3845: worker: support more return types in posix worker
> Yuya, it had passed tests for me with cbor, so is that a portability issue?
I don't think it's a portability issue. `cbor.load(StringIO(''))` doesn't
raise EOFError.
> One of the benefits of pickle/marshal is that we don't lose information,
> like when tuples become lists. That would be an insidious problem for callers.
Good point.
> Also, in case it wasn't obvious, we'll need another patch to add some
> handling of len(dumps(result)) > PIPE_BUF (which was an existing issue).
Yup. A payload was much smaller than PIPE_BUF, but that's no longer true.
The most straightforward fix will be to create pipe per worker and select()
them. I don't know if there's a simpler mechanism. DGRAM socket can ensure
message boundaries, but it's still has a size limit.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: D3845: worker: support more return types in posix worker
> +while True: > +try: > +yield cbor.load(fp) > +except EOFError: > +break Unfortunately this doesn't work because the cbor decoder doesn't care for EOF. It tries to raise CBORDEcodeError and fail at `fp.tell()`. We'll have to either fix the upstream cbor library or duplicate some parts to cborutil. (or add an extra length field to feed a single chunk to `cbor.loads()`.) This makes me feel that pickle is "okay" tool. @durin42, any idea? ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: D3845: worker: support more return types in posix worker
> > > I'm not in love with pickle. Could we use json or cbor instead? > > > > Perhaps cbor is better since it can be streamed and the overhead is > pretty > > low. We have to keep the message size small since rfd/wfd is a > multi-writer > > pipe. > > It's been recommended to me that we avoid the streaming flavor of > cbor, so we'd probably just do one-shot messages. I meant multiple one-shot messages can be serialized over the pipe. JSON parser doesn't work in that way. Each message must be written atomically. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: D3845: worker: support more return types in posix worker
> I'm not in love with pickle. Could we use json or cbor instead? Perhaps cbor is better since it can be streamed and the overhead is pretty low. We have to keep the message size small since rfd/wfd is a multi-writer pipe. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
