Couple of replies inline.
On 06/02/15 13:19, Rafael Schloming wrote:
A couple of questions/comments inline, but first off, any reason this
isn't on the list? (I don't mind, just curious.)
No good reason; Alan PMed me and I replied, then though I'd best copy
you guys. It's slightly head explodingly long, so if conversation
develops it might be best to split it up a bit - it was mainly a bit of
a brain dump for Alan and I got carried away :-D I've moved to the Qpid
user and proton lists now for maximum exposure.
On Fri, Feb 6, 2015 at 6:20 AM, Fraser Adams
<fraser.ad...@blueyonder.co.uk <mailto:fraser.ad...@blueyonder.co.uk>>
wrote:
Hi Again Alan,
Sorry this has turned out to be massively TL;DR I kind of got on a
roll......
On the WebSocket thing it's worth pointing out that I've mainly
been in the game of *using* them and have never had the pleasure
of writing an implementation from scratch, but OTOH I've done a
reasonable amount of"hacking around" - but that's mostly been in
JavaScript and Python implementations so for C things are more
awkward.
I've copied Rafael so he's in the loop I've also copied Gordon,
'cause I'd really quite like WebSocket support in qpidd at some
point and Rob because he implemented the WebSocket transport in
the Java Broker and I suspect he knows way more than me -
especially about the trades around different implementation
choices, he also knows the AMQP WebSocket spec intricacies way
better than me so will be better placed to advise gotchas - and
also I guess an approach that makes is easier for proton-j to
introduce WebSocket support too.
I guess first off I should say that when I suggested it might be a
car crash if not thought through I was perhaps being a little
parochial. As you know the JavaScript binding uses emscripten to
compile proton-c to JavaScript - the way that works is the core
grammar is compiled to LLVM bitcode and ends up as asm.js but
library/system calls either wind up calling other compiled code or
they break out into some pure JavaScript, that's the case for the
network code and basically all the posix network calls get trapped
and "faked" in a useful way; library.js and library_sockfs.js are
the main places to look if you're interested.
I guess that the key point is that all the *socket* calls in
proton-c wind up as *WebSocket* stuff in the compiled JavaScript,
for browsers it uses native WebSockets and for Node.js it uses the
ws WebSocket library, which is I think the most popular one.
The gnarly bit that I can think of is that clearly adding
WebSocket support to proton-c risks accidentally causing the
JavaScript stuff to implement WebSocket over WebSocket, which
would be quite funny but probably not terribly useful :-D
It sounds like we quite probably would want to keep any web socket
stuff compile-time optional and simply not build it at all with
emscripten. That would a) avoid the possibility of nested websocketry,
and b) keep useless code out of the already large proton.js file.
It's worth keeping an open mind. The proton-messenger.js stuff it indeed
a bit big, most of that is actually down to the emscripten runtime stuff
and there's a limit to what is easy to strip, there's some work ongoing
on emscripten to make this a bit better. It's <500K minified and it
could be compressed. I'm all for making it smaller but I doubt it'll
make a huge difference I'm afraid. My main concern would be to make sure
that any bits of code to add an API to select between TCP and WebSockets
*weren't* removed I *want* to be able to pass that info to the
emscripten code for when I get round to adding native TCP support.
Another thing I worry a little about is how one implements the
server side, there are two choices
1. Have a separate WebSocket transport
2. Note that WebSockets are actually a frame based protocol
layered on top of TCP and initiated via an HTTP UPGRADE command.
The spec is here: https://tools.ietf.org/html/rfc6455
The Java Broker (currently) takes the former approach, it's a
sensible and convenient approach because it uses Jetty as the Web
Server for its UI and Jetty comes complete with a WebSocket
implementation, but there's a down side - you have to stand up a
separate port so basically AMQP over TCP connections connect to
one port and AMQP over WebSocket connections connect over a
different port.
But as I say WebSockets are actually layered over TCP so it's
possible if you have more control over the server to look at the
data on the TCP buffers and detect that it's actually WebSocket,
so from a user perspective it would be *really nice* and
technically possible to have a single listen port.
When I suggested implementing it in proton I meant the latter, simply
autodetecting and upgrading. Andrew has done some nice work around
this sort of thing already with detecting SSL vs plain and detecting
SASL vs just core AMQP, so autodetecting the HTTP header and doing the
upgrade would be a natural and hopefully straightforward way to go. At
that point you just configure on the transport whether you want to
allow it or not.
That's cool, I thought that was what you had in mind when you mentioned
Andrew's autodetect stuff but thought I should include Rob and mention
the Java Broker approach to see if there were views on the relative
merits of each.
That's kind of a nice segway into another point
1. Use a library
2. Roll your own/copy paste reuse
I'm torn on that, using a library is clearly good from a support
perspective but the counter to that is illustrated by the point I
make about using Jetty in the Java Broker. Of course you might
*want* to force TCP and WebSocket connections to different ports
but do we care more that it's AMQP or what the transport is
(that's an open question BTW, I personally like the idea of just
connecting to a single port but there may be good counter arguments).
On a slightly library related note it's worth mentioning something
on my TODO list. As you probably know the JavaScript binding is
currently limited to using a WebSocket transport, that makes sense
as a starter for ten 'cause obviously browsers have a number of
constraints but for Node.js there's no real reason for that to be
the case. You might have seen the TCP->WebSocket proxy stuff I
wrote, if you have you'll see that it's actually technically
pretty easy to implement at TCP version using the Node.js built in
net library. The main reasons I've not done it yet are:
1. Time - I only get to code at weekends and I'm juggling a couple
of O/S projects at the moment #ifonlyididntneedtoworktolive :-)
2. The emscripten network code really isn't as modular as it
should be - I've done quite a lot of it, but I didn't originate it
and TBH I'd quite like to pick it apart and make it a bit more
layered, lack of time again.
3. The biggest reason though is actually what approach to take. As
above I'd really like to have a Node.js AMQP application that can
just receive either a TCP or a WebSocket connection and *just
work*. That's a bit awkward because I use ws as the WebSocket
server. As it's so well established it's a bit daft to roll my own
but I don't know, it's probably possible to Monkey Patch or
otherwise expose the native TCP socket if a WebSocket connection
fails, though probably the best approach would be to add a feature
to ws and hope that the pull request gets accepted.
Hopefully point three above illustrates one potentially awkward
pinch-point, you'll be adding WebSocket server code in pure C and
I'll be making the backend of the TCP path all WebSockety too :-)
It *should* work out OK because the emscripten stuff will
establish a listener and ws will have removed all of the
WebSockety stuff and just exposes the AMQP frames in the octet
array that comes off the recv call, so the proton-c code *should*
just interpret that as AMQP off a TCP socket and not send it to
any of the proton-c WebSocket code, but depending on how its done
it could go horribly wrong, which is why I figured I'd give a bit
of background.
All of the above has been from the perspective of a server
receiving a WebSocket connection, in some respects though
establishing a connection is actually more awkward....
So it's not like it's technically hard to write the code to have
Node.js establish an AMQP over TCP connection - the net library is
pretty straightforward. The bit that I'm actually struggling with
is how best to let applications choose between a TCP and WebSocket
transport.
As I say the default position is that the proton-c network code
maps to emscripten library_sockfs.js calls that push the data over
WebSockets - so what API to use to tell it to use TCP :-) It's
easy to set a compile time option but that sucks, I'd like an
application to be able to connect using either TCP or WebSockets.
I was wondering if the getprotoent /etc/protocols stuff might be
the best place, that's certainly where you'd get the protocol
number for UDP, TCP or SCTP to the sockets API but that's used
pretty much for the underlying Operating System networking and
WebSockets currently tend to be implemented as user level
libraries over TCP and I'm not aware of an official protocol entry
for WebSockets.
That's the only place in the socket API that I can think of that's
even close to being able to specify that sort of thing though :-/
There are certainly other ways to pass that sort of info down to
emscripten, but as I say I'm struggling for a clean approach.
So that conundrum is kind of relevant to the native proton-c
implementation too 'cause if you want to allow the ability to
connect using either TCP or WebSockets there needs to be something
in the API that allows users to choose. Is there anything in any
of the AMQP specs that have actually covered this? I know that
Qpid C++ has support for rdma but I *think* that's specified in
the Connection config which offers a slightly richer set of
connection options than the sort of amqp connection available in
Proton (certainly from what I'm aware of with Messenger).
I think this should be fairly straightforward in proton-c. We already
have a similar set of scenarios there with SSL and SASL autodetect.
When you configure a Transport, you basically construct a stack of
protocols. In server mode you can it autodetect whether each layer is
present and fall through to the next. In client mode the stack is just
what you have specified.
So I can see how the autodetect could be extended to handle this when
acting as a server, but this last bit was really about what sort of
"API" one might use for a client to establish a connection giving a
runtime choice between a TCP or WebSocket transport. As I say for the
qpid::messaging API the Connection allows quite a range of options but
in Messenger at any rate I think the options are more limited and ISTR
there are some restrictions even on SASL options, so as I say it's
probably not "technically" hard I'm just not sure what the best "specify
the transport" API might be for Proton - if you see what I mean?
So I've covered some of the thought processes going through my
mind as I've been thinking around WebSockets in native proton-c
and most of them aren't really about the technical implementation
of the WebSocket protocol, but I think are worth bearing in mind.
In terms of actually implementing this stuff as Rafael said I
think that options in C are fairly limited. As well as the library
issues I mentioned earlier there's also the little matter of
worrying about license incompatibilities, so my gut feeling is
that the best approach might to get "inspiration" from other
places and to grasp the nettle and implement something that
integrates nicely with Proton.
If it were me implementing this the first place I'd look would be
here:
https://github.com/kanaka/websockify
WebSockify is actually a TCP->WebSocket proxy and TBH bits of it
irritate me, but itQpid user list now for maximum exposure
seems to be used quite a lot. The most significant things though
is that:
1. It implements a pretty complete WebSocket stack in Python,
which you know so should help you figure things out
2. It actually also has an implementation in C (under the "other"
directory) so you might want to look at that
3. I *think* that it accepts TCP or WebSocket connections on its
listen port and just passes through TCP (I can't swear to that but
it rings a bell)
4. It's well used and well maintained so is likely to be a
reliable starting point
I'd also take a look at ws:
https://github.com/websockets/ws
This is clearly JavaScript, but it's one of the most commonly used
and well maintained WebSocket libraries in any language, so again
as a means to figure out how it actually works it might be a good
place to look.
BTW Rafael mentioned the annoying thing about there being lots of
different WebSocket versions, that's certainly true, but is much
less of an issue now. That stuff was generally a pain as the
WebSocket protocol was evolving, but it has now been standardised
in the IETF RFC 6455 I linked earlier and all of the main browsers
primarily support that. Looking on the websockify page there is a
comment "Starting with websockify 0.5.0, only the HyBi / IETF 6455
WebSocket protocol is supported". Unless there's a compelling
customer reason to support pre-standard versions I don't think
it's unreasonable to only support RFC 6455. It's also worth
bearing in mind that the emscripten based JavaScript binding
requires binary support anyway and I think that's only supported
on the final versions of the spec. so there's probably little
actual value in supporting anything else.
Sorry for the essay, hope it's useful.
I'm off to focus on my other project now - I'm currently trying to
write an implementation of webcl using webgl, cause I need to do
some GPU programming and my platform doesn't have opencl
support.......
Cheers,
Frase
On 04/02/15 19:30, Alan Conway wrote:
On Wed, 2015-02-04 at 17:55 +0000, Fraser Adams wrote:
Hi Alan, very quick placeholder response. I've a couple of
family
commitments so won't be able to give a decent response for
a day or so -
don't want you to think I'm rude by not getting back.
It's a topic that deserves a bit of thought 'cause it
could be a bit of
a car-crash if not thought through.
Thanks for the warning! Andrew Stitcher is making changes to the
transport layer that will affect this (make it easier) so I'm
not moving
on proton till that is done. When I do I will consult
carefully with
yourself and Andrew to avoid crashing any cars.
I may try some short term hackery to tide me over but if I do
it will be
in the dispatch codebase and strictly limited to supporting
dispatch's
JS management console. Longer term proton is the right place
to do it
and I will avail of all advice so I do it right ;)
Frase.
On 04/02/15 14:46, Alan Conway wrote:
Hi Fraser,
I need a websocket listener in dispatch. Rafi points
out (rightly) that
it would be of wider use to do it in proton-c than in
dispatch. Quick
JIRA search reveals the word websocket next to your
name - is that
Java-only? I know nada about websockets, and not much
more about JS if
you remember our prev. conversation ;) Actually on
that - I now realize
that (duh!) the java build does the same
barf-in-the-source-dir thing so
I was razzing you for nothing. git-ignore is my friend.
Hoping to pick your brains for anything useful on
websockets. I'm
guessing my options are
- find a C websockets library (looks like there are
some but none
packaged with popular OS distros (at least the ones
popular with me)
- implement the websocket spec in C. Not sure how big
a job that is yet.
The spec is full of non-normative waffle so I'm not
sure just how much
real action there is.
- grab a python ws library and just stick with
dispatch since it has a
nice C+python implementation. That wouldn't help
proton though.
Any thoughts?
Cheers
Alan.
