>> There is one added complication in that the protocol is a datagram
>> protocol at a
>> higher level (although it uses TCP). I am concerned that the whole
>> protocol could
>> block if there is not enough data to encrypt a whole outgoing message
>> but the peer cannot
>> continue until it gets the message.
If you mean that the upper layer protocol is message-oriented rather
than stream-oriented ('datagram' is a Rorschach blot for me that says:
UDP sorry) and the protocol is constructed such that outgoing
message REQ(A) must have produced [a complete] answer message ANS(A)
before the next outgoing message REQ(B) is sent over the wire, then
you're in fancy land anyway, as that is not a class 101 scenario for
TCP, which is by design stream-oriented.
At the TCP level and given such a 'message interdependency'
requirement, which is the extreme form of what I'm assuming you mean
with the mention of 'datagram protocol', you'll need to ensure both
sender, receiver (and any intermediaries) have their TX (transmit) and
RX (receive) buffers flushed entirely before the next 'message'
exchange (REQ(B)->ANS(B)) can take place.
To get a glimmer of what must be done then (and which might be needed
in your case, when your protocol is less extreme and consequently can
- and will - not wait for response messages for previously sent
messages before the next message goes out) think about old-fashioned
telnet: a keypress is way smaller than a TCP packet can be, so telnet
needed a way to push that ENTER keypress out the door and pronto, so
you, the user, would get some 'interactivity' on your console. The
TCP_NONAGLE socket flag has been introduced to service this need way
back when: given a short timeout, tiny buffer fills are flushed into a
TX packet anyway. The receiver will be able to fetch any byte length
data it actually receives, so when we entice the sender into
transmitting even the small chunks, we're good to go there.
> It is presumed that every SSL_write() requires a flush (at TCP level this
> mechanism is called a "Push"). This basically means the data needs to flush
> to the reading API at the far end on exactly the byte boundary (or more)
> data than you sent. This mean you have a guarantee to not starve the
> receiving side of data that the sending API has sent/committed. This is
> true at both the TCP and SSL levels.
>
> If you think about it the SSL level could not make the guarantee easily if
> the lower level did not also provide that guarantee.
^^^^ the guarantee at the lower level is NONAGLE, which is /not/ the
default in TCP stacks as it can result in suboptimal network usage by
transmitting overly small packets on the wire.
I haven't done this sort of thing with SSL on top for a few years now,
but from what I hear in this thread SSL_write(len := 1) will pad such
data while crypting on a per-write invocation basis (note those last
few words!) and thus write a full SSL packet into the TX side of the
socket for each write into the TX pipeline (I may be Dutch, but I live
by the German rule: "Vertrauen ist gut, Kontrolle ist besser", and you
should too: trust is good, but making dang sure is so much better ;-)
)
Also, there's the purely emotional and very unchecked goblin at the
back of my brain who mumbles: "oh yeah? no buffering incoming
plaintext on the TX side so the SSL layer doesn't get to do a lot of
otherwise probably superfluous work when the write chain is abused by
the application layer by writing tiny chunks all the time?" Don't take
my goblin for his word, he's a definite a-hole sometimes ;-) , but it
won't hurt to make sure the 'non-buffering' flush/push aspect of your
write-side BIO chain is guaranteed. Does the OpenSSL documentation
explicitly mention this behaviour? should be the authoritative answer
there.
>From my work with BIOs, I seem to recall the SSL BIO encapsulates
SSL_write et al (or was it vice-versa? Heck, that's what I get when
doing this off the top of my head while not having used SSL for the
last half year), so the findings for one expand to the other.
Injecting other BIOs in your chain (base64, etc.) will impact this 'is
all data flushed throughout == non-buffering TX behaviour' aspect.
Anyway, using NONAGLE (telnet is **NO**nagle, default socket using
applications use the default(!) NAGLE) on the TX side should, assuming
the SSL/BIO chain flushes as well, ensure your outgoing REQ(n) gets
out the door and on the wire. Which leaves the receiver side: as the
transmitter can only 'flush' like that with SSL in the chain when the
flush is on whole SSL message boundary only (indeed resulting in some
SSL 'packets' ending up containing only a single (encrypted) content
byte as a principle), so the receiver should be okay in depleting its
RX buffers as well as the SSL layer there can, theoretically, process
every byte received, thus spitting out the content (plaintext) bytes
to the very last one which was pushed on the wire.
Hence your application layer can then get a the complete incoming
message before it sends a response, without any extra work on the RX
side.
(For those who wonder: HTTP can also be considered a message protocol:
REQ(GET url) gives you ANS(data) in the response. The basic way to
delineate 'message' here while riding on top of TCP (which is /stream/
by design) is to close the TCP stream after each message. (Client
sends message, does half-close, server receives data, the half-close
back there flushes all REQ() bytes out so the server can get it all
and construct the response, which is transmitted and the connection is
now fully-closed by the server.)
Only when you do a few extra things can you keep the connection (HTTP
persistent connections) open, and sending along the number of bytes as
a message parameter is not all of it, so having a look at how
persistent-connection supporting, well-written, HTTPS clients and
servers implement this might be handy. SSL-based telnet clients are
simpler and carry the same technology (read: tweaks) as they need the
same sort of behaviour.)
Note that I have met quite a few applications in the past which used
SSL+TCP for message protocols like these and the argument always was
"what are you complaining about? it works, doesn't it?" and, yes, it
often works without the details. And what's the bother when you can
reboot your machine when things lock up once in a while; just when it
doesn't happen all to often on your watch, hey? The trouble at TCP
level which hides the issue is the timeouts: even without the NONAGLE,
TCP stacks have timeouts ticking away, which will tickle the kernel
into sending those little chunks remaining in the TX buffers anyway
after the timeout expires, even when such a tiny chunk doesn't fill an
MTU up i.e. produces a suboptimally small TCP packet).
The only thing noticable then is the slight delays; in the end they
limit your throughput bandwidth at application level way below the
level attainable by the hardware (network) at hand.
The other noticable issue is that sometimes such apps lock up and it
requires either kill or reboot or at least severing the TCP connection
to 'recover'.
So far message-based traffic over network stream protocols. W Richard
Stevens (R.I.P.) surely explained it way better than I do, alas.
About the ever-recurring WANT_READ/WANT_WRITE stuff, heck, folks might
have a separate high-bandwidth mailing list for that one alone, if
only we collectively knew what the heck we're talking about, here's a
tip for ya to help you detect WANT_READ/WRITE misbehavin' in your
client and/or server code:
since you can edit the code, add extra code which randomly interjects
SSL renegotiation requests (check the OpenSSL API; there's a very
simple call for that) while you have the SSL connection open. The
random renegotiation action will trigger SSL into requesting and
transmitting all sorts of stuff under the hood you don't need to
concern yourself with, but as a consequence it will trigger quite a
few WANT_WRITE at read-side and WANT_WRITE at write side, thus
increasing the number of occurrences of these response codes and hence
giving you the opportunity to detect read/write processing issues that
much faster. At plaintext / application side, it does not impact the
number of bytes you get out of the SSL read side, so no harm done
there, while you forcibly kick the SSL engine into extra work which
will trigger all sorts of otherwise 'spuriously occurring' issues
regarding return codes and your implementation.
Be sure to add such a (semi-)random renegotiation injector at both
client and server side; I have found that many applications are
incorrectly constructed and a couple of hours of this in a test bed
will almost certainly blow them to kingdom come.
Better now than once you've entered production, I'd say. ;-)
And having a gander at the actual network traffic generated by your
app and (!) checking the timing of the outgoing messages (to detect
unexpected timeouts kicking in and causing TX after all) won't hurt:
Wireshark or other packet sniffers are an asset.
(For when you pay attention to detail: note that the TCP-level NONAGLE
behaviour still is timeout based, which often is okay as the timeout
is relatively small, but if you have an extreme case where messages
must be flushed /immediately/ onto the wire while you're using a TCP
stream (so no timeout whatsoever), then you enter the non-portable
zone of IP stack cajoling.
This is the ultimate price you pay for [ab]using a stream protocol for
message-based I/O. (And that's a generic remark, as it always applies
for any such transformation, not just with TCP or SSL/TCP.)
This would mean that you're essentially forcibly 'shaping' your TCP
packets from the application level, which is very much against design
and severely frowned upon and not supported by some IP stacks at all
(we're not talking about in-spec stuff anymore here, after all), but
I've had one occurrence where such was required (and, yes, such is a
99.9% sure indicator someone screwed up in the design elsewhere). If
such is ever the case, I would not spend the effort but advise to use
DTLS instead (which is UDP-based secure comms and thus thinking
'message' all the way through to lowest hardware layer). This is
particularly important when your message protocol comes with large
throughput requirements and a protocol design where request
transmissions can halt until a certain requests response message has
been complete received and processed by the requestor (a design, which
would suffer from round-trip network delay at such intersections no
matter what you do, BTW). You may note that is one reason why
networked game engine communication is often UDP/DTLS based, to name
one example. Fortunately I notice a lot of work is being done on
OpenSSL DTLS, so this might be a viable option for you today.
I've reiterated (and maybe garbled) some material already mentioned by
others here today, but I felt it was needed for a complete picture.
The attention needed for blocking versus non-blocking I/O is very
valid and quite necessary IMO, and so is strong mention of the
attention required to do good towards WANT_READ/WANT_WRITE
implementation details, but I also find rare the explicit mention of
the perils (both on a theoretical and practical basis) of 'making' a
message-based [application-level] protocol work over a stream-based
lower level protocol, such as TCP, as streams are legally allowed (and
should do so to optimize network usage) to buffer TX pipelines along
the way, resulting in a probable inability to deliver messages at the
other end as a whole, before further data is entered in the same
direction. (Read 'probable' as 'incidental': that's what makes the
mistake very easy to execute and hard to diagnose, especially in
environments where TTM (time To Market) is essential -- and aren't we
all part in that game? Unawareness in your reviewing peers permits one
to slip such mistakes through the net unnoticed. Network programming
never was easy and if it is now, I must have missed that newsflash on
my MTV. ;-) )
--
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: g...@hobbelt.com
mobile: +31-6-11 120 978
--------------------------------------------------
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users@openssl.org
Automated List Manager majord...@openssl.org
