You are NOT able to reproduce? I misunderstood your previous comment.
Further testing suggests (to me) that this is a timing issue, where HAProxy
does not discover that the connection is established, if connection
establishment doesn't happen within a very, very short window after the
connection is attempted.
Previously, I only tested "client talks first" (http) using a different
machine as the server.
Consider the following new results:
server talks first (ssh) - connection to local machine - *works*
server talks first (ssh) - connection to a different machine on same LAN -
*works*
server talks first (ssh) - connection to a different machine across
Internet - *works*
client talks first (http) - connection to local machine - *works*
client talks first (http) - connection to a different machine on same
LAN - *does
not work*
client talks first (http) - connection to a different machine across
Internet - *does not work*
The difference here seems to be the timing of the connection establishment,
and the presence or absence of additional events. (Note that when I say
"local machine" I do not mean 127.0.0.1; I am still using the local
machine's Ethernet IP when talking to services on the local machine.)
When you are testing, are you using a remote machine, so that there is a
brief delay in connection establishment? If not, this may explain why you
do not see the same behavior, since local connections do not appear to have
the same problem.
Most interesting, based on my "timing" theory, I found a workaround, which
seems very wrong in principle; so wrong, in fact, that I can't believe I
tried it; however, using the following tactic, I am able to make an
outgoing socket connection to a different machine, when client talks first.
local sock = core.tcp();
sock:settimeout(3);
local written = sock:send("GET
/latest/meta-data/placement/availability-zone HTTP/1.0\r\nHost:
169.254.169.254\r\n\r\n");
local connected, con_err = sock:connect("169.254.169.254",80);
...
This strange code works. I hope you will agree that writing to the socket
before connecting seems very wrong, and I was surprised to find that this
code works successfully when connecting to a different machine --
presumably because I'm pre-loading the outbound buffer, so the server's
response to my request actually triggers an event that does not occur in a
condition where the client talks first and when there is a delay in
connection establishment, even a very brief delay.
