This was discovered due to testing session expiration per a prior Zookeeper
users thread. Upon deeper investigation it seems that the second client is
unable to connect to Zookeeper because it is supplying the wrong password. This
results in the connection being dropped and the client getting a
EXPIRED_SESSION_STATE, when its not actually expired, its a bad password.
I've changed the subject here to ensure this gets more attention for the bug it
likely is.
I dropped additional debug statements into the C API to echo the password out,
and the password it gets is not what Python sees. With one password the C
binding got back, the Python API had *no password at all*, or it has less than
16 bytes, which is clearly a bug as the password is always 16 bytes.
I tweaked the C layer to output this after connection:
LOG_INFO(("Password is: %02x", zh->client_id.passwd));
Granted, I should probably use a better print modifier, I'm a C newb
unfortunately. This is what I see in my logs during the problem:
ZooKeeper: INFO: check_events@1747: session establishment complete on server
[127.0.0.1:20000], sessionId=0x139507b99fe00c6, negotiated timeout=10000
ZooKeeper: INFO: check_events@1748: Password is: 2502978
kazoo.testing: DEBUG: Password is:
Note, yes, there should be a password from the kazoo.testing line, its where
I'm spitting out the hex password in Python. Problem is, there *is no password*
available from Python. The Zookeeper server generates 16 random bytes for the
password, and when it gets to Python... there's nothing.
This doesn't happen every time, which has me thinking that there is something
in the C binding that is resulting in the mistranslation of the password when
it contains certain characters getting truncated. I've noticed that the amount
of characters in the Python one when the test fails is between 0-15 characters
in length. Not the 16 it should be.
At this point, I don't know exactly where the bytes are being dropped. Digging
through 2 layers is also very discouraging. Having a pure Python API to talk to
Zookeeper has suddenly become much more attractive.
Cheers,
Ben
