On Dec 14, 2007, at 3:08, Aaron Stone wrote: Attached is the most current documentation of the binary protocol after Thanks very much for getting this started. I filled in and corrected a bit in here. There's still more than needs to be written, but the hard part seems to have been taken care of. |
Notes regarding the proposed binary protocol from Yahoo's hosted memcached hackathon on 2007-12-13:
This document current as of 10:52 AM on 2007-12-14.
TODO:
- incr/decr response format
- complete TODO
Command Definitions:
CMD_GET = 0
CMD_SET = 1
CMD_ADD = 2
CMD_REPLACE = 3
CMD_DELETE = 4
CMD_INCR = 5
CMD_DECR = 6
CMD_QUIT = 7
CMD_FLUSH = 8
CMD_GETQ = 9
CMD_NOOP = 10
CMD_VERSION = 11
Error Definitions:
ERR_UNKNOWN_CMD = 0x81
ERR_NOT_FOUND = 0x1
ERR_EXISTS = 0x2
Packet Definitions:
Bits run across the top. Period (.) represent the
start of each byte (bit 0 in that byte, so to speak).
The characters -, +, | are used for Ascii art.
Number of bits in each field are in parens.
Numbers on the left side are helpful for counting
four byte word lines in the packet.
All numbers are in network byte order (big endian).
Request header:
+.1234567.1234567.1234567.1234567+
1 |Mag (8) Cmd (8) Key len (16) |
2 |Len (8) Type(8) Reserved (16) |
3 |Total body length (32) |
4 |Opaque key (32) |
+.1234567.1234567.1234567.1234567+
Response header:
+.1234567.1234567.1234567.1234567+
1 |Mag (8) Cmd (8) Stat(8) Resvd(8)|
2 |Len (8) Type(8) Reserved(16) |
3 |Total body length (32) |
4 |Opaque key (32) |
+.1234567.1234567.1234567.1234567+
Brevity:
The short keywords in the header format expand as follows:
Mag:
magic number
Cmd:
command code
Key length:
length in bytes of the text key that follows the command extras.
Stat:
status of the response (non-zero on error).
Resvd:
Reserved for future use
Len:
length in bytes of the command extras.
Type:
reserved for future use (Sean is using this soon).
Reserved:
reserved for future use (up for grabs).
Total body length:
length of extra + key + value
Opaque key:
will be copied back to you in the response.
General format of a packet:
+.1234567.1234567.1234567.1234567+
1 |HEADER (16 bytes, per above) |
2 | |
3 | |
4 | |
+.1234567.1234567.1234567.1234567+
|Command-specific stuff |
| (note length in byte 5) |
+.1234567.1234567.1234567.1234567+
|Key |
| (note length in bytes 3-4) |
| (never present in responses) |
+.1234567.1234567.1234567.1234567+
|Value |
| (note length of bytes 9-12, |
| minus extra length, |
| minus key length) |
+.1234567.1234567.1234567.1234567+
Magic:
* Magic byte / version. For each version of the protocol, we'll use
a different request/reponse value pair. This is useful for protocol
analyzers to know what a packet is in isolation from which direction
it is moving.
* The version should hopefully correspond only to different meanings
of the command byte. In an ideal world, we will not change the
header format. As reserved bytes are given defined meaning, the
protocol version / magic byte values should be incremented.
* Magic byte values will begin at 0x80 and go up from there.
Commands:
get - single key get (no more multi-get; clients should pipeline)
getq - like get, but quiet. that is, no cache miss, return nothing.
Note: you're not guaranteed a response to a getq cache hit until
you send a non-getq command later, which uncorks the
server which bundles up IOs to send to the client in one go.
Note: clients should implement multi-get (still important for
reducing network roundtrips!) as n pipelined requests, the
first n-1 being getq, the last being a regular
get. that way you're guaranteed to get a response, and
you know when the server's done. you can also do the naive
thing and send n pipelined gets, but then you could potentially
get back a lot of "NOT_FOUND!" error code packets.
alternatively, you can send 'n' getqs, followed by an 'echo'
or 'noop' command.
Get Response Packet Format:
+.1234567.1234567.1234567.1234567+
1 |Flags (32) |
2 |CAS ID (64) |
3 | |
+.1234567.1234567.1234567.1234567+
Followed by the value for the key (the length of which is known
due to the general packet format).
delete
When allows you to 'reserve' a key. When 'when' is set
for, say, ten seconds in the future, all 'add's and 'replace's
on that key will fail until ten seconds from now. 'set' will
succeed regardless of any reserved deletes.
+.1234567.1234567.1234567.1234567+
1 |When (32) |
+.1234567.1234567.1234567.1234567+
set/add/replace
cmd-specific fixed-width fields for set/add/replace:
* 4 byte expiration time
* 4 byte flags
(the 4 byte length is inferred from the total body length,
subtracting (keylen + body length))
* 8 byte CAS id.
+.1234567.1234567.1234567.1234567+
1 |Flags (32) |
2 |Expiration (32) |
3 |CAS ID (64) |
4 | |
+.1234567.1234567.1234567.1234567+
If the CAS ID is nonzero, a set will only succeed iff the
value exists and has a CAS identifier equal to the provided
value.
noop
Used as a keep alive. Flushes outstanding getq's.
Should not have any protocol specific gunk.
Must not have a key.
incr/decr
+.1234567.1234567.1234567.1234567+
1 |Amount (64) |
2 | |
3 |Initial value (64) |
4 | |
5 |Expiration (32) |
+.1234567.1234567.1234567.1234567+
These commands will either add or remove the specified
amount to the requested counter.
If the counter does not exist, one of two things may happen:
1) If the expiration value is all one-bits (0xffffffff), the
operation will fail with NOT FOUND.
2) For all other expiration values, the operation will succeed
by seeding the value for this key with the provided initial
value to expire with the provided expiration time.
Note that in the creation case, flags will be set to zero
(FIXME: Should they be provided here as well?)
-- Dustin Sallings |
