Alright, I've started looking at the code. Would anyone object if I first started working on abstracting the protocol handling routines so that they are handled by callback instead of the current if/else setup?
Then I was going to put all the protocol stuff in prot_ascii.h|c and prot_binary.h|c respectively. This way we could potentially have hundreds of protocols handled without having a lengthy if/else comparison. On Wed, Feb 20, 2008 at 10:06 AM, Clint Webb <[EMAIL PROTECTED]> wrote: > I wasn't expecting such a quick reply. Good point about allowing multiple > protocols. I might pull out some of my old code and see how easy it is to > drop in. > > I thought I'd give a little background on myself and this protocol style. > I used to work in the controls and automation industry. If you've ever > checked your luggage into an airport, or sent anything thru USPS, UPS or > FedEX, or bought anything online from amazon, b&n and other large online > stores, or from Walmart, kmart, target, etc... then your product has likely > had some experience with my coding somewhere along the line. > > I developed this protocol for a tiny little side-project for one of the > above mentioned companies. It was basically a connector that took > information from a large number of different systems and passed it to > another. The requirements changed a lot, so I developed something that > could be fast, but had to be flexible. If I added a feature to the server, > I didn't want to be forced to update all the clients as well. Also, some > of the clients were tiny little embedded controllers, so it had to be pretty > simple too. > > This solution was VERY fast, as all the commands are a single byte and > could be easily mapped to an array of callback routines. This protocol > also had to run on a real-time system also, so we had to ensure that all > operations preformed in a predictable fashion. > > I seperated the commands by their parameter type. 0-31 had no parameter. > 32-63 had a single byte parameter. 64-95 had a 32-bit parameter. 96-127 > had a single-byte paramter which was the length of a stream of data to > follow (short-string). 128-159 had a 32-bit integer that was the length of > a stream of data to follow. This was our 5 different parameter types. A > command could only have one parameter. > > This way, the IO handling code could retrieve all the necessary data and > then pass it off to a callback routine for that command. > > Each socket connection would have a set of buffers set aside for the > incoming data. In this case we would want a buffer set aside to hold the > key and value data. > > To speed up processing and ensure that the minimum data set has been > provided, we used a 32-bit (or was it 64-bit?) word as flags. Each > operation would set or clear a flag(s). So when a GO command is received, > it can quickly determine what 'action' needs to take place, and which > 'parameters' have been provided. > > If we ran out of room having to handle more than 256 commands, we would > use a command of 0xFF which would expect that the next byte would be another > command (from a set different to the first). I never actually implemented > it though. The most commands I ever used was about 100 or so. > > I cant imagine that a variable-length structured protocol could be much > faster than that. Still the emphasis of this protocol is not so much on > speed, but on flexibility to add functionality to the protocol (by adding > commands) without breaking existing clients (and without having to handle > multiple versions of the protocol). > > The 'noreply' stuff that I have seen around the list could probably > benefit from this protocol. I haven't looked close enough at the CAS stuff > either, but I suspect that would be easy to implement too. > > Also, those that want to shave off a few extra bytes in their client, have > the option of sending a request that only includes the bits they want. If > you care about expiry leave it out, same with flags, tags, cas id's, and > anything else. Plus you can stream-line some of your requests by not using > the CLEAR command, and re-using the state. > > Dang, if I had a little more time on my hands right now, I'd be really > tempted to implement it. I don't actually have a *need* for this protocol > in memcached, it was purely an intellectual itch ever since I saw people > complaining about the existing protocols being difficult to expand. > > > > On Feb 20, 2008 4:14 PM, Dustin Sallings <[EMAIL PROTECTED]> wrote: > > > > > On Feb 19, 2008, at 22:20, Clint Webb wrote: > > > > > I know a considerable amount of thought and work has gone into the > > > existing flavour of the binary protocol, and I dont expect that work > > > to be discarded, I'm really only mentioning this new concept now as > > > an alternative for the future if we ever find the current binary > > > protocol to be too restrictive and inflexible. And something to > > > think about, or even use elsewhere. > > > > The is certainly interesting. The first step of doing the binary > > protocol implementation was to create a change that allowed multiple > > protocols to coexist. It would be possible to implement this to run > > in parallel with the existing protocols in the 1.3 codebase. > > > > Intuitively, it doesn't seem as efficient to process as what > > we've > > got now, but I like being proven wrong, so I'd welcome another front- > > end protocol. :) > > > > Of course, I wrote the majority of the current binary protocol > > code > > about six months ago, so I'd really like to at least have one in more > > people's hands. > > > > -- > > Dustin Sallings > > > > > > > -- > "Be excellent to each other" > -- "Be excellent to each other"
