Yo all,
The original plan for today was to pop on the list and yell "Hey! Have
some -rc's to test! Go nuts! Thanks for all the patches!" - but my
awesome fever for the last few days has pushed that off by ... a few
more days.
But it's totally cool, because it's happening. For the first time in a
long time memcached will act like it's releasing early, often. Thanks to
everyone for all of the recent code submissions and bugfixes :)
So we have two -rc's which are coming up as soon as I can wrap a few bug
reports and merge all this code:
1.3.0-rc (the binary tree, in its current state)
1.2.5-rc (non-binary branch with all of the recent fixes and patches)
There might not be an immediate merge of 1.2.5 -> 1.3.0, but I'm not
convinced that matters immediately.
I believe should probably give the binary tree a couple of real
development releases until we've had a few client implementations. So in
the case of the protocol having to change due to issues with actual
client integration, we won't be backing out on a stable API.
SO! I'd like to lay out my own wishlist for memcached, some of which is
underway, some not:
- Binary protocol. It's been too long, lets finish it. We need more
native tests, and we need to benchmark its speed relative to the text
protocol, make sure we're actually coming out ahead here.
- Dynamic storage engines. Toru has a promising proof of concept that
I'd love to merge for an eventual 1.4 release.
Most of the memcached forks have been due to folks not liking the
storage method. While I disagree with how a lot of these forks are done,
there's definitely merit in the flexibility. A lot of these folks start
from scratch, but I believe memcached should be a simple, efficient,
network platform for dead simple key=value datastores, no matter how
they're actually implemented.
- Dedupe the error messages. We have 11 instances of "SERVER_ERROR out
of memory". This makes it impossible to even guess what happened, even
with full context. Did we fail to allocate a msghdr? Did do_item_alloc
fail? Who knows. We can do this for 1.2.5 if there're people on board.
I'll float the patches.
- Add stats about failures. I'd love counters for all of the times
something failed without causing an eviction. Sometimes machines will be
hosed for a long time before we realize the hit rate is way down due to
the OS refusing 50% of the mallocs. Hard to tell if you've hit maxconns,
blah blah blah. More easy work, maybe for 1.2.5.
- Method of dumping refcounts of "locked" items. I'm tracking down what
I suspect is a refcount bug, and I expect us to have more of these in
the future. It'll be great to be able to verify these offhand. If
someone isolates a bum server, they can dump slabs, refcount, etc, and
send to the list.
- Full item dump. I guess we should do this with the appropriate
caveats. There's a potential patch floating around, we'll see about it.
- Fixing a lot of dynamic memory buffers. Per connection, per freelist,
whatever. A lot of these can be turned into linked lists if, say, the
'item' struct had a 'next' pointer. This should remove most of the
various realloc logic.
- Clean up the network iov's to work like above, and also manage generic
freelisted buffers. Which would remove the need for the specialized cas
suffix buffer. These should be wrapped so they can be linklisted,
instead of managed via dynamic arrays per connection. Could also be used
with the binary protocol to reduce syscalls.
- TCP_CORK/TCP_NOPUSH where appropriate. I totally agree here, and it
used to explicitly do this.
- New slabber engine. I view this as a bonus to making the slabber code
dynamic. We can start cooking an experimental new slabber to do
slab-reassignment, fragmentation avoidance, etc. Without really hosing
mainline.
- More tests. We need better test coverage. Please submit tests!
- "Standard benchmarks". Memcached's contributions have historically
been based on "we ran this in production, here're some tests and proof
it works fine". Recently it's been more about rapid cleanups and the new
storage engine independence. It'd be really great to have some (say
libmemcached based) tests to exercize (under realistic AND unrealistic
measures!) everything.
- More documentation. I figure eventually someone will write docs with
better english, but there're tons of ideas floating around. We should
get them all down. Along with standard troubleshooting info, of which
there is practically none!
- Cache::Memcached updates. Uh, whoops. This doesn't support any of the
recent features.
---
On community related measures:
- Buildbot. I totally started on this, but have not completed it. I want
to offer a buildbot network to the developer community. If you want a
platform supported, provide a slave. Otherwise we will not target it
(besides a couple of standby's).
- GIT. SVN is dead. We'll need to move on to git or hg in order to scale
the developer pace. It's my understanding that just as many folks on the
list prefer GIT as they do Mercurial, and SVN is mainly a convenience.
Anything will be better. I could crack out weekly releases if I didn't
have to worry about SVN branches/merges.
- Add more committers. I promise you guys; we're in an awkward position
with commit access, but people at sixapart absolutely acknowledge and
care about it. I've been having meetings, discussions, etc, about
dealing with this sooner than later. I mostly help with memcached on my
own time. However, sixapart does love memcached and the rest of the
danga projects. I can't say anything definitive, of course, but I feel
like I can share the feeling of "we give a shit" with all of you folks.
Part of having more committers means we need more dedicated reviewers.
Memcached is a stable project trusted by many, many websites.
- We should do more, less formal, hackathons. These are great for
hashing out minor issues, and I feel with more frequence they'd also be
really good for cranking out code, too ;)
---
That's all for now.
I'm missing a number of line items (please don't be offended if I missed
mentioning your submissions!) If it's in my inbox, it'll get a fair
review and probably go in.
have fun,
-Dormando
