I want to do something crazy. Someone talk me off the ledge:
record Bundle {
string key;
array<bytes> msgs;
}
Producers individually serialize a bunch of messages that share a key, then
serialize a bundle and post to a topic.
A generic Flattener service is configured by startup parameters to listen
to 1...n kafka topics containing bundles, then blindly forward the bundled
messages on to configured output topics one at a time. (Blindly meaning it
takes the bytes from the array and puts them on the wire.)
Use case:
I have services that respond to small operations (update record, delete
record, etc). At times, I want batches of ops that are gauranteed not to
be interleaved with other ops for the same key.
To accomplish this, my thought was to position a Flattener in front of each
of the services in question. Normal, one-off commands get stored in 1-item
bundles, true batchs are bundled into bigger ones.
I don't use a specific field type for the inner messages, because I'd like
to be able to re-use Flattener all over the place
Does this make any sense at all? Potential drawbacks?
With multiple producers to a topic, is there another way to publish a
series of messages that gaurantees they won't be interleaved? (It seems
like that goes again the whole spirit of Kafka.)
Thanks!
-John
