Re: The idea of "composite key" to make log compaction more flexible - question / proposal

2017-10-09 Thread Michal Michalski
Hey Jay, Thanks for reply. Yes, this should do the job. We were thinking about something that's abstracting away this logic from user (e.g. the same way Cassandra handles its PK definitions in CQL - "hiding" the row key and optional clustering key behind the concept of "primary key"), but introdu

Re: The idea of "composite key" to make log compaction more flexible - question / proposal

2017-10-05 Thread Jay Kreps
I think you can do this now by using a custom partitioner, no? https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/producer/Partitioner.html -Jay On Mon, Oct 2, 2017 at 6:29 AM Michal Michalski wrote: > Hi, > > TL;DR: I'd love to be able to make log compaction more "granular" than j

The idea of "composite key" to make log compaction more flexible - question / proposal

2017-10-02 Thread Michal Michalski
Hi, TL;DR: I'd love to be able to make log compaction more "granular" than just per-partition-key, so I was thinking about the concept of a "composite key", where partitioning logic is using one part of the key, while compaction uses the whole key - is this something desirable / doable / worth a K