Hello Danila, Since version 0.10.0.0 Kafka has introduce a timestamp field on the per-message granularity and modified its log retention policy to be based on the timestamps (for details I'd recommend reading KIP-31: https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message), maybe this will fit your needs.
Currently there is no mechanism for customizable compaction algorithm. If you are running an older version and cannot upgrade to 0.10.0.0 or newer anytime soon, one thing I can think of is to read the raw messages and filter them based on the timestamp in the payload, and write to a new topic. For the raw message topic, you can then set a time-based log retention period sufficient for the filtering consumer to read the messages. Guozhang On Thu, Oct 20, 2016 at 9:46 AM, Danila Ermakov <danila.erma...@db.com> wrote: > Classification: Public > Hi > > I’d like to messages to be removed from log as part of compaction based on > message payload internal attribute. I.e. I have some individual expiration > date for each message and I’m not interested in keeping it afterwards. Is > there any way to inject custom compaction algorithm? > > > > Regards > Danila Ermakov > > > > --- > Die Europäische Kommission hat unter http://ec.europa.eu/consumers/odr/ > eine Europäische Online-Streitbeilegungsplattform (OS-Plattform) > errichtet. Die OS-Plattform kann ein Verbraucher für die außergerichtliche > Beilegung einer Streitigkeit aus Online-Verträgen mit einem in der EU > niedergelassenen Unternehmen nutzen. > > Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der > EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche > Bank finden Sie unter https://www.deutsche-bank.de/Pflichtangaben. Diese > E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. > Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich > erhalten haben, informieren Sie bitte sofort den Absender und vernichten > Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe > dieser E-Mail ist nicht gestattet. > > The European Commission has established a European online dispute > resolution platform (OS platform) under http://ec.europa.eu/consumers/odr/. > The OS platform can be used by a consumer for the extra-judicial settlement > of a dispute of online contracts with a provider established in the EU > companies. > > Please refer to https://www.db.com/disclosures for information (including > mandatory corporate particulars) on selected Deutsche Bank branches and > group companies registered or incorporated in the European Union. This > e-mail may contain confidential and/or privileged information. If you are > not the intended recipient (or have received this e-mail in error) please > notify the sender immediately and delete this e-mail. Any unauthorized > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. > -- -- Guozhang
