Hi Jorge! On Fri, 30 Jun 2023 at 15:47, Jorge Esteban Quilcate Otoya <quilcate.jo...@gmail.com> wrote: > > Thank you both for the replies! A couple more comments: > > The current proposal is to have ‘record.validation.policy’ per topic > (default null). A flag would be something like > ‘record.validation.policy.enable’ (default=false) may be simpler to > configure from the user perspective. > > Also, at the moment, is a bit unclear to me what value the topic config > ‘record.validation.policy’ should contain: is the policy class name? How is > the policy expected to use the name received? >
The 'record.validation.policy' will typically contain a value that is meaningful to the policy implementation. For example, a schema registry might support different strategies to associate a schema with a topic. The policy could use this property to determine which strategy is in use and then evaluate whether the record is valid. We decided to reserve the 'null' value to mean disable validation for this topic to avoid the need for introducing a second inter-dependent boolean property. > > Thanks! I think adding a simple example of a Policy implementation and how > plugin developer may use this hints (and metadata as well) may bring some > clarity to the proposal. > We've added a sample to the KIP, hope this helps. We expect the RecordIntrospectionHints to be a declaration the policy makes, which the implementation of the KIP may use to optimise record iteration avoiding a full decompression in the case where a message is received with compression type matching the topic compression config. Currently Kafka optimizes that case by supplying an iterator that does not provide access to the record data, only answers hasKey/hasValue checks. HTH, best Edo & Adrian
