[jira] [Created] (KAFKA-13926) Proposal to have "HasField" predicate for kafka connect
Kumud Kumar Srivatsava Tirupati created KAFKA-13926: --- Summary: Proposal to have "HasField" predicate for kafka connect Key: KAFKA-13926 URL: https://issues.apache.org/jira/browse/KAFKA-13926 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Kumud Kumar Srivatsava Tirupati Hello, Today's connect predicates enables checks on the record metadata. However, this can be limiting considering {*}many inbuilt and custom transformations that we (community) use are more key/value centric{*}. Some use-cases this can solve: * Data type conversions of certain pre-identified fields for records coming across datasets only if those fields exist. [Ex: TimestampConverter can be run only if the specified date field exists irrespective of the record metadata] * Skip running certain transform if a given field does/does not exist. A lot of inbuilt transforms raise exceptions (Ex: InsertField transform if the field already exists) thereby breaking the task. Giving this control enable users to consciously configure for such cases. * Even though some inbuilt transforms explicitly handle these cases, it would still be an unnecessary pass-through loop. * Considering each connector usually deals with multiple datasets (Even 100s for a database CDC connector), metadata-centric predicate checking will be somewhat limiting when we talk about such pre-identified custom metadata fields in the records. I know some of these cases can be handled within the transforms itself but that defeats the purpose of having predicates. We have built this predicate for us and it is found to be extremely helpful. Please let me know your thoughts on the same so that I can raise a PR. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (KAFKA-13926) Proposal to have "HasField" predicate for kafka connect
[ https://issues.apache.org/jira/browse/KAFKA-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540756#comment-17540756 ] Kumud Kumar Srivatsava Tirupati commented on KAFKA-13926: - Sure [~showuon]. I thought adding a new predicate doesn't affect any existing public API. But, will follow the KIP process. Thanks. > Proposal to have "HasField" predicate for kafka connect > --- > > Key: KAFKA-13926 > URL: https://issues.apache.org/jira/browse/KAFKA-13926 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Kumud Kumar Srivatsava Tirupati >Assignee: Kumud Kumar Srivatsava Tirupati >Priority: Major > > Hello, > Today's connect predicates enables checks on the record metadata. However, > this can be limiting considering {*}many inbuilt and custom transformations > that we (community) use are more key/value centric{*}. > Some use-cases this can solve: > * Data type conversions of certain pre-identified fields for records coming > across datasets only if those fields exist. [Ex: TimestampConverter can be > run only if the specified date field exists irrespective of the record > metadata] > * Skip running certain transform if a given field does/does not exist. A lot > of inbuilt transforms raise exceptions (Ex: InsertField transform if the > field already exists) thereby breaking the task. Giving this control enable > users to consciously configure for such cases. > * Even though some inbuilt transforms explicitly handle these cases, it > would still be an unnecessary pass-through loop. > * Considering each connector usually deals with multiple datasets (Even 100s > for a database CDC connector), metadata-centric predicate checking will be > somewhat limiting when we talk about such pre-identified custom metadata > fields in the records. > I know some of these cases can be handled within the transforms itself but > that defeats the purpose of having predicates. > We have built this predicate for us and it is found to be extremely helpful. > Please let me know your thoughts on the same so that I can raise a PR. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (KAFKA-13926) Proposal to have "HasField" predicate for kafka connect
[ https://issues.apache.org/jira/browse/KAFKA-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumud Kumar Srivatsava Tirupati updated KAFKA-13926: Description: Hello, Today's connect predicates enables checks on the record metadata. However, this can be limiting considering {*}many inbuilt and custom transformations that we (community) use are more key/value centric{*}. Some use-cases this can solve: * Data type conversions of certain pre-identified fields for records coming across datasets only if those fields exist. [Ex: TimestampConverter can be run only if the specified date field exists irrespective of the record metadata] * Skip running certain transform if a given field does/does not exist. A lot of inbuilt transforms raise exceptions (Ex: InsertField transform if the field already exists) thereby breaking the task. Giving this control enable users to consciously configure for such cases. * Even though some inbuilt transforms explicitly handle these cases, it would still be an unnecessary pass-through loop. * Considering each connector usually deals with multiple datasets (Even 100s for a database CDC connector), metadata-centric predicate checking will be somewhat limiting when we talk about such pre-identified custom metadata fields in the records. I know some of these cases can be handled within the transforms itself but that defeats the purpose of having predicates. We have built this predicate for us and it is found to be extremely helpful. Please let me know your thoughts on the same so that I can raise a PR. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-845%3A+%27HasField%27+predicate+for+kafka+connect was: Hello, Today's connect predicates enables checks on the record metadata. However, this can be limiting considering {*}many inbuilt and custom transformations that we (community) use are more key/value centric{*}. Some use-cases this can solve: * Data type conversions of certain pre-identified fields for records coming across datasets only if those fields exist. [Ex: TimestampConverter can be run only if the specified date field exists irrespective of the record metadata] * Skip running certain transform if a given field does/does not exist. A lot of inbuilt transforms raise exceptions (Ex: InsertField transform if the field already exists) thereby breaking the task. Giving this control enable users to consciously configure for such cases. * Even though some inbuilt transforms explicitly handle these cases, it would still be an unnecessary pass-through loop. * Considering each connector usually deals with multiple datasets (Even 100s for a database CDC connector), metadata-centric predicate checking will be somewhat limiting when we talk about such pre-identified custom metadata fields in the records. I know some of these cases can be handled within the transforms itself but that defeats the purpose of having predicates. We have built this predicate for us and it is found to be extremely helpful. Please let me know your thoughts on the same so that I can raise a PR. > Proposal to have "HasField" predicate for kafka connect > --- > > Key: KAFKA-13926 > URL: https://issues.apache.org/jira/browse/KAFKA-13926 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Kumud Kumar Srivatsava Tirupati >Assignee: Kumud Kumar Srivatsava Tirupati >Priority: Major > > Hello, > Today's connect predicates enables checks on the record metadata. However, > this can be limiting considering {*}many inbuilt and custom transformations > that we (community) use are more key/value centric{*}. > Some use-cases this can solve: > * Data type conversions of certain pre-identified fields for records coming > across datasets only if those fields exist. [Ex: TimestampConverter can be > run only if the specified date field exists irrespective of the record > metadata] > * Skip running certain transform if a given field does/does not exist. A lot > of inbuilt transforms raise exceptions (Ex: InsertField transform if the > field already exists) thereby breaking the task. Giving this control enable > users to consciously configure for such cases. > * Even though some inbuilt transforms explicitly handle these cases, it > would still be an unnecessary pass-through loop. > * Considering each connector usually deals with multiple datasets (Even 100s > for a database CDC connector), metadata-centric predicate checking will be > somewhat limiting when we talk about such pre-identified custom metadata > fields in the records. > I know some of these cases can be handled within the transforms itself but > that defeats the purpose of having predicates. > We have built this predicate for us
[jira] [Commented] (KAFKA-13926) Proposal to have "HasField" predicate for kafka connect
[ https://issues.apache.org/jira/browse/KAFKA-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17542074#comment-17542074 ] Kumud Kumar Srivatsava Tirupati commented on KAFKA-13926: - [~showuon] Added a new KIP at [https://cwiki.apache.org/confluence/display/KAFKA/KIP-845%3A+%27HasField%27+predicate+for+kafka+connect] Please review > Proposal to have "HasField" predicate for kafka connect > --- > > Key: KAFKA-13926 > URL: https://issues.apache.org/jira/browse/KAFKA-13926 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Kumud Kumar Srivatsava Tirupati >Assignee: Kumud Kumar Srivatsava Tirupati >Priority: Major > > Hello, > Today's connect predicates enables checks on the record metadata. However, > this can be limiting considering {*}many inbuilt and custom transformations > that we (community) use are more key/value centric{*}. > Some use-cases this can solve: > * Data type conversions of certain pre-identified fields for records coming > across datasets only if those fields exist. [Ex: TimestampConverter can be > run only if the specified date field exists irrespective of the record > metadata] > * Skip running certain transform if a given field does/does not exist. A lot > of inbuilt transforms raise exceptions (Ex: InsertField transform if the > field already exists) thereby breaking the task. Giving this control enable > users to consciously configure for such cases. > * Even though some inbuilt transforms explicitly handle these cases, it > would still be an unnecessary pass-through loop. > * Considering each connector usually deals with multiple datasets (Even 100s > for a database CDC connector), metadata-centric predicate checking will be > somewhat limiting when we talk about such pre-identified custom metadata > fields in the records. > I know some of these cases can be handled within the transforms itself but > that defeats the purpose of having predicates. > We have built this predicate for us and it is found to be extremely helpful. > Please let me know your thoughts on the same so that I can raise a PR. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (KAFKA-13926) Proposal to have "HasField" predicate for kafka connect
[ https://issues.apache.org/jira/browse/KAFKA-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumud Kumar Srivatsava Tirupati resolved KAFKA-13926. - Resolution: Won't Fix Dropping in favor of improving the existing SMTs as per the discussion. https://lists.apache.org/thread/odbj7793plyz7xxyy6d71c3xn7zng49f > Proposal to have "HasField" predicate for kafka connect > --- > > Key: KAFKA-13926 > URL: https://issues.apache.org/jira/browse/KAFKA-13926 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Kumud Kumar Srivatsava Tirupati >Assignee: Kumud Kumar Srivatsava Tirupati >Priority: Major > > Hello, > Today's connect predicates enables checks on the record metadata. However, > this can be limiting considering {*}many inbuilt and custom transformations > that we (community) use are more key/value centric{*}. > Some use-cases this can solve: > * Data type conversions of certain pre-identified fields for records coming > across datasets only if those fields exist. [Ex: TimestampConverter can be > run only if the specified date field exists irrespective of the record > metadata] > * Skip running certain transform if a given field does/does not exist. A lot > of inbuilt transforms raise exceptions (Ex: InsertField transform if the > field already exists) thereby breaking the task. Giving this control enable > users to consciously configure for such cases. > * Even though some inbuilt transforms explicitly handle these cases, it > would still be an unnecessary pass-through loop. > * Considering each connector usually deals with multiple datasets (Even 100s > for a database CDC connector), metadata-centric predicate checking will be > somewhat limiting when we talk about such pre-identified custom metadata > fields in the records. > I know some of these cases can be handled within the transforms itself but > that defeats the purpose of having predicates. > We have built this predicate for us and it is found to be extremely helpful. > Please let me know your thoughts on the same so that I can raise a PR. > > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-845%3A+%27HasField%27+predicate+for+kafka+connect -- This message was sent by Atlassian Jira (v8.20.7#820007)