[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need for a merge transformer.
A merge transformation example (We respect the [transformation
configuration|http://kafka.apache.org/documentation.html#connect_transforms])
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
*field.root* : Field name for the root field.
*field.list* : The list of fields to merge under the root field. Suffix with
*{color:#8eb021}*{color}* to keep it in top level, and prefix it with
*{color:#8eb021}?{color}* if it's optional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
*field.root* : Field name for the root field.
*field.list* : The list of fields to merge under the root field. Suffix with
*{color:#8eb021}*{color}* to keep it in top level, and prefix it with
*{color:#8eb021}?{color}* if it's optional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need for a merge transformer.
> A merge transformation example (We respect the [transformation
> configuration|http://kafka.apache.org/documentation.html#connect_transforms])
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> *field.root* : Field name for the root field.
> *field.list* : The list of fields to merge under the root field. Suffix with
> *{color:#8eb021}*{color}* to keep it in top level, and prefix it with
> *{color:#8eb021}?{color}* if it's optional.
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
*field.root* : Field name for the root field.
*field.list* : The list of fields to merge under the root field. Suffix with
{color:#8eb021}*{color} to keep it in top level, and prefix it with
{color:#8eb021}?{color} if it's opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
*field.*root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with
{color:#8eb021}*{color} to keep it in top level, and prefix it with
{color:#8eb021}?{color} if it's opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> *field.root* : Field name for the root field.
> *field.list* : The list of fields to merge under the root field. Suffix with
> {color:#8eb021}*{color} to keep it in top level, and prefix it with
> {color:#8eb021}?{color} if it's opotional.
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
*field.root* : Field name for the root field.
*field.list* : The list of fields to merge under the root field. Suffix with
*{color:#8eb021}*{color}* to keep it in top level, and prefix it with
*{color:#8eb021}?{color}* if it's optional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
*field.root* : Field name for the root field.
*field.list* : The list of fields to merge under the root field. Suffix with
{color:#8eb021}*{color} to keep it in top level, and prefix it with
{color:#8eb021}?{color} if it's opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> *field.root* : Field name for the root field.
> *field.list* : The list of fields to merge under the root field. Suffix with
> *{color:#8eb021}*{color}* to keep it in top level, and prefix it with
> *{color:#8eb021}?{color}* if it's optional.
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
*field.*root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with
{color:#8eb021}*{color} to keep it in top level, and prefix it with
{color:#8eb021}?{color} if it's opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
field.root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with
{color:#8eb021}*{color} to keep it in top level, and prefix it with ? if it's
opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> *field.*root : Field name for the root field.
> field.list: The list of fields to merge under the root field. Suffix with
> {color:#8eb021}*{color} to keep it in top level, and prefix it with
> {color:#8eb021}?{color} if it's opotional.
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
field.root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with
{color:#8eb021}*{color} to keep it in top level, and prefix it with ? if it's
opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
field.root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with `*`
to keep it in top level, and prefix it if it's opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> field.root : Field name for the root field.
> field.list: The list of fields to merge under the root field. Suffix with
> {color:#8eb021}*{color} to keep it in top level, and prefix it with ? if it's
> opotional.
>
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
field.root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with `*`
to keep it in top level, and prefix it if it's opotional.
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
field.root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with * to
keep it in top level, and prefix it if it's opotional
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> field.root : Field name for the root field.
> field.list: The list of fields to merge under the root field. Suffix with `*`
> to keep it in top level, and prefix it if it's opotional.
>
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
field.root : Field name for the root field.
field.list: The list of fields to merge under the root field. Suffix with * to
keep it in top level, and prefix it if it's opotional
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> field.root : Field name for the root field.
> field.list: The list of fields to merge under the root field. Suffix with *
> to keep it in top level, and prefix it if it's opotional
>
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
This is our MR for this transformer [https://github.com/apache/kafka/pull/5405]
was:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
This is our MR for this transformer
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> This is our MR for this transformer
> [https://github.com/apache/kafka/pull/5405]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7189) Add a Merge Transformer for Kafka Connect
[
https://issues.apache.org/jira/browse/KAFKA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdelhamide EL ARIB updated KAFKA-7189:
---
Description:
Like the
[flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
This is our MR for this transformer
was:
Like the
[flatten|[https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],]
there is the need too for a merge transformer.
Example transformation :
We want to add the offset and the partition for each record, and after that
merge them into one field _metadata :
{code:java}
"transforms":"AddOffset, AddPartition, MergeFields",
"transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddOffset.offset.field":"offset!",
"transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.AddPartition.partition.field":"partition!",
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
"transforms.MergeFields.field.list":"offset,partition",
"transforms.MergeFields.field.root":"_metadata"
{code}
This is our MR for this transformer
> Add a Merge Transformer for Kafka Connect
> -
>
> Key: KAFKA-7189
> URL: https://issues.apache.org/jira/browse/KAFKA-7189
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Abdelhamide EL ARIB
>Priority: Minor
>
> Like the
> [flatten|https://docs.confluent.io/current/connect/transforms/flatten.html#flatten],
> there is the need too for a merge transformer.
> Example transformation :
> We want to add the offset and the partition for each record, and after that
> merge them into one field _metadata :
>
> {code:java}
> "transforms":"AddOffset, AddPartition, MergeFields",
> "transforms.AddOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddOffset.offset.field":"offset!",
> "transforms.AddPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value",
> "transforms.AddPartition.partition.field":"partition!",
> "transforms.MergeFields.type":"org.apache.kafka.connect.transforms.Merge$Value",
> "transforms.MergeFields.field.list":"offset,partition",
> "transforms.MergeFields.field.root":"_metadata"
> {code}
>
> This is our MR for this transformer
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
