[jira] [Commented] (KAFKA-10728) Mirroring data without decompressing with MirrorMaker 2.0
[
https://issues.apache.org/jira/browse/KAFKA-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281352#comment-17281352
]
Henry Cai commented on KAFKA-10728:
---
Please take a look at
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-712%3A+Shallow+Mirroring,]
this should help.
> Mirroring data without decompressing with MirrorMaker 2.0
> -
>
> Key: KAFKA-10728
> URL: https://issues.apache.org/jira/browse/KAFKA-10728
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
>Reporter: Eazhilan Nagarajan
>Priority: Major
>
> Hello,
>
> I use MirrorMaker 2.0 to copy data across two Kafka clusters and it's all
> working fine. Recently we enabled compressing while producing data into any
> topic which had a very positive impact on the storage and other resources but
> while mirroring, the data seems to be decompressed at the target Kafka
> cluster. I tried enabling compression using the below config in MM2, the data
> at the target cluster is compressed now, the decompress and re-compress
> continues to happen and it eats up a lot of resources unnecessarily.
>
> {noformat}
> - alias: my-passive-cluster
> authentication:
> passwordSecret:
> password: password
> secretName: passive-cluster-secret
> type: scram-sha-512
> username: user-1
> bootstrapServers: my-passive-cluster.com:443
> config:
> config.storage.replication.factor: 3
> offset.storage.replication.factor: 3
> status.storage.replication.factor: 3
> producer.compression.type: gzip{noformat}
> I found couple of Jira issues talking about it but I don't know if the
> shallow iterator option is available now.
> https://issues.apache.org/jira/browse/KAFKA-732,
> https://issues.apache.org/jira/browse/KAFKA-845
>
> Kindly let me if this is currently available or if it'll be available in the
> future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10728) Mirroring data without decompressing with MirrorMaker 2.0
[
https://issues.apache.org/jira/browse/KAFKA-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234491#comment-17234491
]
Eazhilan Nagarajan commented on KAFKA-10728:
Well similarly is there an option to tell the consumer of MM2 to not decompress
data from topics while mirroring?
> Mirroring data without decompressing with MirrorMaker 2.0
> -
>
> Key: KAFKA-10728
> URL: https://issues.apache.org/jira/browse/KAFKA-10728
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
>Reporter: Eazhilan Nagarajan
>Priority: Major
>
> Hello,
>
> I use MirrorMaker 2.0 to copy data across two Kafka clusters and it's all
> working fine. Recently we enabled compressing while producing data into any
> topic which had a very positive impact on the storage and other resources but
> while mirroring, the data seems to be decompressed at the target Kafka
> cluster. I tried enabling compression using the below config in MM2, the data
> at the target cluster is compressed now, the decompress and re-compress
> continues to happen and it eats up a lot of resources unnecessarily.
>
> {noformat}
> - alias: my-passive-cluster
> authentication:
> passwordSecret:
> password: password
> secretName: passive-cluster-secret
> type: scram-sha-512
> username: user-1
> bootstrapServers: my-passive-cluster.com:443
> config:
> config.storage.replication.factor: 3
> offset.storage.replication.factor: 3
> status.storage.replication.factor: 3
> producer.compression.type: gzip{noformat}
> I found couple of Jira issues talking about it but I don't know if the
> shallow iterator option is available now.
> https://issues.apache.org/jira/browse/KAFKA-732,
> https://issues.apache.org/jira/browse/KAFKA-845
>
> Kindly let me if this is currently available or if it'll be available in the
> future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10728) Mirroring data without decompressing with MirrorMaker 2.0
[
https://issues.apache.org/jira/browse/KAFKA-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234255#comment-17234255
]
Ning Zhang commented on KAFKA-10728:
my first impression is: unless the producer of MM2 is explicitly set to use
`uncompressed` with [https://kafka.apache.org/documentation/#compression.type]
it will use the default compression value
> Mirroring data without decompressing with MirrorMaker 2.0
> -
>
> Key: KAFKA-10728
> URL: https://issues.apache.org/jira/browse/KAFKA-10728
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
>Reporter: Eazhilan Nagarajan
>Priority: Major
>
> Hello,
>
> I use MirrorMaker 2.0 to copy data across two Kafka clusters and it's all
> working fine. Recently we enabled compressing while producing data into any
> topic which had a very positive impact on the storage and other resources but
> while mirroring, the data seems to be decompressed at the target Kafka
> cluster. I tried enabling compression using the below config in MM2, the data
> at the target cluster is compressed now, the decompress and re-compress
> continues to happen and it eats up a lot of resources unnecessarily.
>
> {noformat}
> - alias: my-passive-cluster
> authentication:
> passwordSecret:
> password: password
> secretName: passive-cluster-secret
> type: scram-sha-512
> username: user-1
> bootstrapServers: my-passive-cluster.com:443
> config:
> config.storage.replication.factor: 3
> offset.storage.replication.factor: 3
> status.storage.replication.factor: 3
> producer.compression.type: gzip{noformat}
> I found couple of Jira issues talking about it but I don't know if the
> shallow iterator option is available now.
> https://issues.apache.org/jira/browse/KAFKA-732,
> https://issues.apache.org/jira/browse/KAFKA-845
>
> Kindly let me if this is currently available or if it'll be available in the
> future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
