[jira] [Comment Edited] (KAFKA-1499) Broker-side compression configuration
[
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152833#comment-14152833
]
Manikumar Reddy edited comment on KAFKA-1499 at 9/30/14 6:21 AM:
-
Hi Jay,
{quote}
As long as there is an option to disable broker compression then compaction
won't work properly
{quote}
Can you explain why compaction won't work properly?
What can be the default compression type? With this new behavior it is must
for users/ops to configure some compression type.
As we are trying to change current behavior, we may have to push this to 0.9
release.
was (Author: omkreddy):
Hi Jay,
{quote}
As long as there is an option to disable broker compression then compaction
won't work properly??
{quote}
Can you explain why compaction won't work properly?
> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
> Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
> Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch,
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch,
> KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch,
> KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch,
> KAFKA-1499_2014-09-25_11:05:57.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1499) Broker-side compression configuration
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148718#comment-14148718 ] Manikumar Reddy edited comment on KAFKA-1499 at 9/26/14 4:19 AM: - Yes, broker.compression.enable property does not belongs to LogConfig. This property is available in kafkaConfig. KafkaConfig is not used in Log.scala. Can I modify the constructor of LogManager, Log.scala classes to pass "broker.compression.enable" property. was (Author: omkreddy): Yes, broker.compression.enable property does not belongs to LogConfig. This config is available in kafkaConfig. KafkaConfig is not used in Log.scala. Can I modify the constructor of LogManager, Log.scala classes to pass "broker.compression.enable" property. > Broker-side compression configuration > - > > Key: KAFKA-1499 > URL: https://issues.apache.org/jira/browse/KAFKA-1499 > Project: Kafka > Issue Type: New Feature >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1499.patch, KAFKA-1499.patch, > KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch, > KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch, > KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch, > KAFKA-1499_2014-09-25_11:05:57.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > A given topic can have messages in mixed compression codecs. i.e., it can > also have a mix of uncompressed/compressed messages. > It will be useful to support a broker-side configuration to recompress > messages to a specific compression codec. i.e., all messages (for all > topics) on the broker will be compressed to this codec. We could have > per-topic overrides as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1499) Broker-side compression configuration
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134925#comment-14134925 ] Manikumar Reddy edited comment on KAFKA-1499 at 9/16/14 5:13 AM: - [~jjkoshy] I think your are commenting about KAFKA-1374(compaction support for compressed topics). I will update stress test results in couple of days. Recap of work done in this patch: 1. Introduced new server side config properties *broker.compression.enable* , *broker.compression.type* and per topic override property *compression.type* 2. If broker.compression.enabled=true, then broker will re-compress the received messages to configured compression type, irrespective of their original compression. was (Author: omkreddy): [~jjkoshy] I think your are commenting about KAFKA-1374(compaction support for compressed topics). I will update stress results in couple of days. Recap of work done in this patch: 1. Introduced new server side config properties *broker.compression.enable* , *broker.compression.type* and per topic override property *compression.type* 2. If broker.compression.enabled=true, then broker will re-compress the received messages to configured compression type, irrespective of their original compression. > Broker-side compression configuration > - > > Key: KAFKA-1499 > URL: https://issues.apache.org/jira/browse/KAFKA-1499 > Project: Kafka > Issue Type: New Feature >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1499.patch, KAFKA-1499.patch, > KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > A given topic can have messages in mixed compression codecs. i.e., it can > also have a mix of uncompressed/compressed messages. > It will be useful to support a broker-side configuration to recompress > messages to a specific compression codec. i.e., all messages (for all > topics) on the broker will be compressed to this codec. We could have > per-topic overrides as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1499) Broker-side compression configuration
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084777#comment-14084777 ] Manikumar Reddy edited comment on KAFKA-1499 at 8/4/14 3:43 PM: I would like to propose new server property "*log.compression.type*" and topic override property "*compression.type*" If set, this property is used for compression type at server-side. All non-compressed messages on the broker will be compressed to this compression type. log.compression.type=none|gzip|snappy (default = none) compression.type=none|gzip|snappy was (Author: omkreddy): I would like to propose new server property "*log.compression.type*" and topic override property "*compression.type*" If set, this property is used for compression type at server-side. All non-compressed messages on the broker will be compressed to this compression type. log.compression.type=none|gzip|snappy (default = none) compression.type=compression.type > Broker-side compression configuration > - > > Key: KAFKA-1499 > URL: https://issues.apache.org/jira/browse/KAFKA-1499 > Project: Kafka > Issue Type: New Feature >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Original Estimate: 72h > Remaining Estimate: 72h > > A given topic can have messages in mixed compression codecs. i.e., it can > also have a mix of uncompressed/compressed messages. > It will be useful to support a broker-side configuration to recompress > messages to a specific compression codec. i.e., all messages (for all > topics) on the broker will be compressed to this codec. We could have > per-topic overrides as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
