Hi,
Thanks for your reply!
OK, I got it. And, there is a parameter named compression.type in
config/producer.properties, which is same usage as "--compression-codec " I
think. I modify compression.type in config/producer.properties firstly, then
run console producer with option "--producer.config <config file>" and send
message, but the compression codec could not change as modification. Do you
know the reason about it? Thanks!
# bin/kafka-console-producer.sh
Read data from standard input and publish it to Kafka.
Option Description
------ -----------
--producer.config <config file> Producer config properties file. Note
that [producer-property] takes
precedence over this config.
# bin/kafka-console-producer.sh --producer.config config/producer.properties
--broker-list localhost:9092 --topic test
Best Regards
Johnny
-----Original Message-----
From: Hans Jespersen [mailto:[email protected]]
Sent: 2016年10月17日 14:29
To: [email protected]; Radoslaw Gruchalski
Subject: RE: A question about kafka
Because the producer-property option is used to set other properties that are
not compression type.
//[email protected]
-------- Original message --------From: ZHU Hua B
<[email protected]> Date: 10/16/16 11:20 PM (GMT-08:00) To:
Radoslaw Gruchalski <[email protected]>, [email protected] Subject: RE:
A question about kafka Hi,
Thanks for your reply!
If console producer only allows for compression codec argument, why we could
found option —producer-property defined in ConsoleProducer.scala? And we could
find the usage also if we running console producer? The version we used is
Kafka 0.10.0.0. Thanks!
# ./kafka-console-producer.sh
Read data from standard input and publish it to Kafka.
Option Description
------ -----------
--compression-codec [compression-codec] The compression codec: either 'none',
'gzip', 'snappy', or 'lz4'.If
specified without value, then it
defaults to 'gzip'
--producer-property <producer_prop> A mechanism to pass user-defined
properties in the form key=value to
the producer.
--producer.config <config file> Producer config properties file. Note
that [producer-property] takes
precedence over this config.
--property <prop> A mechanism to pass user-defined
properties in the form key=value to
the message reader. This allows
custom configuration for a user-
defined message reader.
Best Regards
Johnny
From: Radoslaw Gruchalski [mailto:[email protected]]
Sent: 2016年10月17日 14:02
To: ZHU Hua B; [email protected]
Subject: RE: A question about kafka
Hi,
I believe the answer is in the code. This is where the --compression-codec is
processed:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L143
and this is —producer-property:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L234
The usage is here:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L114
The answer is: The console producer allows for compression codec only with
—compression-codec argument.
–
Best regards,
Radek Gruchalski
[email protected]<mailto:[email protected]>
On October 17, 2016 at 7:46:41 AM, ZHU Hua B
([email protected]<mailto:[email protected]>) wrote:
Hi,
Anybody could help to answer this question? Thanks!
Best Regards
Johnny
-----Original Message-----
From: ZHU Hua B
Sent: 2016年10月14日 16:41
To: [email protected]<mailto:[email protected]>
Subject: [COMMERCIAL] A question about kafka
Hi,
I have a question about kafka, could you please help to have a look?
I want to send a message from producer with snappy compression codec. So I run
the command "bin/kafka-console-producer.sh --compression-codec snappy
--broker-list localhost:9092 --topic test", after that I checked the data log,
compresscodec is SnappyCompressionCodec as expectation.
Then I tried another command "bin/kafka-console-producer.sh --producer-property
compression.type=snappy --broker-list localhost:9092 --topic test" to send a
message from producer with snappy also, but this time I found compresscodec is
NoCompressionCodec in data log.
I double checked the option of "kafka-console-producer.sh" as below, seems both
"--compression-codec" and "--producer-property" could be used to set up
compression codec, why the second way failed? The "key=value" I found in
config/producer.properties, so I think it is regular. Thanks!
# bin/kafka-console-producer.sh --help
--compression-codec [compression-codec] The compression codec: either
'none','gzip', 'snappy', or 'lz4'.If specified without value, then it defaults
to 'gzip'.
--producer-property <producer_prop> A mechanism to pass user-defined properties
in the form key=value to the producer.
--producer.config <config file> Producer config properties file. Note that
[producer-property] takes precedence over this config.
Best Regards
Johnny
