Re: Kafka 0.9 producer doesn't work

Did you start your consumer before sending message? Broker version? Cheers, Steve On Wed, Jan 20, 2016, 3:57 PM BYEONG-GI KIM wrote: > Hello. > > I set up the Kafka testbed environment on my VirtualBox, which simply has a > Kafka broker. > > I tested the simple consumer & producer scripts, aka

Re: Kafka 0.9 producer doesn't work

Sure, I started consumer before starting and sending messages from producer, and my broker version, if you mean the kafka version, is 0.9.0. Best regards Kim 2016-01-20 17:28 GMT+09:00 Steve Tian : > Did you start your consumer before sending message? Broker version? > > Cheers, Steve > > On W

Re: Kafka 0.9 producer doesn't work

It is possible that you closed the producer before the messages accumulated in batch had been sent out. You can modify your producer as below to make it a sync call and test again. producer.send(new ProducerRecord("test", 0, Integer.toString(i), Integer.toString(i))).get(); On Wed, 20 Jan 2016 a

Re: Kafka 0.9 producer doesn't work

Your code works in my environment. Are you able to run your producer code inside your vm? You can also debug via changing the log level to DEGUG/TRACE. Cheers, Steve On Wed, Jan 20, 2016, 4:30 PM BYEONG-GI KIM wrote: > Sure, I started consumer before starting and sending messages from > produ

Re: Consumer - Failed to find leader

Hi Harsh/Ismael, Any suggestions or inputs for the above issue? When i run the producer client, I still get this error ./kafka-console-producer.sh --broker-list hostname:9094 --topic topic3 *[2016-01-05 10:16:20,272] ERROR Error when sending message to topic test with key: null, value: 5 bytes

unable to set consumer group

Hello. I am trying to create consumer using kafka_python-0.9.5. I expect that several instances of the script == consumer = KafkaConsumer('some-topic', auto_offset_reset='largest', metadata_broker_list=['localhost:9092'],

SimpleConsumer.getOffsetsBefore() in 0.9 KafkaConsumer

Hi, I'm currently looking into implementing a load shedding strategy into Flink's Kafka consumer. Therefore, I would like to allow users to request the latest offset of the subscribed TopicPartitions, so that they can a) determine the lag b) maybe set the next fetch offset to the latest offset (o

Create Kafka Topic Programatically

Kafka Users, How can I create a kafka topic programatically? I would like to create the topics when I initialize my application. It should also be in such a way that if the topic already exists, the initialization code should do nothing! Thanks and Regards, Joe

Re: Create Kafka Topic Programatically

Hi Joe, There is an option in the producer called auto.create.topics.enable, so the producer can just start sending data to a topic and the topic will be created with the default values. 2016-01-20 13:19 GMT+01:00 Joe San : > Kafka Users, > > How can I create a kafka topic programatically? > >

Re: Create Kafka Topic Programatically

I doubt that might be enough. Could you tell me if the auto.create.topics.enable satisfies the following requirement? 1. I want to create a topic with a specific name 2. If I restart the producer client, if the topic with that name already exists, it should do nothing and use the topic as is 3. Up

Re: Create Kafka Topic Programatically

Hi, I believe it is a broker property. It will create the topic with the name you provide. The topic will not get deleted unless you manually delete it. It wont get re-created on subsequent calls (unless you've deleted it) HTH, Damian On 20 January 2016 at 13:14, Joe San wrote: > I doubt that

Re: Create Kafka Topic Programatically

Hi Joe, We're doing a similar thing that you're looking for with some of our app nodes. We use the kafka.admin.AdminTools and its topicExists()/createTopic() methods. Some additional code (e.g. Kafka context check/creation) might be needed for dev/test environments if you're using a specific Kafka

Re: Possible Error Code for Kafka API Calls

Thanks Gwen! When I first wrote the possible error codes in code I was also thinking about auto-generate the table in docs if possible. But unfortunately cannot find a possible manner. When we add more error codes / change error codes for requests / responses we need to keep track on the wiki, whi

Re: Support customized security protocol

Hi Tao, On Wed, Jan 20, 2016 at 2:25 AM, tao xiao wrote: > The PR provides a new SASL mech but it doesn't provide a pluggable way to > implement user's own logic to do authentication. So I don't think the PR > will meet my need. > Yes, as I said, that would not be enough for your use-case. But

Re: Create Kafka Topic Programatically

This works, but it's clumsy and has limitations. Unfortunately, I'm not aware of any alternatives. We do some programmatic topic creation when we want to create a topic with a non-default configuration. But specifying the configuration is an all or nothing affair. For example, there's no way to

Re: unable to set consumer group

version 0.9.5 of kafka-python does not support coordinated consumer groups. You can get this feature in the master branch on github ( https://github.com/dpkp/kafka-python) using kafka 0.9.0.0 brokers. I expect to release the updates to pypi soon, but for now you'll have to install from source. Oth

Kafka APIs version

Hi team, In the Kafka protocol wiki it states that version 0 is the only supported version in all APIs. I want to know if this still remains true? if not which APIs are now using version 1?

Re: Support customized security protocol

Hi Connie, On Wed, Jan 20, 2016 at 3:56 AM, Connie Yang wrote: > @Ismael, what's the status of the SASL/PLAIN PR, > https://github.com/apache/kafka/pull/341? Rajini said he would create a KIP for it: https://issues.apache.org/jira/browse/KAFKA-2658?focusedCommentId=14987903&page=com.atlassian

Re: unable to set consumer group

Dana, thank you for comprehensive response. It seems that we have to be 0.8 compatible. We've already discovered pykafka and BalancedConsumer is our favorite choice, but it might be that assigning partition explicitly suits our needs better. 20.01.2016, 19:58, "Dana Powers" : > version 0.9.5

Re: Create Kafka Topic Programatically

https://issues.apache.org/jira/browse/KAFKA-2945 and related jiras is probably what you are looking for. That is planned for the next release. On Wed, Jan 20, 2016 at 8:41 AM, Tommy Becker wrote: > This works, but it's clumsy and has limitations. Unfortunately, I'm not > aware of any alternative

Re: Kafka APIs version

That is definitely no longer true. A number of requests are at version 1 or higher. I will file a jira for this. On Wed, Jan 20, 2016 at 8:30 AM, tao xiao wrote: > Hi team, > > In the Kafka protocol wiki it states that version 0 is the only supported > version in all APIs. I want to know if this

Re: security: encryption at rest and key rotation idea

Hi Jens, I got your point but some of our use case cannot just rely on TTL. We try to have long expiry for message and rather compact them (dedup) so we can replay messages as system of records. When key is lost, we will invalid the old key so message encrypted by old message will not be able to

Re: security: encryption at rest and key rotation idea

You could do this with (I expect) reasonable efficiency and with no changes to Kafka code by using multiple topics. You can have a script that in a streaming manner reads out all messages in a topic, decrypts them with the old key, encrypts them with the new key, and adds them to a new topic. At

Re: Create Kafka Topic Programatically

Awesome, glad to see that's being worked on! On 01/20/2016 12:39 PM, Joel Koshy wrote: https://issues.apache.org/jira/browse/KAFKA-2945 and related jiras is probably what you are looking for. That is planned for the next release. On Wed, Jan 20, 2016 at 8:41 AM, Tommy Becker

Re: security: encryption at rest and key rotation idea

Jim, So I guess the problem of copying to a different topic (or would rather have a replicated cluster) is when existing consumer do the "switch" to new topic, how is the offset to be set correctly so they don't replay the whole thing again. While we can certain do idempotency with consumer, the

Having multi-threaded Kafka Consumer per partition, is it possible and recommended, if so any sample snippet?

Hello, We are using Kafka 0.9 version and are having large number of messages pushed to specific partition within a kafka topic. And there are multiple such partitions within this topic. We have one consumer assigned per partition within this topic, and we are maintaining offset manually within th

acks

We are using kafka 0.8.2.1 and set acks to 2, see the following warning: sent a produce request with request.required.acks of 2, which is now deprecated and will be removed in next release. Valid values are -1, 0 or 1. Please consult Kafka documentation for supported and recommended configuration.

Re: acks

Hi Fang, take a look at the docs on KIP-1 for some background info on acks policy: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1+-+Remove+support+of+request.required.acks -Dana On Wed, Jan 20, 2016 at 3:50 PM, Fang Wong wrote: > We are using kafka 0.8.2.1 and set acks to 2, see the f

Re: Kafka 0.9 producer doesn't work

Thank you for the reply. I tested it and the result showed timeout. I guess my kafka setting is something wrong, not the source code. I'll check what I missed. Best regards Kim 2016-01-20 18:03 GMT+09:00 tao xiao : > It is possible that you closed the producer before the messages accumulated

Re: Kafka 0.9 producer doesn't work

Dear Steve Could you tell me what kafka version you are using for the source code's package? I included the kafka library from maven repository ( http://mvnrepository.com/artifact/org.apache.kafka), and the artifactId is kafka_2.11 and version is 0.9.0.0. The link is as below: http://mvnreposito

Re: Broker Exception: Attempt to read with a maximum offset less than start offset

Hi, Any suggestions for this issue or do I need to provide more information ? Any links I can refer to would be also very helpful. Thanks, Manu Zhang On Tue, Jan 19, 2016 at 8:41 PM, Manu Zhang wrote: > Hi all, > > Is KAFKA-725 Broker Exception: Attempt to read with a maximum offset less > th

Re: Kafka 0.9 producer doesn't work

Hello. I packaged it to an executable jar file and executed it on the VM, and yes, it was successfully worked. I'm really confuse why it didn't work on my Windows10 environment where is on the host environment and worked well on the VM environment... It is weird indeed. Best regards Kim 2016-0

Re: Kafka 0.9 producer doesn't work

Yes, that's the version I was using. If all you need is Java client, then you can try: org.apache.kafka kafka-clients 0.9.0.0 Cheers, Steve On Thu, Jan 21, 2016, 9:04 AM BYEONG-GI KIM wrote: > Dear Steve > > Could you tell me what kafka version you are using for the source code's

Re: Kafka 0.9 producer doesn't work

Have you checked the firewall setting on vm/host? On Thu, Jan 21, 2016, 10:29 AM BYEONG-GI KIM wrote: > Hello. > > I packaged it to an executable jar file and executed it on the VM, and > yes, it was successfully worked. > > I'm really confuse why it didn't work on my Windows10 environment where

Re: Kafka 0.9 producer doesn't work

Yes, I opened the kafka-server/zookeeper ports of firewalld. And I finally noticed what was wrong; I didn't set "advertised.host.name" property in config/server.properties file... It should be set to the IP of the VM where the kafka server and zookeeper is running. After I put the IP of the VM in

Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message

Hi Ismael, I looked again at the problem where I get ACK error 2 (InvalidMessage). When the error occurs, I see the error message with stack trace shown below. For the 8-partition topic "shown_news_stories" which I am sending messages to, only partition 7 has its lead replica on the broker running

Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message

Hi Dave, This change was introduced in https://issues.apache.org/jira/browse/KAFKA-1755 for compacted topics. > > Interestingly, none of the messages currently going to the topic use > message > compaction (i.e. they all have empty keys), although at some time in the > past > I may have sent a fe