Re: Kafka Connect Hdfs Sink not sinking

Are you writing new data into the topic that the HDFS sink is trying to read data from? This line [2016-10-28 10:56:48,408] TRACE hdfs-sink-0 polling consumer with timeout 58820 ms (org.apache.kafka.connect.runtime.WorkerSinkTask:221) indicates it's going to wait for about 60s until some data arr

Question regarding dynamic subscriber environment

Kafka's current nature is does not support to dynamic subscriber environment where the topic interest of the subscriber is keep change overtime unless subscriber terminate the particular connection and reconnect. What I have planned to implement or design is to enable the dynamic subscriber enviro

Re: Duplicate consumer group in ListGroups in Kafka 0.10.1?

Oh, and one more tidbit: below are the responses if I manually send a DescribeGroupsRequest to each of the brokers with the given consumer group name: (from Broker #1): DescribeGroupsResponse_v0(groups=[(error_code=0, group='details-log-etl', state='Stable', protocol_type='consumer', protocol='sta

Duplicate consumer group in ListGroups in Kafka 0.10.1?

Here's another strange bug that we're seeing after upgrading to Kafka 0.10.1.0: one of our consumer groups is appearing twice in the list, and appears to belong to two different nodes. % kafka-consumer-groups.sh --bootstrap-server localhost:40172 --list | sort | uniq -c | sort -n | grep -v '^ *1'

Re: 0.8.2.1 Client not able to connect with Kafka 0.10.0.1 cluster even the cluster has message format version 0.8.2.

Hi Ismael, Below is the stack trace from client. java.io.EOFException: Received -1 when reading from channel, socket has >> likely been closed. > > at kafka.utils.Utils$.read(Utils.scala:381) > > at >> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) > > at kafka

Consumer group restart on in-memory Kafka cluster

Hello. I'm using in-memory Kafka cluster 0.9.0.1 - 2.11 (2 in-memory KafkaServers bound with in-memory zookeeper) with java-based consumer for integration tests. One of the test scenarios checks that application still receives messages after it was restarted. It relies on the fact that restar

Re: 0.8.2.1 Client not able to connect with Kafka 0.10.0.1 cluster even the cluster has message format version 0.8.2.

OffsetFetchRequest should have api key = 9, but for some reason your broker is receiving a request with api key = 10 (GroupCoordinatorRequest). Can you provide the stacktrace from the client as well? Ismael On Tue, Nov 1, 2016 at 12:13 PM, Madhukar Bharti wrote: > Hi, > > We have upgraded our c

Re: Segments being deleted too early after upgrading 0.9.0.1 to 0.10.1.0

Hi, James, That's a good point. KAFKA-3802 should cause the log segments to be kept longer, instead of shorter. So, there is probably something else that's causing this behavior. Could you try if you can reproduce this? When you do that, one thing you could try is to set log.segment.delete.delay.m

0.8.2.1 Client not able to connect with Kafka 0.10.0.1 cluster even the cluster has message format version 0.8.2.

Hi, We have upgraded our cluster from 0.8.2.1 to 0.10.0.1. Now *log.message.format.version* is set to *0.8.2. * We are still using Kafka 0.8.2.1 jars in client side, when we tried to fetch consumer offset using BlockingChannel and OffsetFetchRequest as in example here

Re: Kafka edge nodes

You could have a http front-end to kafka in your primary compute zone and push requests from your edges (http POST) to this frontend which will then use a Kafka producer to publish messages to your central kafka cluster. From: Peter N. Sent: Monday, October 31,

Re: Modify in-flight messages

Dear Michael, Thanks for sharing this information with me. I am aware of the fact that each message has a timestamp indicating either the log append or creation time. But in my case, this is not enough since I want to derive the consumption time of messages. The reason for this is because we

RE: Kafka edge nodes

You want the servers in the primary zone to put messages onto Kafka and applications in the edge nodes to read and process them? -Dave This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or enti

Kafka edge nodes

I have a primary compute zone that takes in all the requests we've seen and processes them. I have a lot of edge nodes. How would I push the data to be processed in that zone? Thank you

Re: 0.10.1.0 - commitSync() doesn't contribute to "aliveness" of a consumer?

For reference here's teh complete stacktrace (it's triggered when commitSync gets called): org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between

Re: NPEs for practically empty log

IMO, it's a bug and it shouldn't be throwing NPEs. If this is reproducible then maybe you can file a JIRA so that someone from the dev team can take a look. -Jaikiran On Friday 21 October 2016 10:37 AM, Максим Гумеров wrote: Hi! I see WARNs on kafka startup even if I only have a single empty

0.10.1.0 - commitSync() doesn't contribute to "aliveness" of a consumer?

We are using Kafka 0.10.1.0 (server) and Java client API (the new API) for consumers. One of the issues we have been running into is that the consumer is considered "dead" by the co-ordinator because of the lack of activity within a specific period of time. In reality, the consumer is still ali

Re: Modify in-flight messages

Hi Dominik, On both the consumer and producer you can use the "interceptor.classes" config to specify classes that intercept and can modify records when they are sent/received. Also as of Kafka 0.10, messages have a timestamp field. See http://kafka.apache.org/0101/javadoc/org/apache/kafka/clients

Modify in-flight messages

Is it possible to somehow modify the Kafka message payload before being sent to the consumer for consumption? Such as for example adding a timestamp to the current message payload indicating the time of message consumption. Dominik Šafarić

Re: Duplicate messages when reading with multiple consumers in same group

To answer my own question for people who may run into this same issue in the future: it appears that this behavior is by design, or at least parts of Kafka are written to work around this issue. Looking at the class kafka.javaapi.consumer.ConsumerRebalanceListener, it can be seen that the method b

Can mirroring preserve every topic's partition for all the messages?

can mirror sync topic's partition and group's offset?