Re: kafka 0.8 producer issue

Can u share the code that you are using to publish the message. Also can u whether a small message is.published On Nov 25, 2015 9:25 PM, "Kudumula, Surender" wrote: > Hi all > I am trying to get the producer working. It was working before but now > getting the following issue. I have created a ne

Re: Increasing replication factor reliable?

So u have two nodes running where you want to increase the replication factor 2 because of fault tolerance. That won't be a problem On Nov 25, 2015 6:26 AM, "Dillian Murphey" wrote: > Is it safe to run this on an active production topic? A topic was created > without a replication factor of 2 an

Re: kafka producer and client use same groupid

Can u check couple of things 1. Message size that u are sending ttry sending small message 2.check which encoder defaultencoder or string encoder u are using to consume the message , are u serializing the message while sending normal stream. 3.u create.any partition of topic just create only topic

Re: Kafka broker goes down when consumer is stopped.

Did u check with ps -ef Kafka whether Kafka broker is running or not On Nov 25, 2015 4:56 PM, "Shaikh, Mazhar A (Mazhar)" < mazhar.sha...@in.verizon.com> wrote: > Hi Team, > > In my test setup, Kafka broker goes down when consumer is stopped. > > Below are the version & logs. > > Request you help

Notable failure scenarios in high-level/new consumer

Hello Folks, I am trying to build fault tolerance on the consumer side, so as to make sure that all failure scenarios are handled. On Data integrity side, there are primary 2 requirements:- 1. No Data loss 2. No data duplication I'm particularly interested in data duplication. e.g. there are var

Re: 0.9.0.0[error]

I think it has nothing to do with those clients. actually i write a consumer client with the erlang programming language, but i have not used it yet. i just use the script: kafka-topic.sh to create a topic named blcs , then this error is reported 2015-11-26 13:37 GMT+08:00 Jun Rao : > Are you r

Re: 0.9.0.0[error]

Are you running any non-java client, especially a consumer? Thanks, Jun On Wed, Nov 25, 2015 at 6:38 PM, Fredo Lee wrote: > this is my config file for original file with some changed by me. > > broker.id=1 > listeners=PLAINTEXT://:9092 > num.partitions=10 > log.dirs=/tmp/kafka-logs1 > zookeepe

Re: Flush Messages in KafkaProducer Buffer

Hello Gwen, Currently I am using 0.8.2, is it recommended to upgrade to 0.9.0? does it will backward compatible to 0.8.2 brokers? Is there any way to 'recover' buffered messages in case there is exception when calling the close() method? Thank you, On Thu, Nov 26, 2015 at 8:26 AM, Gwen Shapira

Re: 0.9.0.0[error]

this is my config file for original file with some changed by me. broker.id=1 listeners=PLAINTEXT://:9092 num.partitions=10 log.dirs=/tmp/kafka-logs1 zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=2000 delete.topic.enable=true default.replication.factor=2 auto.leader.rebalance.en

Re: Kafka Broker Max Connection

I havn't found it in the document. 2015-11-25 19:40 GMT+08:00 Muqtafi Akhmad : > but is there any config that limit connections 'globally'? is it only > limited by host's maximum number of open file descriptor? > > Thank you, > > On Tue, Nov 24, 2015 at 6:53 PM, 马哲超 wrote: > > > There is a conf

Re: Increasing replication factor reliable?

Yeah - it will increase IO and network utilization by a lot while data is replication, but it should be safe. On Tue, Nov 24, 2015 at 4:56 PM, Dillian Murphey wrote: > Is it safe to run this on an active production topic? A topic was created > without a replication factor of 2 and I want to inc

Re: Kafka broker goes down when consumer is stopped.

It looks like you have a single broker (with id = 0 ) and that topic1 has a single replica and the broker is alive and well. The socket error is our bug (shouldn't be an error) and doesn't indicate that the broker is down. On Wed, Nov 25, 2015 at 3:26 AM, Shaikh, Mazhar A (Mazhar) < mazhar.sha...

Re: Flush Messages in KafkaProducer Buffer

In 0.9.0, close() has a timeout parameter that allows specifying how long to wait for the in-flight messages to complete (definition of complete depends on value of "acks" parameter). On Wed, Nov 25, 2015 at 3:58 AM, Muqtafi Akhmad wrote: > Hello guys, > > I am using KafkaProducer (org.apache.ka

Re: All brokers are running but some partitions' leader is -1

Great to know that. Thanks Gwen! On Wed, Nov 25, 2015 at 12:03 PM, Gwen Shapira wrote: > 1. Yes, you can do a rolling upgrade of brokers from 0.8.2 to 0.9.0. The > important thing is to upgrade the brokers before you upgrade any of the > clients. > > 2. I'm not aware of issues with 0.9.0 and Spa

Kafka 0.8.2.1 - how to read from __consumer_offsets topic?

Hello, I'm trying to find out which offsets my current High-Level consumers are working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in the server.properties of Kafka - which, I think, means that offsets are stored in Kafka. (I also verified that no offsets are stored in Zookeeper

Re: Kafka Connect and Spark/Storm Comparisons

Hey Dave, We're separating the problem of getting data in and out of Kafka from the problem of transforming it. If you think about ETL (Extract, Transform, Load), what Kafka Connect does is E and L really really well and not T at all; the focus in stream processing systems is T with E and L being

Re: All brokers are running but some partitions' leader is -1

1. Yes, you can do a rolling upgrade of brokers from 0.8.2 to 0.9.0. The important thing is to upgrade the brokers before you upgrade any of the clients. 2. I'm not aware of issues with 0.9.0 and SparkStreaming. However, definitely do your own testing to make sure. On Wed, Nov 25, 2015 at 11:25 A

Re: All brokers are running but some partitions' leader is -1

Hi Gwen, Yes, we're going to upgrade the 0.9.0 version. Regarding the upgrade, we definitely don't want to have down time of our cluster. So the upgrade will be machine by machine. Will the release 0.9.0 work with the Aug's version together in the same Kafka cluster? Also we currently run spark str

Re: kafka 0.8 producer issue

Hi, >From the information that you've provided, I think your callback is the culprit here. It is seen from the stacktrace:- at com.hpe.ssmamp.kafka.KafkaPDFAProducer$1.onCompletion( KafkaPDFAProducer.java:62) Please provide more information like a code snippet etc, so that we can tell more

RE: kafka 0.8 producer issue

Hi prabhjot I just realized its not producing to kafka that’s why iam getting this null pointer. In my localhost I can see kafka info/debug logs. But in HDP cluster I cannot see any exceptions. How should I configure the logger to show exceptions in kafka log files thanks -Original Messag

Re: Kafka Connect and Spark/Storm Comparisons

Spark's direct stream kafka integration should take advantage of data locality if you're running Spark executors on the same nodes as Kafka brokers. On Wed, Nov 25, 2015 at 9:50 AM, Dave Ariens wrote: > I just finished reading up on Kafka Connect< > http://kafka.apache.org/documentation.html#con

Re: Consumer group disappears and consumers loops

Hello Martin, It seems your consumer's heartbeat.interval.ms config value is too small (default is 3 seconds) for your environment, consider increasing it and see if this issue goes away. At the same time, we have some better error handling fixes in trunk which will be included in the next point

[ANNOUNCE] CFP open for ApacheCon North America 2016

Community growth starts by talking with those interested in your project. ApacheCon North America is coming, are you? We are delighted to announce that the Call For Presentations (CFP) is now open for ApacheCon North America. You can submit your proposed sessions at http://events.linuxfoundation.o

RE: Regarding The Kafka Offset Management Issue In Direct Stream Approach.

Charan, You may find this Gist useful for storing/retrieving offsets for Kafka topics: https://gist.github.com/ariens/e6a39bc3dbeb11467e53 From: Cody Koeninger [c...@koeninger.org] Sent: Friday, November 06, 2015 10:10 AM To: users@kafka.apache.org Subjec

mvn repository for kafka 0.9 jars

Nice, switching to maven central. Thanks a lot Wanny From: Jun Rao [j...@confluent.io] Sent: Wednesday, November 25, 2015 8:07 AM To: users@kafka.apache.org Subject: Re: mvn repository for kafka 0.9 jars Wanny, The 0.9.0.0 Kafka jars are already in maven

Re: mvn repository for kafka 0.9 jars

Wanny, The 0.9.0.0 Kafka jars are already in maven central ( http://central.maven.org/maven2/org/apache/kafka/kafka_2.10/0.9.0.0/). Thanks, Jun On Wed, Nov 25, 2015 at 8:02 AM, MrWanny wrote: > Hi, > > is there any plan to publish the kafka jars to mvn repository? > http://mvnrepository.com/a

mvn repository for kafka 0.9 jars

Hi, is there any plan to publish the kafka jars to mvn repository? http://mvnrepository.com/artifact/org.apache.kafka current the latest available is 0.8.2.2 Thanks Wanny

Re: 0.9.0.0[error]

Fredo, Thanks for reporting this. Are you starting a brand new 0.9.0.0 cluster? Are there steps that one can follow to reproduce this issue easily? Jun On Tue, Nov 24, 2015 at 10:52 PM, Fredo Lee wrote: > The content below is the report for kafka > > when i try to fetch coordinator broker, i g

kafka 0.8 producer issue

Hi all I am trying to get the producer working. It was working before but now getting the following issue. I have created a new topic as well just in case if it was the issue with topic but still no luck. I have increased the message size in broker as iam trying to send atleast 3mb message here

Kafka Connect and Spark/Storm Comparisons

I just finished reading up on Kafka Connect and am trying to wrap my head around where it fits within the big data ecosystem. Other than the high level overview provided in the docs I haven't heard much about this feature. My limited understan

Re: New kafka-consumer-groups.sh not showing inactive consumer gorup

Hi Tao, Yes this is the intended behavior, since it only queries the consumer coordinator's in-memory cache about the consumer group metadata, which only contains active consumer groups, if a group has no live members any more it is treated as dead and removed from cache. To get committed offsets

Re: upgrade to 0.9 and auto generated broker id

broker.id is the identifier of the broker. If you have existing data, you should preserve the broker.id. If you restart the broker with a new id, all replicas stored with the old broker id are considered gone. Thanks, Jun On Wed, Nov 25, 2015 at 7:20 AM, Evgeniy Shishkin wrote: > Hello, > > i

flush() vs close()

Hello, I notice that 0.9 Kafka release had a new flush() call added to the producer API (See https://issues.apache.org/jira/browse/KAFKA-1865). I am trying to understand what is the difference between producer.flush() and producer.close()? They seem to be doing pretty similar work. If not, can s

upgrade to 0.9 and auto generated broker id

Hello, i have a question regarding upgrade to 0.9 Is it recommended to keep broker.id in config after upgrade? Will be there any dataloss or stuck replication if we remove broker.id and restart broker? Thanks.

Consumer group disappears and consumers loops

Hi, I'm experiencing some very strange issues with 0.9. I get these log messages from the new consumer: [main] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred while committing offsets for group aaa-bbb-reader [main] WARN org.apache.kafka.c

Re: kafka producer and client use same groupid

Hello Surender, do you use the high level consumer, low level consumer, or using connector provided by Spark / Hadoop? On Wed, Nov 25, 2015 at 1:29 AM, Kudumula, Surender < surender.kudum...@hpe.com> wrote: > Hi Prabhjot > Thanks for your quick response. I never get any response to my queries:(

Re: message order problem

This behavior might be expected, quoting description in apache kafka documentation here Kafka only provides a total order over messages within a partition, not > between different partitions in a topic. Per-partition ordering combined > with the ability

Re: 0.9.0.0[error]

hello Fredo, Can you provide your program's code? There might be some clues On Wed, Nov 25, 2015 at 2:27 PM, Fredo Lee wrote: > four kafka nodes, i get these errors > one node, it works well. > > 2015-11-25 14:52 GMT+08:00 Fredo Lee : > > > > > The content below is the report for kafka > > > >

Does Simple Consumer Thread Safe?

Hello guys, is it safe to build a simple consumer pool that accessed by many threads? does SimpleConsumer thread safe? Thank you, -- Muqtafi Akhmad Software Engineer Traveloka

New kafka-consumer-groups.sh not showing inactive consumer gorup

Hi team, In 0.9.0.0 a new tool kafka-consumer-groups.sh is introduced to show offset and lag for a particular consumer group prefer over the old one kafka-consumer-offset-checker.sh. However I found that the new tool only works for consumer groups that are currently active but not for consumer gro

Question for consumer

I use kafka 0.8.1.1 with Java. There are many partitions in my service. Each partition has a producer or a consumer through a state control it. When state is master, i will stop consumer and shutdown the ConsumerConnecter. When state is slave, i will create a new ConsumerConnecter. So i must have

Flush Messages in KafkaProducer Buffer

Hello guys, I am using KafkaProducer (org.apache.kafka.clients.producer.KafkaProducer) to send messages to Kafka broker. I set BUFFER_MEMORY_CONFIG to some MBs. If there is a case where we need to shutdown kafka producer when there are messages in producer's buffer, is there any way to flush these

Re: Kafka Broker Max Connection

but is there any config that limit connections 'globally'? is it only limited by host's maximum number of open file descriptor? Thank you, On Tue, Nov 24, 2015 at 6:53 PM, 马哲超 wrote: > There is a configuration for broker, "max.connections.per.ip", and default > is Int.MaxValue. > > Here's the

Kafka broker goes down when consumer is stopped.

Hi Team, In my test setup, Kafka broker goes down when consumer is stopped. Below are the version & logs. Request you help to solve this issue. Kafka Version : kafka_2.10-0.8.2.1 C++ application using librdkafka_2.10-0.8.2.1 library for consumer & producer. server.log: [2015-11-25 05