Re: using the new logstash-kafka plugin

Scott you will have to do just one topic per input right now but multiple topics per group, whitelisting and blacklisting just got merged into jruby-kafka and I'm working them up the chain to my logstash-kafka and then pass it to the logstash-input/output/-kafka plugin.

delete log file

Hi all, The kafka broker's disk is almost full, and the log file is still opened by kafka through lsof command. I know to change log.retention.hours in server.properties, but I don't want to restart the kafka server. Is there any utility to delete log file without impacting kafka? Thanks

Re: clarification of the per-topic retention.bytes setting

Hi David, The per-topic configs will just override the global configs for that specific topic; for the retention.bytes config it will be applied to all partitions of that topic. So if you have two topics each with two partitions and replication factor 1 with retention.bytes valued A then the

Re: How to setup inter DC replication in Kafka 0.8.1.1

For now (0.8.2) if you have a hard crash then you will likely have duplicates or even data loss unfortunately, I do not have a good solution to this on top of my head. On Wed, Jan 21, 2015 at 10:14 PM, Madhukar Bharti bhartimadhu...@gmail.com wrote: Thanks Guozhang for your reply. Checked the

clarification of the per-topic retention.bytes setting

Hi list, please help me understand the per-topic retention.* setting (in kafka 0.8.1.1) done by: bin/kafka-topics.sh --zookeeper $ZK --alter --topic $TOPIC --config retention.bytes=VALUE I understand from this:

Help: Kafka LeaderNotAvailableException

Hi, Let me overview on the issue that I am facing on producing message in Kafka: I have horthonworks HDP-2.1 installed, along that we have Kafka on other node. * On kafka node: Start Zookeepeer Start Kafka Broker service Send message/producer Consume message - Works (Note: here we start Zookeeper

Re: using the new logstash-kafka plugin

Hey Joe, with other input types (like file) one can reference things like the path in the filter section. Is it possible to refer to the topic_id in the filter section? I tried and nothing obvious worked. We are encoding a few things (like host name, and type) in the name of the topic, and would

Re: using the new logstash-kafka plugin

Just trying to get everything in prior to the 1.5 release. From: Scott Chapman sc...@woofplanet.com Sent: Thursday, January 22, 2015 9:32 AM To: users@kafka.apache.org Subject: Re: using the new logstash-kafka plugin Awesome, what release are you

Re: kafka production server test

Hi, Guozhang Can I run this package remotely test another server? which mean I run this package on dev but testing kafka system on production? thanks AL On Thu, Jan 22, 2015 at 2:55 PM, Sa Li sal...@gmail.com wrote: Hi, Guozhang, Good to know such package, will try it now. :-) thanks

Re: warning on startup of consumer app with 0.8.2 rc2

Hmm, kafka-console-consumer in 0.8.2 rc2 is running fine. Do you have multiple kafka jars in your classpath? Thanks, Jun On Thu, Jan 22, 2015 at 4:58 PM, Jason Rosenberg j...@squareup.com wrote: 2015-01-23 00:55:25,273 WARN [async-message-sender-0] common.AppInfo$ - Can't read Kafka version

Re: Partition reassignment doesn't work on few partitions

Perhaps you can upgrade all brokers and then try? Thanks, Jun On Wed, Jan 21, 2015 at 9:53 PM, Raghu Udiyar ra...@helpshift.com wrote: No errors in the state-change log or the controller. Its as if the controller never got the request for that partition. Regarding the upgrade, we did

Re: delete log file

In general, you shouldn't delete any files while the broker is up. If you have no other choices, you can try deleting the older log segments and hope that no consumer or the log cleaner are using them. Thanks, Jun On Thu, Jan 22, 2015 at 12:40 AM, YuanJia Li yuanjia8...@163.com wrote: Hi all,

Re: Help: Kafka LeaderNotAvailableException

Any error in the controller and the broker log? Thanks, Jun On Thu, Jan 22, 2015 at 1:33 AM, vishal_ja...@dell.com wrote: Hi, Let me overview on the issue that I am facing on producing message in Kafka: I have horthonworks HDP-2.1 installed, along that we have Kafka on other node. * On

Re: Help: Kafka LeaderNotAvailableException

It sounds like you have two zookeepers, one for HDP and one for Kafka. Did you move Kafka from one zookeeper to another? Perhaps Kafka finds the topics (logs) on disk, but they do not exist in ZK because you are using a different zookeeper now. Gwen On Thu, Jan 22, 2015 at 6:38 PM, Jun Rao

Re: kafka production server test

Hi, Guozhang, Good to know such package, will try it now. :-) thanks On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com wrote: Hi Sa, Have you looked into the system test package? It contains a suite of tests on different failure modes of Kafka brokers. Guozhang On Thu,

Re: kafka production server test

Hi Sa, Have you looked into the system test package? It contains a suite of tests on different failure modes of Kafka brokers. Guozhang On Thu, Jan 22, 2015 at 12:00 PM, Sa Li sal...@gmail.com wrote: Hi, All We are about to deliver kafka production server, I have been working on different

warning on startup of consumer app with 0.8.2 rc2

2015-01-23 00:55:25,273 WARN [async-message-sender-0] common.AppInfo$ - Can't read Kafka version from MANIFEST.MF. Possible cause: java.lang.NullPointerException

Re: Re: delete log file

Hi Yang， I think my problem is not the same with yours. My production environment is fine, log.retention.hours is 720. My disk is almost full just beacause of too much data. I want to have a utility or command to delete manually, instead of waiting until it expires. Thanks for your reply.

kafka topic not exist after creation with api

Hi all , My application creates kafka topic at runtime with AdminUtils.createTopic but that topic is not available when I try to produce to it. If I run bin/kafka-console-consumer.sh --topic $topic --zookeeper $zkStr --from-beginning it would throw UnknownTopicOrPartitionException. Also, I

New mirror maker consumer.config question

Hi, I discovered that the new mirror maker implementation in trunk now only accept one consumer.config property instead of a list of them which means we can only supply one source per mirror maker process. Is it a reason for it? If I have multiple source kafka clusters do I need to setup multiple

Apache Spark broadcast error: Error sending message as driverActor is null [message = UpdateBlockInfo(BlockManagerId

HiI'm using Apache Spark 1.1.0 and I'm currently having issue with broadcast method. So when I call broadcast function on a small dataset to a 5 nodes cluster, I experiencing the Error sending message as driverActor is null after broadcast the variables several times (apps running under jboss). 

Re: Apache Spark broadcast error: Error sending message as driverActor is null [message = UpdateBlockInfo(BlockManagerId

Yep, sorry, had a rough day.. On Thursday, January 22, 2015 2:25 PM, Guozhang Wang wangg...@gmail.com wrote: Hi Zijing, Sounds like you sent to the wrong mailing list :P Guozhang On Thu, Jan 22, 2015 at 11:12 AM, Zijing Guo alter...@yahoo.com.invalid wrote: HiI'm using Apache

Re: Apache Spark broadcast error: Error sending message as driverActor is null [message = UpdateBlockInfo(BlockManagerId

Hi Zijing, Sounds like you sent to the wrong mailing list :P Guozhang On Thu, Jan 22, 2015 at 11:12 AM, Zijing Guo alter...@yahoo.com.invalid wrote: HiI'm using Apache Spark 1.1.0 and I'm currently having issue with broadcast method. So when I call broadcast function on a small dataset to a

kafka production server test

Hi, All We are about to deliver kafka production server, I have been working on different test, like performance test from linkedin. This is a 3-node cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I need to do, like network, node failure, flush time, etc. Is there is completed

Re: Help: Kafka LeaderNotAvailableException

Vishal, Does this error happen every time you are sending? Or just the first time? Joe Stein On Thu, Jan 22, 2015 at 4:33 AM, vishal_ja...@dell.com wrote: Hi, Let me overview on the issue that I am facing on producing message in Kafka: I have horthonworks HDP-2.1 installed, along that we

Counting # of Message Brokers Receive Per Minute Per Topic

Hi Kafka Team, I need to count message received by entire Kafka Broker Cluster for a particular topic. I have 3 brokers, so do I need to sum the COUNT metric or just one server count reflect all server count. It seems that count is always increasing (although metric name is *MessagesInPerSec*

kafka sending duplicate content to consumer

Hi, i want know in which situation does kafka send same event multiple times to consumer. Is there a consumer side configuration to tell kafka to send only once and stop retries? -- Regards Tousif Khazi