how to order message between different partition

Hi all, I have a topic with 3 partitions, and each partion has its sequency in kafka. How to order message between different partion? Anyone has experiences about that, thanks. Thanks & Regards YuanJia Li

Re: how to order message between different partition

The messages are ordered per partition. No order between partitions. If you really need ordering use one partition. 2015-01-08 9:44 GMT+01:00 YuanJia Li : > Hi all, > I have a topic with 3 partitions, and each partion has its sequency in > kafka. > How to order message between different partio

RE: The message would be consumed if message "un-committed"

so consumer can not read the offset between leader.hw and min(followers.leo)? Thanks， Lax > Date: Tue, 6 Jan 2015 08:44:39 -0800 > Subject: Re: The message would be consumed if message "un-committed" > From: gharatmayures...@gmail.com > To: users@kafka.apache.org > > Only Committed messages are

Re: latency - how to reduce?

I would like to test locally first as it is easier than setting up a test cluster to model the production, yet the script kafka-producer-perf-test is not available for windows. Jun, what kind of "basic I/O testing on the local FS" did you have in mind? Thanks, Shlomi On Tue, Jan 6, 2015 at 5:40

Re: latency - how to reduce?

I do see the Windows based scripts in the tar file - but haven't them though.You should find them under bin/windows. Also you can always use other Windows stress testing tools/suites to check your local I/O performance.. From: Shlomi Hazan To: users@kafka.apache.org; Jayesh Thakrar Sen

Re: The message would be consumed if message "un-committed"

The consumer can only read up to leader High water mark. Thanks, Mayuresh On Thu, Jan 8, 2015 at 2:53 AM, chenlax wrote: > so consumer can not read the offset between leader.hw and > min(followers.leo)? > > Thanks， > Lax > > > > Date: Tue, 6 Jan 2015 08:44:39 -0800 > > Subject: Re: The message

kafka monitoring

Hello, All I understand many of you are using jmxtrans along with graphite/ganglia to pull out metrics, according to https://kafka.apache.org/081/ops.html, it says "The easiest way to see the available metrics to fire up jconsole and point it at a running kafka client or server; this will all bro

Re: NotLeaderForPartitionException while doing performance test

Thanks, Jaikiran I was trying to duplicate the same issue by running the same performance test on master node of cluster , say exemplary-birds.master, and I did see such error again org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partitio

Re: kafka monitoring

Hi, you need to set jmx remote port. you can set this by executing below line in terminal and start server. (or) add below line to kafka-run-class.sh and start server. export JMX_PORT= (jmx remote port) and connect jconsole by giving brokerip: On Fri, Jan 9, 2015 at 12:38 AM,

Re: kafka monitoring

You need to export the JMX_PORT for kafka to use on startup before starting up /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop **

Re: kafka monitoring

Hi Sa Li, You need to set environment variable $JMX_PORT to enable jmx while starting kafka. See to kafka-run-class.sh on how it is used. Then you can connect to : using Jconsole. Thanks, Raja. On Thu, Jan 8, 2015 at 2:08 PM, Sa Li wrote: > Hello, All > > I understand many of you are using jm

Monitor data to missing topic

Hi all, I'm trying to find an existing metric, or method to monitor for data being sent to a topic that doesn't exist. Is there a JMX stat I should look at now, or some other way you catch this sort of thing? We'd like to get ahead of our users noticing a missing topic, or sending data to a typo

RE: kafka monitoring

Is there a firewall between your DEV and PROD environments? If so you will need to open access on all ports, not just JMX port. It gets complicated with JMX. Gene Robichaux Manager, Database Operations Match.com 8300 Douglas Avenue I Suite 800 I Dallas, TX  75225 -Original Message-

RE: The message would be consumed if message "un-committed"

The consumer can only read up to leader High water mark. Thanks, Mayuresh From: chenlax [lax...@hotmail.com] Sent: Thursday, January 08, 2015 2:53 AM To: users kafka Subject: RE: The message would be consumed if message "un-committed" so consumer can not

Offset backup and restore in Kafka v0.8.2

We currently checkpoint partition offsets from ZK into an external database, and we have a process for restoring those offsets to ZK in case we ever need to replay data from Kafka. Since offsets are now stored on the broker, the ConsumerOffsetChecker no longer works. Is there a new way (tool) to d

Re: kafka monitoring

Thank you very much for all the reply, I am able to connect jconsole now, by set env JMX_PORT= start server. However, when I connect it I found there is a port conflict with the kafka-run-class.sh, Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 999

Re: kafka monitoring

In addition, I found all the attributes in jconsole MBeans are cool, but not being graphed, so again, if I want to view the real-time graphing, jmxtrans + graphite is the solution? thanks AL On Thu, Jan 8, 2015 at 1:35 PM, Sa Li wrote: > Thank you very much for all the reply, I am able to conn

Re: Offset backup and restore in Kafka v0.8.2

There is a patch for KAFKA-1013 that has a tool for importing and exporting offsets to and from kafka. Using that tool you can write topic/partion/offset to a file and import it back to kafka. Thanks, Mayuresh On Thu, Jan 8, 2015 at 1:21 PM, Kyle Banker wrote: > We currently checkpoint partiti

Producer exception handling with required.ack = -1

Hi All, Just want to double check with you regarding producers using required.acks = -1. - Producer is guaranteed to receive responses within certain time. Because the satisfied request checking is for in-sync replica. If some replica got stuck, it will be removed from in-sync replica an

Re: Producer exception handling with required.ack = -1

Yes. That's correct. Thanks, Mayuresh Sent from my iPhone > On Jan 8, 2015, at 2:55 PM, Xiaoyu Wang wrote: > > Hi All, > > Just want to double check with you regarding producers using required.acks > = -1. > > - Producer is guaranteed to receive responses within certain time. > Because

Re: Consumer and offset management support in 0.8.2 and 0.9

I took a stab at fixing this issue in KAFKA-1841. This is actually a bit tricky to fix. In addition to the incompatible wire protocol change of version 0 of OffsetCommitRequest, there is also a functionality change. Specifically, in 0.8.1, version 0 of OffsetCommitRequest writes to ZK and version

zookeeper monitoring

Hi, all I've just figured out the monitoring of kafka by jconsole, I want to do the same thing to zookeeper. Zookeeper site says "The class *org.apache.zookeeper.server.quorum.QuorumPeerMain* will start a JMX manageable ZooKeeper server. This class registers the proper MBeans during initalization

RE: The message would be consumed if message "un-committed"

the offset between leader.hw and leader.leo(leo > hw) can not be consumed by consumer,is it? so messages can not be consumed if follower does not pull them on time(the follower in ISR list)? Thanks， Lax > From: mgha...@linkedin.com.INVALID > To: users@kafka.apache.org > Subject: RE: The messag

Re: Consumer and offset management support in 0.8.2 and 0.9

This approach makes sense to me -- thanks, Jun. -Dana On Thu, Jan 8, 2015 at 3:57 PM, Jun Rao wrote: > I took a stab at fixing this issue in KAFKA-1841. > > This is actually a bit tricky to fix. In addition to the incompatible wire > protocol change of version 0 of OffsetCommitRequest, there is

Re: Follow-up On Important Issues for 0.8.2

Adding User Community to see if any one knows behavior of Producer for issue #1) and status of 2). Thanks, Bhavesh On Fri, Jan 2, 2015 at 12:37 PM, Bhavesh Mistry wrote: > Hi Kafka Dev Team, > > I am following-up with you guys regarding New (Java) Producer behavior in > event of network or fi

Re: kafka deleted old logs but not released

Create a JIRA for this https://issues.apache.org/jira/browse/KAFKA-1853 -Jaikiran On Thursday 08 January 2015 01:18 PM, Jaikiran Pai wrote: Apart from the fact that the file rename is failing (the API notes that there are chances of the rename failing), it looks like the implementation in FileM