Kafka running on Ceph

2016-05-23 Thread Connie Yang
Hi All, Does anyone have any performance metrics running Kafka on Ceph? I briefly gathered at the 2016 Kafka Summit that there's an ongoing work between the Kafka community and RedHat in getting Kafka running successfully on Ceph. Is this correct? If so, what's timeline for that? Thanks Connie

Re: Support customized security protocol

2016-01-19 Thread Connie Yang
@Ismael, what's the status of the SASL/PLAIN PR, https://github.com/apache/kafka/pull/341? On Tue, Jan 19, 2016 at 6:25 PM, tao xiao wrote: > The PR provides a new SASL mech but it doesn't provide a pluggable way to > implement user's own logic to do authentication. So I don't think the PR > w

Re: Got java.util.IllegalFormatConversionException when running MirrorMaker off trunk code

2015-03-07 Thread Connie Yang
When will KAFKA-1997 be available? Thanks Connie On Sat, Mar 7, 2015 at 12:48 AM, Jiangjie Qin wrote: > Hi Tao, > > Thanks a lot for finding the bug. We are actually rewriting the mirror > maker in KAFKA-1997 with a much simplified solution using the newly added > flush() call in new java produ

Re: [VOTE] 0.8.2.1 Candidate 1

2015-02-18 Thread Connie Yang
+1 On Feb 18, 2015 7:23 PM, "Matt Narrell" wrote: > +1 > > > On Feb 18, 2015, at 7:56 PM, Jun Rao wrote: > > > > This is the first candidate for release of Apache Kafka 0.8.2.1. This > > only fixes one critical issue (KAFKA-1952) in 0.8.2.0. > > > > Release Notes for the 0.8.2.1 release > > > ht

Immediate Kafka security solution before 0.9 release

2015-02-17 Thread Connie Yang
Hi All, Before Kafka 0.9 release is available, is there an immediate security solution that we can leverage? I've come across https://github.com/relango/kafka/tree/kafka_security and the IP address filter patch from Kafka 0.8.3, which has not have a set release date. Thanks, Connie

How to upgrade Kafka broker to a newer version without impacting live traffic?

2014-08-27 Thread Connie Yang
Hi All, If I want to upgrade from Kafka 0.8.1.1 to a newer version, what are the steps that I need to do without impacting production traffic? More specifically, how do I have the new brokers to continue writing to the existing logs assuming the file format has not changed? Is there a migration

Re: How to recover from a disk full situation in Kafka cluster?

2014-07-21 Thread Connie Yang
s gone through some level of testing now > and we will appreciate any feedback/bugs that you can report back. > > Thanks, > Neha > > > On Fri, Jul 18, 2014 at 4:23 PM, Connie Yang > wrote: > > > Sure, I will try to take a snapshot of the data dis

Re: How to recover from a disk full situation in Kafka cluster?

2014-07-18 Thread Connie Yang
oducer > sticks to certain partitions, is 10 mins. So it effectively does not > distribute data evenly across all partitions. > > If you see the same behavior next time, try to take a snapshot of data > distribution across all partitions to verify this theory. > > Thanks, > N

Re: How to recover from a disk full situation in Kafka cluster?

2014-07-17 Thread Connie Yang
broker, one at a time. It is also preferrable to run preferred replica > election after a complete cluster bounce so the leaders are well > distributed. > > Also, it will help if you can send around the output of the describe topic > tool. I wonder if your topics have a replication factor

How to recover from a disk full situation in Kafka cluster?

2014-07-17 Thread Connie Yang
Hi All, Our Kafka cluster ran out of disk space yesterday. After we freed up the cluster disk space and adjusted the broker data retention policy, we noticed that the cluster partition was not balanced based on topic describe script came from Kafka 0.8.1.1 distribution. So, we tried to rebalance

What happens to Kafka when ZK lost its quorum or becomes unstable?

2014-05-13 Thread Connie Yang
Hi, Can the producers, brokers and consumers still be processing messages when their ZK cluster lost its quorum or becomes unstable? I know this is rather general question as it may depends on what configuration these use. So, please enumerate all of those combinations. Thanks, Connie

What happens to Kafka when ZK lost its quorum?

2014-05-13 Thread Connie Yang
Hi all, Can Kafka producers, brokers and consumers still be processing messages and functioning in their normal states if Zookeeper lost its quorum? Thanks, Connie