Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

[Jason Rosenberg]

I can confirm that the per topic metrics are not coming through to the
yammer metrics registry.  I do see them in jmx (via jconsole), but the
MetricsRegistry does not have them.
All the other metrics are coming through that appear in jmx.

This is with single node instance running locally.

Jason



On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy 
wrote:

> If you are using multi-node cluster, then metrics may be reported from
> other servers.
> pl check all the servers in the cluster.
>
> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker  wrote:
>
> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages
> in
> > and out for all topics together and for each individual topic.
> >
> > After upgrading to v0.8.2.0, these metrics are no longer being reported.
> >
> > I'm only seeing the following:
> > BrokerTopicMetrics
> > - BytesInPerSec
> > - BytesOutPerSec
> > - BytesRejectedPerSec
> > - MessagesInPerSec
> >
> > What's more, despite lots of successful writes to the cluster, the values
> > for these remaining metrics are all zero.
> >
> > I saw that there was some refactoring of metric naming code. Was the
> > behavior supposed to have changed?
> >
> > Many thanks in advance.
> >
>

Can't create a topic; can't delete it either

[Sumit Rangwala]

While running kafka in production I found an issue where a topic wasn't
getting created even with auto topic enabled. I then went ahead and created
the topic manually (from the command line). I then delete the topic, again
manually. Now my broker won't allow me to either create *the* topic or
delete *the* topic. (other topic creation and deletion is working fine).

The topic is in "marked for deletion" stage for more than 3 hours.

$ bin/kafka-topics.sh --zookeeper zookeeper1:2181/replication/kafka  --list
--topic GRIFFIN-TldAdFormat.csv-1422321736886
GRIFFIN-TldAdFormat.csv-1422321736886 - marked for deletion

If this is a known issue, is there a workaround?

Sumit

Re: kafka production server test

[Guozhang Wang]

Hmm, so you just want to start producer / consumer clients which points to
the existing brokers in production. I think what you described here should
work, but I may be wrong. Please let me know if that does not work.

Guozhang



On Mon, Jan 26, 2015 at 1:50 PM, Sa Li  wrote:

> Thank you for reply Guozhang, right now I can get it work out of box, run
> testcase_1 on VM and access production. However, from my point of view, we
> really like to test the existing configs on productions, which means for
> example, replica_basic_test.py won't start zookeeper&kafka since I want to
> test the started brokers. I am thinking if I can comment out this part:
>
> self.log_message("starting zookeepers")
>
> kafka_system_test_utils.start_zookeepers(self.systemTestEnv,
> self.testcaseEnv)
> self.anonLogger.info("sleeping for 2s")
> time.sleep(2)
> self.log_message("starting brokers")
> kafka_system_test_utils.start_brokers(self.systemTestEnv,
> self.testcaseEnv)
> self.anonLogger.info("sleeping for 5s")
> time.sleep(5)
>
> Now I plan to modify the properties files in
> /system_test/replication_testsuite/config/, and cluster_config.json and
> testcase_1_properties.json on
> /system_test/replication_testsuite/testcase_1/, to make these config files
> exactly same as what we have on production. Does it work or I need to
> change some other dependencies to get it work?
>
> thanks
>
> AL
>
> On Mon, Jan 26, 2015 at 12:16 PM, Guozhang Wang 
> wrote:
>
> > Sa,
> >
> > I believe your questions have mostly been answered by Ewen, and sorry for
> > getting late on this.
> >
> > As you notice the current system test's out-of-the-box experience is not
> > very good, and we are proposing ways to improve that situation:
> >
> > KAFKA-1748 
> >
> > KAFKA-1589 
> >
> > And we are adding some more test cases at the same time:
> >
> > KAFKA-1888 
> >
> > So if you have new observations while using the package or if you are
> > willing to contribute to those tickets you are mostly welcomed.
> >
> > Guozhang
> >
> >
> > On Thu, Jan 22, 2015 at 3:02 PM, Sa Li  wrote:
> >
> > > Hi, Guozhang
> > >
> > > Can I run this package remotely test another server? which mean I run
> > this
> > > package on dev but testing kafka system on production?
> > >
> > > thanks
> > >
> > > AL
> > >
> > > On Thu, Jan 22, 2015 at 2:55 PM, Sa Li  wrote:
> > >
> > > > Hi, Guozhang,
> > > >
> > > > Good to know such package, will try it now. :-)
> > > >
> > > > thanks
> > > >
> > > > On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > >> Hi Sa,
> > > >>
> > > >> Have you looked into the system test package? It contains a suite of
> > > tests
> > > >> on different failure modes of Kafka brokers.
> > > >>
> > > >> Guozhang
> > > >>
> > > >>
> > > >> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li  wrote:
> > > >>
> > > >> > Hi, All
> > > >> >
> > > >> > We are about to deliver kafka production server, I have been
> working
> > > on
> > > >> > different test, like performance test from linkedin. This is a
> > 3-node
> > > >> > cluster, with 5 nodes zkEnsemble. I assume there are lots of
> tests I
> > > >> need
> > > >> > to do, like network, node failure, flush time, etc. Is there is
> > > >> completed
> > > >> > guide to instruct the tests for kafka production servers?
> > > >> >
> > > >> > thanks
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Alec Li
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> -- Guozhang
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Alec Li
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Alec Li
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
> Alec Li
>



-- 
-- Guozhang

Kafka consumer connection error

[Mahesh Kumar]

Hi all,   I am currently working on logstash to mongodb.logstash's input 
(producer) is kafka and output (consumer) is mongodb.It is worked fine for past 
two days.Today it is not working and i got the following error.i googled this 
error,still didnt find any correct solution.kindly help me to resolve.
[2015-01-24 16:16:13,196] INFO [ReplicaFetcherManager on broker 0] Removed 
fetcher for partitions [kafka3,0],[kafka3,1] 
(kafka.server.ReplicaFetcherManager)[2015-01-24 16:16:17,443] ERROR Closing 
socket for /10.10.10.25 because of error 
(kafka.network.Processor)java.io.IOException: Connection reset by peerat 
sun.nio.ch.FileDispatcherImpl.read0(Native Method)at 
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)at 
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)at 
sun.nio.ch.IOUtil.read(IOUtil.java:197)at 
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)at 
kafka.utils.Utils$.read(Utils.scala:375)at 
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)at
 kafka.network.Processor.read(SocketServer.scala:347)at 
kafka.network.Processor.run(SocketServer.scala:245)at 
java.lang.Thread.run(Thread.java:744)[2015-01-24 16:16:55,956] INFO Closing 
socket connection to /10.10.10.25. (kafka.network.Processor)[2015-01-24 
16:19:49,285] ERROR Closing socket for /10.10.10.25 because of error 
(kafka.network.Processor)java.io.IOException: Connection reset by peerat 
sun.nio.ch.FileDispatcherImpl.read0(Native Method)at 
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)at 
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)at 
sun.nio.ch.IOUtil.read(IOUtil.java:197)at 
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)at 
kafka.utils.Utils$.read(Utils.scala:375)at 
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)at
 kafka.network.Processor.read(SocketServer.scala:347)at 
kafka.network.Processor.run(SocketServer.scala:245)at 
java.lang.Thread.run(Thread.java:744)[2015-01-24 16:20:18,100] INFO Closing 
socket connection to /10.10.10.25. (kafka.network.Processor)[2015-01-24 
16:21:30,156] ERROR Closing socket for /10.10.10.25 because of error 
(kafka.network.Processor)java.io.IOException: Connection reset by peerat 
sun.nio.ch.FileDispatcherImpl.read0(Native Method)at 
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)at 
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)at 
sun.nio.ch.IOUtil.read(IOUtil.java:197)at 
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)at 
kafka.utils.Utils$.read(Utils.scala:375)at 
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)at
 kafka.network.Processor.read(SocketServer.scala:347)at 
kafka.network.Processor.run(SocketServer.scala:245)at 
java.lang.Thread.run(Thread.java:744)[2015-01-24 17:49:56,224] INFO Closing 
socket connection to /10.10.10.25. (kafka.network.Processor)
ThanksMahesh.S

Re: Kafka Out of Memory error

[Pranay Agarwal]

Thanks a lot Gwen. I bumped up the JVM to 1g on the consumer side and it
works :)

All the consumer belong to the same group and I am using the High level
group API to consume from the kafka. It seems there is some initial meta
data exchange or something about all the partitions are sent to all the
consumer. Also, I launch 10 consumer from each machine at a time and keep
on adding till 200 consumers. I see that initial consumers seem to require
initially lot more cpu and memory. Should I launch all the consumers at one
go instead of adding 10 at a time?

On different issue, I couldn't find anyway of keeping the current read
offset metadata while using the High level API( I am using the gem
mentioned in earlier mail). Is there anyway to record the current read
metadata periodically to monitor the progress of the consumer. Further,
everytime a consumer dies and restart it seems to start reading from the
beginning, is there anyway to read from last read offsets only?

Thanks
-Pranay

On Mon, Jan 19, 2015 at 6:54 PM, Gwen Shapira  wrote:

> Two things:
> 1. The OOM happened on the consumer, right? So the memory that matters
> is the RAM on the consumer machine, not on the Kafka cluster nodes.
>
> 2. If the consumers belong to the same consumer group, each will
> consume a subset of the partitions and will only need to allocate
> memory for those partitions.
>
> So, assuming all your consumers belong to the same group:
> 2 consumers  -> each has 500 partitions -> each uses 500MB.
>
> The total remains 1GB no matter how many consumers you have, as long
> as they are all in the same group.
>
> If the consumer belong to different groups (i.e. they read copies of
> the same messages from the same partitions), then yes, you are limited
> to 8 per server (probably less because there are other stuff on the
> server).
>
> Gwen
>
> On Mon, Jan 19, 2015 at 3:06 PM, Pranay Agarwal
>  wrote:
> > Thanks a lot Natty.
> >
> > I am using this Ruby gem on the client side with all the default config
> >
> https://github.com/joekiller/jruby-kafka/blob/master/lib/jruby-kafka/group.rb
> > and the value fetch.message.max.bytes is set to 1MB.
> >
> > Currently I only have 3 nodes setup in the Kafka cluster (with 8 GB RAM)
> > and if 1 consumer if going to take 1000 partitions X 1mb ~ 1GB, does it
> > mean 1 kafka node can at best support 8 consumer only? Also, when I do
> > top/free on the Kafka cluster nodes (Both zookeeper and kafka is deployed
> > on each 3 nodes of the cluster) I don't see lots of memory being used on
> > the machine. Also, even with this calculation, I shouldn't be facing any
> > issue with only 1 consumer, as I have 8GB of JVM space given to Kafka
> > nodes, right?
> >
> > Thanks
> > -Pranay
> >
> > On Mon, Jan 19, 2015 at 2:53 PM, Jonathan Natkins 
> > wrote:
> >
> >> The fetch.message.max.size is actually a client-side configuration. With
> >> regard to increasing the number of threads, I think the calculation may
> be
> >> a little more subtle than what you're proposing, and frankly, it's
> unlikely
> >> that your servers can handle allocating 200MB x 1000 threads = 200GB of
> >> memory at a single time.
> >>
> >> I believe that if you have every partition on a single broker, and all
> of
> >> your consumer threads are requesting data simultaneously, then yes, the
> >> broker would attempt to allocate 200GB of heap, and probably you'll hit
> an
> >> OOME. However, since each consumer is only reading from one partition,
> >> those 1000 threads should be making requests that are spread out over
> the
> >> entire Kafka cluster. Depending on the memory on your servers, you may
> need
> >> to increase the number of brokers in your cluster to support the 1000
> >> threads. For example, I would expect that you can support this with 10
> >> brokers if each broker has something north of 20GB of heap allocated.
> >>
> >> Some of this is a little bit of guess work on my part, and I'm not super
> >> confident of my numbers...Can anybody else on the list validate my math?
> >>
> >> Thanks,
> >> Natty
> >>
> >> Jonathan "Natty" Natkins
> >> StreamSets | Customer Engagement Engineer
> >> mobile: 609.577.1600 | linkedin 
> >>
> >>
> >> On Mon, Jan 19, 2015 at 2:34 PM, Pranay Agarwal <
> agarwalpran...@gmail.com>
> >> wrote:
> >>
> >> > Thanks Natty.
> >> >
> >> > Is there any config which I need to change on the client side as well?
> >> > Also, currently I am trying with only 1 consumer thread. Does the
> >> equation
> >> > changes to
> >> > (#partitions)*(fetchsize)*(#consumer_threads) in case I try to read
> with
> >> > 1000 threads from from topic2(1000 partitions)?
> >> >
> >> > -Pranay
> >> >
> >> > On Mon, Jan 19, 2015 at 2:26 PM, Jonathan Natkins <
> na...@streamsets.com>
> >> > wrote:
> >> >
> >> > > Hi Pranay,
> >> > >
> >> > > I think the JIRA you're referencing is a bit orthogonal to the OOME
> >> that
> >> > > you're experiencing. Based on the stacktrace, it looks like your

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

[Manikumar Reddy]

If you are using multi-node cluster, then metrics may be reported from
other servers.
pl check all the servers in the cluster.

On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker  wrote:

> I've been using a custom KafkaMetricsReporter to report Kafka broker
> metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in
> and out for all topics together and for each individual topic.
>
> After upgrading to v0.8.2.0, these metrics are no longer being reported.
>
> I'm only seeing the following:
> BrokerTopicMetrics
> - BytesInPerSec
> - BytesOutPerSec
> - BytesRejectedPerSec
> - MessagesInPerSec
>
> What's more, despite lots of successful writes to the cluster, the values
> for these remaining metrics are all zero.
>
> I saw that there was some refactoring of metric naming code. Was the
> behavior supposed to have changed?
>
> Many thanks in advance.
>

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Otis Gospodnetic]

Hi,

Don't use Graphite, so I don't know.  Kyle, maybe you can share more info?
What do you mean by "reported to Yammer" for example?  And when you say
Yammer/Graphite, are you trying to say that you are using the Graphite
Reporter?  If so, can you try other Yammer Reporters and see if there is a
more general issue or something limited to either Graphite or Graphite
Reporter?

I am pretty sure we are able to see all Kafka 0.8.2 metrics nicely in SPM
(in non-public version of the Kafka monitoring agent).

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Mon, Jan 26, 2015 at 7:37 PM, Jun Rao  wrote:

> Hmm, that's not the intention. The per-topic mbeans are definitely
> registered by Yammer. So, not sure why it's not reported to Graphite.
>
> Otis, Vladimir,
>
> Do you guys know?
>
> Thanks,
>
> Jun
>
>
>
> On Mon, Jan 26, 2015 at 4:08 PM, Kyle Banker  wrote:
>
> > This is still preliminary, but it looks as if the change to metric names
> > for per-topic metrics (bytes/messages in/out) is preventing these metrics
> > from being reported to Yammer/Graphite. If this isn't intentional, it
> > should probably be addressed before release.
> >
> > On Wed, Jan 21, 2015 at 9:28 AM, Jun Rao  wrote:
> >
> > > This is the second candidate for release of Apache Kafka 0.8.2.0. There
> > has
> > > been some changes since the 0.8.2 beta release, especially in the new
> > java
> > > producer api and jmx mbean names. It would be great if people can test
> > this
> > > out thoroughly.
> > >
> > > Release Notes for the 0.8.2.0 release
> > >
> > >
> >
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Monday, Jan 26h, 7pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2
> > > (SHA256)
> > > checksum.
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/
> > >
> > > * Maven artifacts to be voted upon prior to release:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * scala-doc
> > > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/scaladoc/
> > >
> > > * java-doc
> > > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/javadoc/
> > >
> > > * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 tag
> > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=058d58adef2ab2787e49d8efeefd61bb3d32f99c
> > > (commit 0b312a6b9f0833d38eec434bfff4c647c1814564)
> > >
> > > /***
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> >
>

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Jun Rao]

Kyle,

Also, which reporter are you using for Graphite?

Thanks,

Jun

On Mon, Jan 26, 2015 at 4:08 PM, Kyle Banker  wrote:

> This is still preliminary, but it looks as if the change to metric names
> for per-topic metrics (bytes/messages in/out) is preventing these metrics
> from being reported to Yammer/Graphite. If this isn't intentional, it
> should probably be addressed before release.
>
> On Wed, Jan 21, 2015 at 9:28 AM, Jun Rao  wrote:
>
> > This is the second candidate for release of Apache Kafka 0.8.2.0. There
> has
> > been some changes since the 0.8.2 beta release, especially in the new
> java
> > producer api and jmx mbean names. It would be great if people can test
> this
> > out thoroughly.
> >
> > Release Notes for the 0.8.2.0 release
> >
> >
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, Jan 26h, 7pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2
> > (SHA256)
> > checksum.
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > * scala-doc
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/scaladoc/
> >
> > * java-doc
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/javadoc/
> >
> > * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 tag
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=058d58adef2ab2787e49d8efeefd61bb3d32f99c
> > (commit 0b312a6b9f0833d38eec434bfff4c647c1814564)
> >
> > /***
> >
> > Thanks,
> >
> > Jun
> >
>

Re: [DISCUSSION] Boot dependency in the new producer

[Steven Wu]

Jay, I don't think this line will bootstrap full metadata (for all topics).
it will just construct the cluster object with bootstrap host. you need to
do "metadata.add(topic)"  to set interest of a topic's partition metadata.

Guozhang, I personally think this is ok. it just do a few DNS lookup or TCP
connection before first send.

On Mon, Jan 26, 2015 at 2:07 PM, Jay Kreps  wrote:

> Oh, yes, I guess I thought you meant that construction of the client would
> block on the metadata request.
>
> I don't personally think that is a problem because if it fails it will
> retry in the background, right?
>
> But actually I think this is probably violating another desirable criteria
> we had talked about which was keeping the producer from bootstrapping the
> full metadata for all partitions. If it is doing that during construction
> time presumably the resulting metadata request is for all partitions, no?
> That isn't a huge problem, but I think isn't what was intended.
>
> -Jay
>
> On Mon, Jan 26, 2015 at 1:34 PM, Guozhang Wang  wrote:
>
> > It will set the needUpdate flag to true and hence the background Sender
> > will try to talk to the bootstrap servers.
> >
> > Guozhang
> >
> > On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps  wrote:
> >
> > > Hey Guozhang,
> > >
> > > That line shouldn't cause any connections to Kafka to be established,
> > does
> > > it? All that is doing is creating the Cluster pojo using the supplied
> > > addresses. The use of InetSocketAddress may cause some dns stuff to
> > happen,
> > > though...
> > >
> > > -Jay
> > >
> > > On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am not sure if we have discussed about this before, but recently I
> > > > realized that we have introduced boot dependency of the kafka-server
> > > > specified by the "bootstrap.servers" config in the new producer. More
> > > > specifically, although in the old producer we also have a similar
> > config
> > > > for specifying the broker list, the producer will not try to connect
> to
> > > > those brokers until the first message send call is triggered; whereas
> > in
> > > > the new producer, it will try to talk to them in construction time
> via:
> > > >
> > > > update(Cluster.bootstrap(addresses), time.milliseconds());
> > > >
> > > >
> > > > I personally am neutral to this change, as in most cases the
> > > corresponding
> > > > kafka server should be up and running before the producer clients are
> > > > deployed, but there are still some corner cases when it is not true,
> > for
> > > > example some standalone deployment tests of the app embedded with
> some
> > > > clients, etc. So I would like to bring this up to people's attention
> if
> > > we
> > > > have not discussed about it before: do we think this is OK to
> introduce
> > > > this boot dependency in the new producer?
> > > >
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Jun Rao]

Hmm, that's not the intention. The per-topic mbeans are definitely
registered by Yammer. So, not sure why it's not reported to Graphite.

Otis, Vladimir,

Do you guys know?

Thanks,

Jun



On Mon, Jan 26, 2015 at 4:08 PM, Kyle Banker  wrote:

> This is still preliminary, but it looks as if the change to metric names
> for per-topic metrics (bytes/messages in/out) is preventing these metrics
> from being reported to Yammer/Graphite. If this isn't intentional, it
> should probably be addressed before release.
>
> On Wed, Jan 21, 2015 at 9:28 AM, Jun Rao  wrote:
>
> > This is the second candidate for release of Apache Kafka 0.8.2.0. There
> has
> > been some changes since the 0.8.2 beta release, especially in the new
> java
> > producer api and jmx mbean names. It would be great if people can test
> this
> > out thoroughly.
> >
> > Release Notes for the 0.8.2.0 release
> >
> >
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, Jan 26h, 7pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2
> > (SHA256)
> > checksum.
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > * scala-doc
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/scaladoc/
> >
> > * java-doc
> > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/javadoc/
> >
> > * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 tag
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=058d58adef2ab2787e49d8efeefd61bb3d32f99c
> > (commit 0b312a6b9f0833d38eec434bfff4c647c1814564)
> >
> > /***
> >
> > Thanks,
> >
> > Jun
> >
>

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Kyle Banker]

This is still preliminary, but it looks as if the change to metric names
for per-topic metrics (bytes/messages in/out) is preventing these metrics
from being reported to Yammer/Graphite. If this isn't intentional, it
should probably be addressed before release.

On Wed, Jan 21, 2015 at 9:28 AM, Jun Rao  wrote:

> This is the second candidate for release of Apache Kafka 0.8.2.0. There has
> been some changes since the 0.8.2 beta release, especially in the new java
> producer api and jmx mbean names. It would be great if people can test this
> out thoroughly.
>
> Release Notes for the 0.8.2.0 release
>
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Jan 26h, 7pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2
> (SHA256)
> checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/javadoc/
>
> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=058d58adef2ab2787e49d8efeefd61bb3d32f99c
> (commit 0b312a6b9f0833d38eec434bfff4c647c1814564)
>
> /***
>
> Thanks,
>
> Jun
>

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

[Jun Rao]

The jmx beans created by Yammer have been changed to the new names.
However, the global mbeans have different names from the topic level
mbeans. The new mbeans have completely different names from the old ones
since they no longer have quotes. Does that answer your question?

Thanks,

Jun

On Mon, Jan 26, 2015 at 3:38 PM, Kyle Banker  wrote:

> Thanks, Jun. I'm pretty sure, though not 100% confident, that this has
> caused a change in the way that these metrics are reported to Yammer
> metrics (I believe that they are stomping on each other). Is that intended
> as well?
>
> On Mon, Jan 26, 2015 at 4:13 PM, Jun Rao  wrote:
>
> > Yes, we refactored the mbean names in 0.8.2.0 to make them more standard.
> > Those metrics are now registered under the following names. I did some
> > quick test and the values do get updated.
> >
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=test
> >
> > The full list of the refactored mbean names can be found at
> > http://kafka.apache.org/082/documentation.html#monitoring
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jan 26, 2015 at 2:42 PM, Kyle Banker 
> wrote:
> >
> > > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> messages
> > in
> > > and out for all topics together and for each individual topic.
> > >
> > > After upgrading to v0.8.2.0, these metrics are no longer being
> reported.
> > >
> > > I'm only seeing the following:
> > > BrokerTopicMetrics
> > > - BytesInPerSec
> > > - BytesOutPerSec
> > > - BytesRejectedPerSec
> > > - MessagesInPerSec
> > >
> > > What's more, despite lots of successful writes to the cluster, the
> values
> > > for these remaining metrics are all zero.
> > >
> > > I saw that there was some refactoring of metric naming code. Was the
> > > behavior supposed to have changed?
> > >
> > > Many thanks in advance.
> > >
> >
>

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

[Kyle Banker]

Thanks, Jun. I'm pretty sure, though not 100% confident, that this has
caused a change in the way that these metrics are reported to Yammer
metrics (I believe that they are stomping on each other). Is that intended
as well?

On Mon, Jan 26, 2015 at 4:13 PM, Jun Rao  wrote:

> Yes, we refactored the mbean names in 0.8.2.0 to make them more standard.
> Those metrics are now registered under the following names. I did some
> quick test and the values do get updated.
>
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
> kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=test
>
> The full list of the refactored mbean names can be found at
> http://kafka.apache.org/082/documentation.html#monitoring
>
> Thanks,
>
> Jun
>
>
> On Mon, Jan 26, 2015 at 2:42 PM, Kyle Banker  wrote:
>
> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages
> in
> > and out for all topics together and for each individual topic.
> >
> > After upgrading to v0.8.2.0, these metrics are no longer being reported.
> >
> > I'm only seeing the following:
> > BrokerTopicMetrics
> > - BytesInPerSec
> > - BytesOutPerSec
> > - BytesRejectedPerSec
> > - MessagesInPerSec
> >
> > What's more, despite lots of successful writes to the cluster, the values
> > for these remaining metrics are all zero.
> >
> > I saw that there was some refactoring of metric naming code. Was the
> > behavior supposed to have changed?
> >
> > Many thanks in advance.
> >
>

Re: Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

[Jun Rao]

Yes, we refactored the mbean names in 0.8.2.0 to make them more standard.
Those metrics are now registered under the following names. I did some
quick test and the values do get updated.

kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=test

The full list of the refactored mbean names can be found at
http://kafka.apache.org/082/documentation.html#monitoring

Thanks,

Jun


On Mon, Jan 26, 2015 at 2:42 PM, Kyle Banker  wrote:

> I've been using a custom KafkaMetricsReporter to report Kafka broker
> metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in
> and out for all topics together and for each individual topic.
>
> After upgrading to v0.8.2.0, these metrics are no longer being reported.
>
> I'm only seeing the following:
> BrokerTopicMetrics
> - BytesInPerSec
> - BytesOutPerSec
> - BytesRejectedPerSec
> - MessagesInPerSec
>
> What's more, despite lots of successful writes to the cluster, the values
> for these remaining metrics are all zero.
>
> I saw that there was some refactoring of metric naming code. Was the
> behavior supposed to have changed?
>
> Many thanks in advance.
>

Missing Per-Topic BrokerTopicMetrics in v0.8.2.0

[Kyle Banker]

I've been using a custom KafkaMetricsReporter to report Kafka broker
metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in
and out for all topics together and for each individual topic.

After upgrading to v0.8.2.0, these metrics are no longer being reported.

I'm only seeing the following:
BrokerTopicMetrics
- BytesInPerSec
- BytesOutPerSec
- BytesRejectedPerSec
- MessagesInPerSec

What's more, despite lots of successful writes to the cluster, the values
for these remaining metrics are all zero.

I saw that there was some refactoring of metric naming code. Was the
behavior supposed to have changed?

Many thanks in advance.

Re: SimpleConsumer leaks sockets on an UnresolvedAddressException

[Rajiv Kurian]

Meant to write a run loop.

void run() {
  while (running) {
if (simpleConsumer == null) {
  simpleConsumer = new SimpleConsumer(host, port,
(int) kafkaSocketTimeout, kafkaRExeiveBufferSize, clientName);
}
try {
  // Do stuff with simpleConsumer.
} catch (Exception e) {
  logger.error(e);  // Assume UnresolvedAddressException.
  if (consumer != null) {
simpleConsumer.close();
simpleConsumer = null;
  }
}
  }
}

On Mon, Jan 26, 2015 at 2:27 PM, Rajiv Kurian  wrote:

> Here is my typical flow:
> void run() {
>   if (simpleConsumer == null) {
> simpleConsumer = new SimpleConsumer(host, port, (int) kafkaSocketTimeout,
> kafkaRExeiveBufferSize, clientName);
>   }
>   try {
> // Do stuff with simpleConsumer.
>} catch (Exception e) {
>  if (consumer != null) {
>simpleConsumer.close();
>simpleConsumer = null;
>  }
>   }
> }
>
> If there is a problem with the host name, or some DNS issues, we get an
> UnresolvedAddressException as expected and attempt to close the
> simpleConsumer. However this does not really get rid of the underlying
> socket. So we end up leaking a FD every time this happens. Though this is
> not a common case I think there needs to be a way on the SimpleConsumer to
> get rid of all OS resources that it is holding onto. Right now if this
> keeps happening the number of FDs being consumed by the process keeps
> increasing till we hit the OS limits. As a user I cannot do anything else
> but call simpleConsumer.close(). We need to be able to close the underlying
> socketChannel/socket when this kind of an error happens.
>
> To reproduce, one can just run this code but just put in any garbage host
> name, running lsof -p while running this will show that the open FDs
> increases without limit.
>
> Thanks,
> Rajiv
>
>

Re: Kafka sending messages with zero copy

[Rajiv Kurian]

Hi Guozhang,

I am a bit busy at work. When I get the change I'll definitely try to get a
proof of concept going. Not the kafka protocol, but just the buffering and
threading structures, maybe just write to another socket. I think it would
be useful just to get the queueing and buffer management going and prove
that it can be done in a zero copy way in a multi producer single consumer
environment. If that is working, then the single consumer can be the kafka
network sync thread.

Thanks,
Rajiv

On Fri, Jan 16, 2015 at 11:58 AM, Guozhang Wang  wrote:

> Hi Rajiv,
>
> Thanks for this proposal, it would be great if you can upload some
> implementation patch for the CAS idea and show some memory usage / perf
> differences.
>
> Guozhang
>
> On Sun, Dec 14, 2014 at 9:27 PM, Rajiv Kurian 
> wrote:
>
> > Resuscitating this thread. I've done some more experiments and profiling.
> > My messages are very tiny (currently 25 bytes) per message and creating
> > multiple objects per message leads to a lot of churn. The memory churn
> > through creation of convenience objects is more than the memory being
> used
> > by my objects right now. I could probably batch my messages further, to
> > make this effect less pronounced.​ I did some rather unscientific
> > experiments with a flyweight approach on top of the ByteBuffer for a
> simple
> > messaging API (peer to peer NIO based so not a real comparison) and the
> > numbers were very satisfactory and there is no garbage created in steady
> > state at all. Though I don't expect such good numbers from actually going
> > through the broker + all the other extra stuff that a real producer would
> > do, I think there is great potential here.
> >
> > The general mechanism for me is this:
> > i) A buffer (I used Unsafe but I imagine ByteBuffer having similar
> > performance) is created per partition.
> > ii) A CAS loop (in Java 7 and less) or even better unsafe.getAndAddInt()
> in
> > Java 8 can be used to claim a chunk of bytes on the per topic buffer.
> This
> > code can be invoked from multiple threads in a wait free manner
> (wait-free
> > in Java 8, since getAndAddInt() is wait-free).  Once a region in the
> buffer
> > is claimed, it can be operated on using the flyweight method that we
> talked
> > about. If the buffer doesn't have enough space then we can drop the
> message
> > or move onto a new buffer. Further this creates absolutely zero objects
> in
> > steady state (only a few objects created in the beginning). Even if the
> > flyweight method is not desired, the API can just take byte arrays or
> > objects that need to be serialized and copy them onto the per topic
> buffers
> > in a similar way. This API has been validated in Aeron too, so I am
> pretty
> > confident that it will work well. For the zero copy technique here is a
> > link to Aeron API with zero copy -
> > https://github.com/real-logic/Aeron/issues/18. The regular one copies
> byte
> > arrays but without any object creation.
> > iii) The producer send thread can then just go in FIFO order through the
> > buffer sending messages that have been committed using NIO to rotate
> > between brokers. We might need a background thread to zero out used
> buffers
> > too.
> >
> > I've left out some details, but again none of this very revolutionary -
> > it's mostly the same techniques used in Aeron. I really think that we can
> > keep the API ga rbage free and wait-free (even in the multi producer
> case)
> > without compromising how pretty it looks - the total zero copy API will
> low
> > level, but it should only be used by advanced users. Moreover the usual
> > producer.send(msg, topic, partition) can use the efficient ByteBuffer
> > offset API internally without it itself creating any garbage. With the
> > technique I talked about there is no need for an intermediate queue of
> any
> > kind since the underlying ByteBuffer per partition acts as the queue.
> >
> > I can do more experiments with some real producer code instead of my toy
> > code to further validate the idea, but I am pretty sure that both
> > throughput and jitter characteristics will improve thanks to lower
> > contention (wait-free in java 8 with a single getAndAddInt() operation
> for
> > sync ) and better cache locality (C like buffers and a few constant
> number
> > of objects per partition). If you guys are interested, I'd love to talk
> > more. Again just to reiterate, I don't think the API will suffer at all -
> > most of this can be done under the covers. Additionally it will open up
> > things so that a low level zero copy API is possible.
> >
> > Thanks,
> > Rajiv
> >
>
>
>
> --
> -- Guozhang
>

SimpleConsumer leaks sockets on an UnresolvedAddressException

[Rajiv Kurian]

Here is my typical flow:
void run() {
  if (simpleConsumer == null) {
simpleConsumer = new SimpleConsumer(host, port, (int) kafkaSocketTimeout,
kafkaRExeiveBufferSize, clientName);
  }
  try {
// Do stuff with simpleConsumer.
   } catch (Exception e) {
 if (consumer != null) {
   simpleConsumer.close();
   simpleConsumer = null;
 }
  }
}

If there is a problem with the host name, or some DNS issues, we get an
UnresolvedAddressException as expected and attempt to close the
simpleConsumer. However this does not really get rid of the underlying
socket. So we end up leaking a FD every time this happens. Though this is
not a common case I think there needs to be a way on the SimpleConsumer to
get rid of all OS resources that it is holding onto. Right now if this
keeps happening the number of FDs being consumed by the process keeps
increasing till we hit the OS limits. As a user I cannot do anything else
but call simpleConsumer.close(). We need to be able to close the underlying
socketChannel/socket when this kind of an error happens.

To reproduce, one can just run this code but just put in any garbage host
name, running lsof -p while running this will show that the open FDs
increases without limit.

Thanks,
Rajiv

Re: does kafka support "COMMIT" of a batch ?

[Yang]

thanks  for the info
yang

On Mon, Jan 26, 2015 at 2:09 PM, Jay Kreps  wrote:

> We did a relatively complete prototype but it isn't integrated into the
> mainline code yet and there isn't a target release date. There is rather a
> lot of testing and compatability work that would have to be done to fully
> productionize it. I suspect someone will pick it up in 2015 but I wouldn't
> block on it.
>
> -Jay
>
> On Mon, Jan 26, 2015 at 1:55 PM, Yang  wrote:
>
> > well I guess this is what I need
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
> >
> > anyone knows how far this effort has gone?
> >
> > thanks
> > Yang
> >
> > On Mon, Jan 26, 2015 at 12:06 PM, Yang  wrote:
> >
> > > we are pulling from a DB, maybe 100k rows at a time. for each row we
> send
> > > it to kafka.
> > >
> > > now the problem is , if any errors happen during the db pulling (such
> as
> > > an earthquake), we stop.
> > > next time we wake up. we can't write the same record again into kafka,
> > > otherwise there might be over counting.
> > >
> > >
> > > does kafka support a "BATCH mode" with commit ? so basically at the
> start
> > > of my session, I declare "start transaction", and after all records
> have
> > > been pulled and sent to kafka, I declare "commit" .
> > >
> > > Thanks
> > > Yang
> > >
> >
>

Re: [DISCUSSION] Boot dependency in the new producer

[Jay Kreps]

Oh, yes, I guess I thought you meant that construction of the client would
block on the metadata request.

I don't personally think that is a problem because if it fails it will
retry in the background, right?

But actually I think this is probably violating another desirable criteria
we had talked about which was keeping the producer from bootstrapping the
full metadata for all partitions. If it is doing that during construction
time presumably the resulting metadata request is for all partitions, no?
That isn't a huge problem, but I think isn't what was intended.

-Jay

On Mon, Jan 26, 2015 at 1:34 PM, Guozhang Wang  wrote:

> It will set the needUpdate flag to true and hence the background Sender
> will try to talk to the bootstrap servers.
>
> Guozhang
>
> On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps  wrote:
>
> > Hey Guozhang,
> >
> > That line shouldn't cause any connections to Kafka to be established,
> does
> > it? All that is doing is creating the Cluster pojo using the supplied
> > addresses. The use of InetSocketAddress may cause some dns stuff to
> happen,
> > though...
> >
> > -Jay
> >
> > On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang 
> > wrote:
> >
> > > Hi all,
> > >
> > > I am not sure if we have discussed about this before, but recently I
> > > realized that we have introduced boot dependency of the kafka-server
> > > specified by the "bootstrap.servers" config in the new producer. More
> > > specifically, although in the old producer we also have a similar
> config
> > > for specifying the broker list, the producer will not try to connect to
> > > those brokers until the first message send call is triggered; whereas
> in
> > > the new producer, it will try to talk to them in construction time via:
> > >
> > > update(Cluster.bootstrap(addresses), time.milliseconds());
> > >
> > >
> > > I personally am neutral to this change, as in most cases the
> > corresponding
> > > kafka server should be up and running before the producer clients are
> > > deployed, but there are still some corner cases when it is not true,
> for
> > > example some standalone deployment tests of the app embedded with some
> > > clients, etc. So I would like to bring this up to people's attention if
> > we
> > > have not discussed about it before: do we think this is OK to introduce
> > > this boot dependency in the new producer?
> > >
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: does kafka support "COMMIT" of a batch ?

[Jay Kreps]

We did a relatively complete prototype but it isn't integrated into the
mainline code yet and there isn't a target release date. There is rather a
lot of testing and compatability work that would have to be done to fully
productionize it. I suspect someone will pick it up in 2015 but I wouldn't
block on it.

-Jay

On Mon, Jan 26, 2015 at 1:55 PM, Yang  wrote:

> well I guess this is what I need
>
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
>
> anyone knows how far this effort has gone?
>
> thanks
> Yang
>
> On Mon, Jan 26, 2015 at 12:06 PM, Yang  wrote:
>
> > we are pulling from a DB, maybe 100k rows at a time. for each row we send
> > it to kafka.
> >
> > now the problem is , if any errors happen during the db pulling (such as
> > an earthquake), we stop.
> > next time we wake up. we can't write the same record again into kafka,
> > otherwise there might be over counting.
> >
> >
> > does kafka support a "BATCH mode" with commit ? so basically at the start
> > of my session, I declare "start transaction", and after all records have
> > been pulled and sent to kafka, I declare "commit" .
> >
> > Thanks
> > Yang
> >
>

Re: unable to delete topic with 0.8.2 rc2

[Sumit Rangwala]

I was planning to start a new thread by my experience is similar to the one
described here. I am currently using 0.8.2-beta and my setup has a producer
producing to topic "DC-DATA", kafka brokers with auto-topic-creation
enabled, and mirrormaker set to replicate any ".*-DATA" topics. After the
producer produces some data another thread in my code goes ahead and
deletes the topic DC-DATA. And I am seeing a similar behavior, i.e, the
topic reappears after deleting. I, however, changed the thread in my code
to wake up periodically and if it finds the topic (using getAllTopics) it
tries to delete the topic. What I found interesting is that after few
delete attempts the topic indeed gets deleted and does not reappear.  Seems
like some interplay between auto-topic creating and consumer notification
when a consumer is subscribed to a topic specified as a regex.


Sumit





On Mon, Jan 26, 2015 at 1:49 PM, Jun Rao  wrote:

> Joel,
>
> That's probably because console consumer always uses wildcard for
> consumption.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy  wrote:
>
> > I think this should work even on non-wildcard consumers. (i.e., I just
> > tried it locally and it appears to work fine).
> >
> > Here is what I did:
> >
> > - Leave a console consumer up and running, bounce the broker to turn
> >   off auto-create
> > - Delete the topic
> > - The consumer should rebalance and stop consuming that topic
> > - Bounce the broker to turn on auto-create
> >
> > The only issue is that while your broker has auto-create off you will
> > see errors on some producers - which you can announce as a brief
> > outage to your users.
> >
> > Joel
> >
> > On Mon, Jan 26, 2015 at 01:02:57PM -0800, Jun Rao wrote:
> > > Actually, I think the deletion of a topic will trigger a rebalance on
> all
> > > wildcard subscribers. The time to complete the rebalance depends on
> the #
> > > of topic/partitions.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg 
> > wrote:
> > >
> > > > Yeah Joel,
> > > >
> > > > I just thought of that idea too (e.g. temporarily disable auto topic
> > > > creation).  I can probably make that work (it may result in
> unexpected
> > lack
> > > > of topic creation, but I can message this as a maintenance downtime,
> > etc.).
> > > >
> > > > So, that was my next question, so a consumer rebalance will trigger a
> > fresh
> > > > topic pull from the consumers?  How long is 'long enough' to ensure a
> > > > rebalance has occurred everywhere?
> > > >
> > > > Jason
> > > >
> > > > On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy 
> > wrote:
> > > >
> > > > > Hey Jason,
> > > > >
> > > > > Is it an option for you to do the following:
> > > > >
> > > > > - Bounce in a config change to the brokers to turn off auto-create
> > > > > - (Batch)-delete the topic(s)
> > > > > - Wait long enough for consumers to rebalance (after which they
> will
> > > > >   no longer consume the topic(s))
> > > > > - Bounce in a config change to the brokers to turn on auto-create
> > > > >
> > > > > Joel
> > > > >
> > > > > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > > > > Hi, Jason,
> > > > > >
> > > > > > I am not sure that we should delay the 0.8.2 release. The reasons
> > are
> > > > (1)
> > > > > > There are other features such as the new java producer and
> > Kafka-based
> > > > > > offset management that are potentially useful to people. (2) It
> may
> > > > take
> > > > > > some time to completely fix the issue with deleting topic since
> it
> > > > > involves
> > > > > > wire protocol changes.
> > > > > >
> > > > > > What we can do is probably just to document the limitation of
> > delete
> > > > > topic
> > > > > > in 0.8.2.0 in our site.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg <
> > j...@squareup.com>
> > > > > wrote:
> > > > > >
> > > > > > > I think this shortcoming should probably delay release of 0.8.2
> > until
> > > > > > > resolved, no?
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang <
> > wangg...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > It then seems to me that delete-topic will not actually work
> > > > > "smoothly"
> > > > > > > > until create topic request is added since it is too much to
> > require
> > > > > > > people
> > > > > > > > to turn off their clients while deleting topics. In this case
> > shall
> > > > > we
> > > > > > > make
> > > > > > > > it clear in the release docs of in 0.8.2, or even still mark
> > it as
> > > > > > > > not-supported?
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha 
> > wrote:
> > > > > > > >
> > > > > > > > > Jun,
> > > > > > > > >   I made an attempt at fixing that issue as part of
> this
> > JIRA
> > > > > > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > > > > > As Jay 

Re: unable to delete topic with 0.8.2 rc2

[Joel Koshy]

Hmm.. that's right. completely forgot about that.

On Mon, Jan 26, 2015 at 01:49:33PM -0800, Jun Rao wrote:
> Joel,
> 
> That's probably because console consumer always uses wildcard for
> consumption.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy  wrote:
> 
> > I think this should work even on non-wildcard consumers. (i.e., I just
> > tried it locally and it appears to work fine).
> >
> > Here is what I did:
> >
> > - Leave a console consumer up and running, bounce the broker to turn
> >   off auto-create
> > - Delete the topic
> > - The consumer should rebalance and stop consuming that topic
> > - Bounce the broker to turn on auto-create
> >
> > The only issue is that while your broker has auto-create off you will
> > see errors on some producers - which you can announce as a brief
> > outage to your users.
> >
> > Joel
> >
> > On Mon, Jan 26, 2015 at 01:02:57PM -0800, Jun Rao wrote:
> > > Actually, I think the deletion of a topic will trigger a rebalance on all
> > > wildcard subscribers. The time to complete the rebalance depends on the #
> > > of topic/partitions.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg 
> > wrote:
> > >
> > > > Yeah Joel,
> > > >
> > > > I just thought of that idea too (e.g. temporarily disable auto topic
> > > > creation).  I can probably make that work (it may result in unexpected
> > lack
> > > > of topic creation, but I can message this as a maintenance downtime,
> > etc.).
> > > >
> > > > So, that was my next question, so a consumer rebalance will trigger a
> > fresh
> > > > topic pull from the consumers?  How long is 'long enough' to ensure a
> > > > rebalance has occurred everywhere?
> > > >
> > > > Jason
> > > >
> > > > On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy 
> > wrote:
> > > >
> > > > > Hey Jason,
> > > > >
> > > > > Is it an option for you to do the following:
> > > > >
> > > > > - Bounce in a config change to the brokers to turn off auto-create
> > > > > - (Batch)-delete the topic(s)
> > > > > - Wait long enough for consumers to rebalance (after which they will
> > > > >   no longer consume the topic(s))
> > > > > - Bounce in a config change to the brokers to turn on auto-create
> > > > >
> > > > > Joel
> > > > >
> > > > > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > > > > Hi, Jason,
> > > > > >
> > > > > > I am not sure that we should delay the 0.8.2 release. The reasons
> > are
> > > > (1)
> > > > > > There are other features such as the new java producer and
> > Kafka-based
> > > > > > offset management that are potentially useful to people. (2) It may
> > > > take
> > > > > > some time to completely fix the issue with deleting topic since it
> > > > > involves
> > > > > > wire protocol changes.
> > > > > >
> > > > > > What we can do is probably just to document the limitation of
> > delete
> > > > > topic
> > > > > > in 0.8.2.0 in our site.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg <
> > j...@squareup.com>
> > > > > wrote:
> > > > > >
> > > > > > > I think this shortcoming should probably delay release of 0.8.2
> > until
> > > > > > > resolved, no?
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang <
> > wangg...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > It then seems to me that delete-topic will not actually work
> > > > > "smoothly"
> > > > > > > > until create topic request is added since it is too much to
> > require
> > > > > > > people
> > > > > > > > to turn off their clients while deleting topics. In this case
> > shall
> > > > > we
> > > > > > > make
> > > > > > > > it clear in the release docs of in 0.8.2, or even still mark
> > it as
> > > > > > > > not-supported?
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha 
> > wrote:
> > > > > > > >
> > > > > > > > > Jun,
> > > > > > > > >   I made an attempt at fixing that issue as part of this
> > JIRA
> > > > > > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > > > > > As Jay pointed out there should be admin api if there is more
> > > > info
> > > > > on
> > > > > > > > > this api I am interested in adding/fixing this issue.
> > > > > > > > > Thanks,
> > > > > > > > > Harsha
> > > > > > > > >
> > > > > > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > > > > > Yes, that's the issue. Currently, topics can be
> > auto-created on
> > > > > > > > > > TopicMetadataRequest, which can be issued from both the
> > > > producer
> > > > > and
> > > > > > > > the
> > > > > > > > > > consumer. To prevent that, you would need to stop the
> > producer
> > > > > and
> > > > > > > the
> > > > > > > > > > consumer before deleting a topic. We plan to address this
> > issue
> > > > > once
> > > > > > > we
> > > > > > > > > > have a separate request for creating topics.
> > > > > > > > > >
> 

Re: does kafka support "COMMIT" of a batch ?

[Yang]

well I guess this is what I need
https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka

anyone knows how far this effort has gone?

thanks
Yang

On Mon, Jan 26, 2015 at 12:06 PM, Yang  wrote:

> we are pulling from a DB, maybe 100k rows at a time. for each row we send
> it to kafka.
>
> now the problem is , if any errors happen during the db pulling (such as
> an earthquake), we stop.
> next time we wake up. we can't write the same record again into kafka,
> otherwise there might be over counting.
>
>
> does kafka support a "BATCH mode" with commit ? so basically at the start
> of my session, I declare "start transaction", and after all records have
> been pulled and sent to kafka, I declare "commit" .
>
> Thanks
> Yang
>

Re: kafka production server test

[Sa Li]

Thank you for reply Guozhang, right now I can get it work out of box, run
testcase_1 on VM and access production. However, from my point of view, we
really like to test the existing configs on productions, which means for
example, replica_basic_test.py won't start zookeeper&kafka since I want to
test the started brokers. I am thinking if I can comment out this part:

self.log_message("starting zookeepers")

kafka_system_test_utils.start_zookeepers(self.systemTestEnv,
self.testcaseEnv)
self.anonLogger.info("sleeping for 2s")
time.sleep(2)
self.log_message("starting brokers")
kafka_system_test_utils.start_brokers(self.systemTestEnv,
self.testcaseEnv)
self.anonLogger.info("sleeping for 5s")
time.sleep(5)

Now I plan to modify the properties files in
/system_test/replication_testsuite/config/, and cluster_config.json and
testcase_1_properties.json on
/system_test/replication_testsuite/testcase_1/, to make these config files
exactly same as what we have on production. Does it work or I need to
change some other dependencies to get it work?

thanks

AL

On Mon, Jan 26, 2015 at 12:16 PM, Guozhang Wang  wrote:

> Sa,
>
> I believe your questions have mostly been answered by Ewen, and sorry for
> getting late on this.
>
> As you notice the current system test's out-of-the-box experience is not
> very good, and we are proposing ways to improve that situation:
>
> KAFKA-1748 
>
> KAFKA-1589 
>
> And we are adding some more test cases at the same time:
>
> KAFKA-1888 
>
> So if you have new observations while using the package or if you are
> willing to contribute to those tickets you are mostly welcomed.
>
> Guozhang
>
>
> On Thu, Jan 22, 2015 at 3:02 PM, Sa Li  wrote:
>
> > Hi, Guozhang
> >
> > Can I run this package remotely test another server? which mean I run
> this
> > package on dev but testing kafka system on production?
> >
> > thanks
> >
> > AL
> >
> > On Thu, Jan 22, 2015 at 2:55 PM, Sa Li  wrote:
> >
> > > Hi, Guozhang,
> > >
> > > Good to know such package, will try it now. :-)
> > >
> > > thanks
> > >
> > > On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang 
> > wrote:
> > >
> > >> Hi Sa,
> > >>
> > >> Have you looked into the system test package? It contains a suite of
> > tests
> > >> on different failure modes of Kafka brokers.
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li  wrote:
> > >>
> > >> > Hi, All
> > >> >
> > >> > We are about to deliver kafka production server, I have been working
> > on
> > >> > different test, like performance test from linkedin. This is a
> 3-node
> > >> > cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I
> > >> need
> > >> > to do, like network, node failure, flush time, etc. Is there is
> > >> completed
> > >> > guide to instruct the tests for kafka production servers?
> > >> >
> > >> > thanks
> > >> >
> > >> > --
> > >> >
> > >> > Alec Li
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Alec Li
> > >
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
> -- Guozhang
>



-- 

Alec Li

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Joel,

That's probably because console consumer always uses wildcard for
consumption.

Thanks,

Jun

On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy  wrote:

> I think this should work even on non-wildcard consumers. (i.e., I just
> tried it locally and it appears to work fine).
>
> Here is what I did:
>
> - Leave a console consumer up and running, bounce the broker to turn
>   off auto-create
> - Delete the topic
> - The consumer should rebalance and stop consuming that topic
> - Bounce the broker to turn on auto-create
>
> The only issue is that while your broker has auto-create off you will
> see errors on some producers - which you can announce as a brief
> outage to your users.
>
> Joel
>
> On Mon, Jan 26, 2015 at 01:02:57PM -0800, Jun Rao wrote:
> > Actually, I think the deletion of a topic will trigger a rebalance on all
> > wildcard subscribers. The time to complete the rebalance depends on the #
> > of topic/partitions.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg 
> wrote:
> >
> > > Yeah Joel,
> > >
> > > I just thought of that idea too (e.g. temporarily disable auto topic
> > > creation).  I can probably make that work (it may result in unexpected
> lack
> > > of topic creation, but I can message this as a maintenance downtime,
> etc.).
> > >
> > > So, that was my next question, so a consumer rebalance will trigger a
> fresh
> > > topic pull from the consumers?  How long is 'long enough' to ensure a
> > > rebalance has occurred everywhere?
> > >
> > > Jason
> > >
> > > On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy 
> wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > Is it an option for you to do the following:
> > > >
> > > > - Bounce in a config change to the brokers to turn off auto-create
> > > > - (Batch)-delete the topic(s)
> > > > - Wait long enough for consumers to rebalance (after which they will
> > > >   no longer consume the topic(s))
> > > > - Bounce in a config change to the brokers to turn on auto-create
> > > >
> > > > Joel
> > > >
> > > > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > > > Hi, Jason,
> > > > >
> > > > > I am not sure that we should delay the 0.8.2 release. The reasons
> are
> > > (1)
> > > > > There are other features such as the new java producer and
> Kafka-based
> > > > > offset management that are potentially useful to people. (2) It may
> > > take
> > > > > some time to completely fix the issue with deleting topic since it
> > > > involves
> > > > > wire protocol changes.
> > > > >
> > > > > What we can do is probably just to document the limitation of
> delete
> > > > topic
> > > > > in 0.8.2.0 in our site.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg <
> j...@squareup.com>
> > > > wrote:
> > > > >
> > > > > > I think this shortcoming should probably delay release of 0.8.2
> until
> > > > > > resolved, no?
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang <
> wangg...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > It then seems to me that delete-topic will not actually work
> > > > "smoothly"
> > > > > > > until create topic request is added since it is too much to
> require
> > > > > > people
> > > > > > > to turn off their clients while deleting topics. In this case
> shall
> > > > we
> > > > > > make
> > > > > > > it clear in the release docs of in 0.8.2, or even still mark
> it as
> > > > > > > not-supported?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha 
> wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >   I made an attempt at fixing that issue as part of this
> JIRA
> > > > > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > > > > As Jay pointed out there should be admin api if there is more
> > > info
> > > > on
> > > > > > > > this api I am interested in adding/fixing this issue.
> > > > > > > > Thanks,
> > > > > > > > Harsha
> > > > > > > >
> > > > > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > > > > Yes, that's the issue. Currently, topics can be
> auto-created on
> > > > > > > > > TopicMetadataRequest, which can be issued from both the
> > > producer
> > > > and
> > > > > > > the
> > > > > > > > > consumer. To prevent that, you would need to stop the
> producer
> > > > and
> > > > > > the
> > > > > > > > > consumer before deleting a topic. We plan to address this
> issue
> > > > once
> > > > > > we
> > > > > > > > > have a separate request for creating topics.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha 
> > > wrote:
> > > > > > > > >
> > > > > > > > > > There could be another case where if you have
> > > > > > > auto.create.topics.enable
> > > > > > > > > > to set to true ( its true by default) . Any
> > > > TopicMetadataRequest
> > > > > > can
> > > > > > > > > > rec

Re: unable to delete topic with 0.8.2 rc2

[Joel Koshy]

I think this should work even on non-wildcard consumers. (i.e., I just
tried it locally and it appears to work fine).

Here is what I did:

- Leave a console consumer up and running, bounce the broker to turn
  off auto-create
- Delete the topic
- The consumer should rebalance and stop consuming that topic
- Bounce the broker to turn on auto-create

The only issue is that while your broker has auto-create off you will
see errors on some producers - which you can announce as a brief
outage to your users.

Joel

On Mon, Jan 26, 2015 at 01:02:57PM -0800, Jun Rao wrote:
> Actually, I think the deletion of a topic will trigger a rebalance on all
> wildcard subscribers. The time to complete the rebalance depends on the #
> of topic/partitions.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg  wrote:
> 
> > Yeah Joel,
> >
> > I just thought of that idea too (e.g. temporarily disable auto topic
> > creation).  I can probably make that work (it may result in unexpected lack
> > of topic creation, but I can message this as a maintenance downtime, etc.).
> >
> > So, that was my next question, so a consumer rebalance will trigger a fresh
> > topic pull from the consumers?  How long is 'long enough' to ensure a
> > rebalance has occurred everywhere?
> >
> > Jason
> >
> > On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy  wrote:
> >
> > > Hey Jason,
> > >
> > > Is it an option for you to do the following:
> > >
> > > - Bounce in a config change to the brokers to turn off auto-create
> > > - (Batch)-delete the topic(s)
> > > - Wait long enough for consumers to rebalance (after which they will
> > >   no longer consume the topic(s))
> > > - Bounce in a config change to the brokers to turn on auto-create
> > >
> > > Joel
> > >
> > > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > > Hi, Jason,
> > > >
> > > > I am not sure that we should delay the 0.8.2 release. The reasons are
> > (1)
> > > > There are other features such as the new java producer and Kafka-based
> > > > offset management that are potentially useful to people. (2) It may
> > take
> > > > some time to completely fix the issue with deleting topic since it
> > > involves
> > > > wire protocol changes.
> > > >
> > > > What we can do is probably just to document the limitation of delete
> > > topic
> > > > in 0.8.2.0 in our site.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> > > wrote:
> > > >
> > > > > I think this shortcoming should probably delay release of 0.8.2 until
> > > > > resolved, no?
> > > > >
> > > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> > > wrote:
> > > > >
> > > > > > It then seems to me that delete-topic will not actually work
> > > "smoothly"
> > > > > > until create topic request is added since it is too much to require
> > > > > people
> > > > > > to turn off their clients while deleting topics. In this case shall
> > > we
> > > > > make
> > > > > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > > > > not-supported?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >   I made an attempt at fixing that issue as part of this JIRA
> > > > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > > > As Jay pointed out there should be admin api if there is more
> > info
> > > on
> > > > > > > this api I am interested in adding/fixing this issue.
> > > > > > > Thanks,
> > > > > > > Harsha
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > > > > TopicMetadataRequest, which can be issued from both the
> > producer
> > > and
> > > > > > the
> > > > > > > > consumer. To prevent that, you would need to stop the producer
> > > and
> > > > > the
> > > > > > > > consumer before deleting a topic. We plan to address this issue
> > > once
> > > > > we
> > > > > > > > have a separate request for creating topics.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha 
> > wrote:
> > > > > > > >
> > > > > > > > > There could be another case where if you have
> > > > > > auto.create.topics.enable
> > > > > > > > > to set to true ( its true by default) . Any
> > > TopicMetadataRequest
> > > > > can
> > > > > > > > > recreate topics. So if you issued a delete topic command and
> > > you
> > > > > have
> > > > > > > > > producers running or consumers? too which is issuing a
> > > > > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > > > > -Harsha
> > > > > > > > >
> > > > > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > > > > cversion did change (incremented by 2) when I issue the
> > > delete
> > > > > > > command.
> > > > > > > > > >
> > > > > >

Re: [DISCUSSION] Boot dependency in the new producer

[Guozhang Wang]

It will set the needUpdate flag to true and hence the background Sender
will try to talk to the bootstrap servers.

Guozhang

On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps  wrote:

> Hey Guozhang,
>
> That line shouldn't cause any connections to Kafka to be established, does
> it? All that is doing is creating the Cluster pojo using the supplied
> addresses. The use of InetSocketAddress may cause some dns stuff to happen,
> though...
>
> -Jay
>
> On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang 
> wrote:
>
> > Hi all,
> >
> > I am not sure if we have discussed about this before, but recently I
> > realized that we have introduced boot dependency of the kafka-server
> > specified by the "bootstrap.servers" config in the new producer. More
> > specifically, although in the old producer we also have a similar config
> > for specifying the broker list, the producer will not try to connect to
> > those brokers until the first message send call is triggered; whereas in
> > the new producer, it will try to talk to them in construction time via:
> >
> > update(Cluster.bootstrap(addresses), time.milliseconds());
> >
> >
> > I personally am neutral to this change, as in most cases the
> corresponding
> > kafka server should be up and running before the producer clients are
> > deployed, but there are still some corner cases when it is not true, for
> > example some standalone deployment tests of the app embedded with some
> > clients, etc. So I would like to bring this up to people's attention if
> we
> > have not discussed about it before: do we think this is OK to introduce
> > this boot dependency in the new producer?
> >
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: [DISCUSSION] Boot dependency in the new producer

[Jay Kreps]

Hey Guozhang,

That line shouldn't cause any connections to Kafka to be established, does
it? All that is doing is creating the Cluster pojo using the supplied
addresses. The use of InetSocketAddress may cause some dns stuff to happen,
though...

-Jay

On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang  wrote:

> Hi all,
>
> I am not sure if we have discussed about this before, but recently I
> realized that we have introduced boot dependency of the kafka-server
> specified by the "bootstrap.servers" config in the new producer. More
> specifically, although in the old producer we also have a similar config
> for specifying the broker list, the producer will not try to connect to
> those brokers until the first message send call is triggered; whereas in
> the new producer, it will try to talk to them in construction time via:
>
> update(Cluster.bootstrap(addresses), time.milliseconds());
>
>
> I personally am neutral to this change, as in most cases the corresponding
> kafka server should be up and running before the producer clients are
> deployed, but there are still some corner cases when it is not true, for
> example some standalone deployment tests of the app embedded with some
> clients, etc. So I would like to bring this up to people's attention if we
> have not discussed about it before: do we think this is OK to introduce
> this boot dependency in the new producer?
>
> -- Guozhang
>

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Actually, I think the deletion of a topic will trigger a rebalance on all
wildcard subscribers. The time to complete the rebalance depends on the #
of topic/partitions.

Thanks,

Jun

On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg  wrote:

> Yeah Joel,
>
> I just thought of that idea too (e.g. temporarily disable auto topic
> creation).  I can probably make that work (it may result in unexpected lack
> of topic creation, but I can message this as a maintenance downtime, etc.).
>
> So, that was my next question, so a consumer rebalance will trigger a fresh
> topic pull from the consumers?  How long is 'long enough' to ensure a
> rebalance has occurred everywhere?
>
> Jason
>
> On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy  wrote:
>
> > Hey Jason,
> >
> > Is it an option for you to do the following:
> >
> > - Bounce in a config change to the brokers to turn off auto-create
> > - (Batch)-delete the topic(s)
> > - Wait long enough for consumers to rebalance (after which they will
> >   no longer consume the topic(s))
> > - Bounce in a config change to the brokers to turn on auto-create
> >
> > Joel
> >
> > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > Hi, Jason,
> > >
> > > I am not sure that we should delay the 0.8.2 release. The reasons are
> (1)
> > > There are other features such as the new java producer and Kafka-based
> > > offset management that are potentially useful to people. (2) It may
> take
> > > some time to completely fix the issue with deleting topic since it
> > involves
> > > wire protocol changes.
> > >
> > > What we can do is probably just to document the limitation of delete
> > topic
> > > in 0.8.2.0 in our site.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> > wrote:
> > >
> > > > I think this shortcoming should probably delay release of 0.8.2 until
> > > > resolved, no?
> > > >
> > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> > wrote:
> > > >
> > > > > It then seems to me that delete-topic will not actually work
> > "smoothly"
> > > > > until create topic request is added since it is too much to require
> > > > people
> > > > > to turn off their clients while deleting topics. In this case shall
> > we
> > > > make
> > > > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > > > not-supported?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > > > >
> > > > > > Jun,
> > > > > >   I made an attempt at fixing that issue as part of this JIRA
> > > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > > As Jay pointed out there should be admin api if there is more
> info
> > on
> > > > > > this api I am interested in adding/fixing this issue.
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > > > TopicMetadataRequest, which can be issued from both the
> producer
> > and
> > > > > the
> > > > > > > consumer. To prevent that, you would need to stop the producer
> > and
> > > > the
> > > > > > > consumer before deleting a topic. We plan to address this issue
> > once
> > > > we
> > > > > > > have a separate request for creating topics.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha 
> wrote:
> > > > > > >
> > > > > > > > There could be another case where if you have
> > > > > auto.create.topics.enable
> > > > > > > > to set to true ( its true by default) . Any
> > TopicMetadataRequest
> > > > can
> > > > > > > > recreate topics. So if you issued a delete topic command and
> > you
> > > > have
> > > > > > > > producers running or consumers? too which is issuing a
> > > > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > > > -Harsha
> > > > > > > >
> > > > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > > > cversion did change (incremented by 2) when I issue the
> > delete
> > > > > > command.
> > > > > > > > >
> > > > > > > > > From the logs on the conroller broker (also the leader for
> > the
> > > > > > topic), it
> > > > > > > > > looks like the delete proceeds, and then the topic gets
> > recreated
> > > > > > > > > immediately (highlighted in yellow). It appears maybe it’s
> > due
> > > > to a
> > > > > > > > > consumer client app trying to consume the topic. Also, the
> > > > consumer
> > > > > > is
> > > > > > > > > not
> > > > > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s
> > part of
> > > > > the
> > > > > > > > > problem?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > > > > [DeleteTopicsListener on 6]: Starti

Kafka under replicated partitions..

[nitin sharma]

Hi All,

I would like to know the factors that can cause followers to fall behind
and under-replicated replica getting created?

In my production system , last Friday one of the partitions fell behind and
before i could check my operation team had restarted the follower broker...
Restart did fixed the issue but i would like to know

a. the factors that can cause under replicated partitions
b. how to fix that issue? is restart an option?
c. apart from M-Bean any other way to get to know under-replicated
partition.?


Regards,
Nitin Kumar Sharma.

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

Yeah Jun,

I realize it's not worth blocking things, since it's not trivial to solve.

Thanks for all the work on this.

Jason

On Mon, Jan 26, 2015 at 3:01 PM, Jun Rao  wrote:

> Hi, Jason,
>
> I am not sure that we should delay the 0.8.2 release. The reasons are (1)
> There are other features such as the new java producer and Kafka-based
> offset management that are potentially useful to people. (2) It may take
> some time to completely fix the issue with deleting topic since it involves
> wire protocol changes.
>
> What we can do is probably just to document the limitation of delete topic
> in 0.8.2.0 in our site.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> wrote:
>
> > I think this shortcoming should probably delay release of 0.8.2 until
> > resolved, no?
> >
> > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> wrote:
> >
> > > It then seems to me that delete-topic will not actually work "smoothly"
> > > until create topic request is added since it is too much to require
> > people
> > > to turn off their clients while deleting topics. In this case shall we
> > make
> > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > not-supported?
> > >
> > > Guozhang
> > >
> > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > >
> > > > Jun,
> > > >   I made an attempt at fixing that issue as part of this JIRA
> > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > As Jay pointed out there should be admin api if there is more info on
> > > > this api I am interested in adding/fixing this issue.
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > TopicMetadataRequest, which can be issued from both the producer
> and
> > > the
> > > > > consumer. To prevent that, you would need to stop the producer and
> > the
> > > > > consumer before deleting a topic. We plan to address this issue
> once
> > we
> > > > > have a separate request for creating topics.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > > >
> > > > > > There could be another case where if you have
> > > auto.create.topics.enable
> > > > > > to set to true ( its true by default) . Any TopicMetadataRequest
> > can
> > > > > > recreate topics. So if you issued a delete topic command and you
> > have
> > > > > > producers running or consumers? too which is issuing a
> > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > -Harsha
> > > > > >
> > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > cversion did change (incremented by 2) when I issue the delete
> > > > command.
> > > > > > >
> > > > > > > From the logs on the conroller broker (also the leader for the
> > > > topic), it
> > > > > > > looks like the delete proceeds, and then the topic gets
> recreated
> > > > > > > immediately (highlighted in yellow). It appears maybe it’s due
> > to a
> > > > > > > consumer client app trying to consume the topic. Also, the
> > consumer
> > > > is
> > > > > > > not
> > > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part
> of
> > > the
> > > > > > > problem?
> > > > > > >
> > > > > > >
> > > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > > > mytopic
> > > > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > Topic
> > > > > > > deletion callback for mytopic
> > > > > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > > > > Partition deletion callback for [mytopic,0]
> > > > > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > > > > controller.ReplicaStateMachine - [Replica state machine on
> > > controller
> > > > > > > 6]: Invoking state change to OfflineReplica for replicas
> > > > > > >
> > > > > >
> > > >
> > >
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > > > > controller.KafkaController - [Controller 6]: New leader and ISR
> > for
> > > > > > > partition [mytopic,0] is

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

Yeah Joel,

I just thought of that idea too (e.g. temporarily disable auto topic
creation).  I can probably make that work (it may result in unexpected lack
of topic creation, but I can message this as a maintenance downtime, etc.).

So, that was my next question, so a consumer rebalance will trigger a fresh
topic pull from the consumers?  How long is 'long enough' to ensure a
rebalance has occurred everywhere?

Jason

On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy  wrote:

> Hey Jason,
>
> Is it an option for you to do the following:
>
> - Bounce in a config change to the brokers to turn off auto-create
> - (Batch)-delete the topic(s)
> - Wait long enough for consumers to rebalance (after which they will
>   no longer consume the topic(s))
> - Bounce in a config change to the brokers to turn on auto-create
>
> Joel
>
> On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > Hi, Jason,
> >
> > I am not sure that we should delay the 0.8.2 release. The reasons are (1)
> > There are other features such as the new java producer and Kafka-based
> > offset management that are potentially useful to people. (2) It may take
> > some time to completely fix the issue with deleting topic since it
> involves
> > wire protocol changes.
> >
> > What we can do is probably just to document the limitation of delete
> topic
> > in 0.8.2.0 in our site.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> wrote:
> >
> > > I think this shortcoming should probably delay release of 0.8.2 until
> > > resolved, no?
> > >
> > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> wrote:
> > >
> > > > It then seems to me that delete-topic will not actually work
> "smoothly"
> > > > until create topic request is added since it is too much to require
> > > people
> > > > to turn off their clients while deleting topics. In this case shall
> we
> > > make
> > > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > > not-supported?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > > >
> > > > > Jun,
> > > > >   I made an attempt at fixing that issue as part of this JIRA
> > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > As Jay pointed out there should be admin api if there is more info
> on
> > > > > this api I am interested in adding/fixing this issue.
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > > TopicMetadataRequest, which can be issued from both the producer
> and
> > > > the
> > > > > > consumer. To prevent that, you would need to stop the producer
> and
> > > the
> > > > > > consumer before deleting a topic. We plan to address this issue
> once
> > > we
> > > > > > have a separate request for creating topics.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > > > >
> > > > > > > There could be another case where if you have
> > > > auto.create.topics.enable
> > > > > > > to set to true ( its true by default) . Any
> TopicMetadataRequest
> > > can
> > > > > > > recreate topics. So if you issued a delete topic command and
> you
> > > have
> > > > > > > producers running or consumers? too which is issuing a
> > > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > > -Harsha
> > > > > > >
> > > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > > cversion did change (incremented by 2) when I issue the
> delete
> > > > > command.
> > > > > > > >
> > > > > > > > From the logs on the conroller broker (also the leader for
> the
> > > > > topic), it
> > > > > > > > looks like the delete proceeds, and then the topic gets
> recreated
> > > > > > > > immediately (highlighted in yellow). It appears maybe it’s
> due
> > > to a
> > > > > > > > consumer client app trying to consume the topic. Also, the
> > > consumer
> > > > > is
> > > > > > > > not
> > > > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s
> part of
> > > > the
> > > > > > > > problem?
> > > > > > > >
> > > > > > > >
> > > > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > > > [DeleteTopicsListener on 6]: Starting topic deletion for
> topics
> > > > > > > > mytopic
> > > > > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > > > [delete-topics-thread-6], Handling deletion for topics
> mytopic
> > > > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > > > [delete-topics-thread-6], Deletion of topic mytopic
> (re)started
> > > > > > > > 20

Re: kafka production server test

[Guozhang Wang]

Sa,

I believe your questions have mostly been answered by Ewen, and sorry for
getting late on this.

As you notice the current system test's out-of-the-box experience is not
very good, and we are proposing ways to improve that situation:

KAFKA-1748 

KAFKA-1589 

And we are adding some more test cases at the same time:

KAFKA-1888 

So if you have new observations while using the package or if you are
willing to contribute to those tickets you are mostly welcomed.

Guozhang


On Thu, Jan 22, 2015 at 3:02 PM, Sa Li  wrote:

> Hi, Guozhang
>
> Can I run this package remotely test another server? which mean I run this
> package on dev but testing kafka system on production?
>
> thanks
>
> AL
>
> On Thu, Jan 22, 2015 at 2:55 PM, Sa Li  wrote:
>
> > Hi, Guozhang,
> >
> > Good to know such package, will try it now. :-)
> >
> > thanks
> >
> > On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang 
> wrote:
> >
> >> Hi Sa,
> >>
> >> Have you looked into the system test package? It contains a suite of
> tests
> >> on different failure modes of Kafka brokers.
> >>
> >> Guozhang
> >>
> >>
> >> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li  wrote:
> >>
> >> > Hi, All
> >> >
> >> > We are about to deliver kafka production server, I have been working
> on
> >> > different test, like performance test from linkedin. This is a 3-node
> >> > cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I
> >> need
> >> > to do, like network, node failure, flush time, etc. Is there is
> >> completed
> >> > guide to instruct the tests for kafka production servers?
> >> >
> >> > thanks
> >> >
> >> > --
> >> >
> >> > Alec Li
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
>
> Alec Li
>



-- 
-- Guozhang

does kafka support "COMMIT" of a batch ?

[Yang]

we are pulling from a DB, maybe 100k rows at a time. for each row we send
it to kafka.

now the problem is , if any errors happen during the db pulling (such as an
earthquake), we stop.
next time we wake up. we can't write the same record again into kafka,
otherwise there might be over counting.


does kafka support a "BATCH mode" with commit ? so basically at the start
of my session, I declare "start transaction", and after all records have
been pulled and sent to kafka, I declare "commit" .

Thanks
Yang

Re: unable to delete topic with 0.8.2 rc2

[Joel Koshy]

Hey Jason,

Is it an option for you to do the following:

- Bounce in a config change to the brokers to turn off auto-create
- (Batch)-delete the topic(s)
- Wait long enough for consumers to rebalance (after which they will
  no longer consume the topic(s))
- Bounce in a config change to the brokers to turn on auto-create

Joel

On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> Hi, Jason,
> 
> I am not sure that we should delay the 0.8.2 release. The reasons are (1)
> There are other features such as the new java producer and Kafka-based
> offset management that are potentially useful to people. (2) It may take
> some time to completely fix the issue with deleting topic since it involves
> wire protocol changes.
> 
> What we can do is probably just to document the limitation of delete topic
> in 0.8.2.0 in our site.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg  wrote:
> 
> > I think this shortcoming should probably delay release of 0.8.2 until
> > resolved, no?
> >
> > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang  wrote:
> >
> > > It then seems to me that delete-topic will not actually work "smoothly"
> > > until create topic request is added since it is too much to require
> > people
> > > to turn off their clients while deleting topics. In this case shall we
> > make
> > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > not-supported?
> > >
> > > Guozhang
> > >
> > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > >
> > > > Jun,
> > > >   I made an attempt at fixing that issue as part of this JIRA
> > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > As Jay pointed out there should be admin api if there is more info on
> > > > this api I am interested in adding/fixing this issue.
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > TopicMetadataRequest, which can be issued from both the producer and
> > > the
> > > > > consumer. To prevent that, you would need to stop the producer and
> > the
> > > > > consumer before deleting a topic. We plan to address this issue once
> > we
> > > > > have a separate request for creating topics.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > > >
> > > > > > There could be another case where if you have
> > > auto.create.topics.enable
> > > > > > to set to true ( its true by default) . Any TopicMetadataRequest
> > can
> > > > > > recreate topics. So if you issued a delete topic command and you
> > have
> > > > > > producers running or consumers? too which is issuing a
> > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > -Harsha
> > > > > >
> > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > cversion did change (incremented by 2) when I issue the delete
> > > > command.
> > > > > > >
> > > > > > > From the logs on the conroller broker (also the leader for the
> > > > topic), it
> > > > > > > looks like the delete proceeds, and then the topic gets recreated
> > > > > > > immediately (highlighted in yellow). It appears maybe it’s due
> > to a
> > > > > > > consumer client app trying to consume the topic. Also, the
> > consumer
> > > > is
> > > > > > > not
> > > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of
> > > the
> > > > > > > problem?
> > > > > > >
> > > > > > >
> > > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > > > mytopic
> > > > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > Topic
> > > > > > > deletion callback for mytopic
> > > > > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > > > > Partition deletion callback for [mytopic,0]
> > > > > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > > > > controller.ReplicaStateMachine - [Replica state machine on
> > > controller
> > > > > > > 6]: Invoking state change to OfflineReplica for replicas
> > > > > > >
> > > > > >
> > > >
> > >
> > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Rep

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Bhavesh Mistry]

Hi Kafka Team,

I just wanted to bring this to your attention regarding Java New Producer
limitation compare to old producer.

a) Partition Increasing is limited to configured memory allocation.


buffer.memory
batch.size


The maximum partition you could have before impacting (New Java Producers)
producers is buffer.memory /  batch.size.  So Developer can plan for
horizontal scaling partition from the beginning otherwise production
running code will be impacted based on *block.on.buffer.full
configuration *(block
or BufferExhaustedException).

This limitation does not exits with old scala based Producer.

This will allow user community to buffer more and plan the capacity before
hand.   May be add this info
http://kafka.apache.org/documentation.html#newproducerconfigs about
limitation.

Thanks,

Bhavesh


On Mon, Jan 26, 2015 at 10:28 AM, Joe Stein  wrote:

> +1 (binding)
>
> artifacts and quick start look good. I ran in some client code, minor edits
> from 0-8.2-beta https://github.com/stealthly/scala-kafka/pull/26
>
> On Mon, Jan 26, 2015 at 3:38 AM, Manikumar Reddy 
> wrote:
>
> > +1 (Non-binding)
> > Verified source package, unit tests, release build, topic deletion,
> > compaction and random testing
> >
> > On Mon, Jan 26, 2015 at 6:14 AM, Neha Narkhede 
> wrote:
> >
> >> +1 (binding)
> >> Verified keys, quick start, unit tests.
> >>
> >> On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein 
> wrote:
> >>
> >> > That makes sense, thanks!
> >> >
> >> > On Sat, Jan 24, 2015 at 7:00 PM, Jay Kreps 
> wrote:
> >> >
> >> > > But I think the flaw in trying to guess what kind of serializer they
> >> will
> >> > > use is when we get it wrong. Basically let's say we guess "String".
> >> Say
> >> > 30%
> >> > > of the time we will be right and we will save the two configuration
> >> > lines.
> >> > > 70% of the time we will be wrong and the user gets a super cryptic
> >> > > ClassCastException: "xyz cannot be cast to [B" (because [B is how
> java
> >> > > chooses to display the byte array class just to up the pain), then
> >> they
> >> > > figure out how to subscribe to our mailing list and email us the
> >> cryptic
> >> > > exception, then we explain about how we helpfully set these
> properties
> >> > for
> >> > > them to save them time. :-)
> >> > >
> >> > >
> >>
> https://www.google.com/?gws_rd=ssl#q=kafka+classcastexception+%22%5BB%22
> >> > >
> >> > > I think basically we did this experiment with the old clients and
> the
> >> > > conclusion is that serialization is something you basically have to
> >> think
> >> > > about to use Kafka and trying to guess just makes things worse.
> >> > >
> >> > > -Jay
> >> > >
> >> > > On Sat, Jan 24, 2015 at 2:51 PM, Joe Stein 
> >> wrote:
> >> > >
> >> > >> Maybe. I think the StringSerialzer could look more like a typical
> >> type
> >> > of
> >> > >> message.  Instead of encoding being a property it would be more
> >> > typically
> >> > >> just written in the bytes.
> >> > >>
> >> > >> On Sat, Jan 24, 2015 at 12:12 AM, Jay Kreps 
> >> > wrote:
> >> > >>
> >> > >> > I don't think so--see if you buy my explanation. We previously
> >> > defaulted
> >> > >> > to the byte array serializer and it was a source of unending
> >> > frustration
> >> > >> > and confusion. Since it wasn't a required config people just went
> >> > along
> >> > >> > plugging in whatever objects they had, and thinking that changing
> >> the
> >> > >> > parametric types would somehow help. Then they would get a class
> >> case
> >> > >> > exception and assume our stuff was somehow busted, not realizing
> we
> >> > had
> >> > >> > helpfully configured a type different from what they were passing
> >> in
> >> > >> under
> >> > >> > the covers. So I think it is actually good for people to think:
> how
> >> > am I
> >> > >> > serializing my data, and getting that exception will make them
> ask
> >> > that
> >> > >> > question right?
> >> > >> >
> >> > >> > -Jay
> >> > >> >
> >> > >> > On Fri, Jan 23, 2015 at 9:06 PM, Joe Stein  >
> >> > >> wrote:
> >> > >> >
> >> > >> >> Should value.serializer in the new java producer be defaulted to
> >> > >> >> Array[Byte] ?
> >> > >> >>
> >> > >> >> I was working on testing some upgrade paths and got this
> >> > >> >>
> >> > >> >> ! return exception in callback when buffer cannot accept
> >> message
> >> > >> >>
> >> > >> >>   ConfigException: Missing required configuration
> >> > >> "value.serializer"
> >> > >> >> which has no default value. (ConfigDef.java:124)
> >> > >> >>
> >> > >> >>
> >>  org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:235)
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >>
> >> >
> >>
> org.apache.kafka.clients.producer.KafkaProd

Re: Regarding Kafka release 0.8.2-beta

[Jun Rao]

The new consumer api is actually excluded from the javadoc that we generate.

Thanks,

Jun

On Mon, Jan 26, 2015 at 11:54 AM, Jason Rosenberg  wrote:

> shouldn't the new consumer api be removed from the 0.8.2 code base then?
>
> On Fri, Jan 23, 2015 at 10:30 AM, Joe Stein  wrote:
>
> > The new consumer is scheduled for 0.9.0.
> >
> > Currently Kafka release candidate 2 for 0.8.2.0 is being voted on.
> >
> > There is an in progress patch to the new consumer that you can try out
> > https://issues.apache.org/jira/browse/KAFKA-1760
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
> > On Fri, Jan 23, 2015 at 1:55 AM, Reeni Mathew 
> > wrote:
> >
> > > Hi Team,
> > >
> > > I was playing around with your recent release 0.8.2-beta.
> > > Producer worked fine whereas new consumer did not.
> > >
> > > org.apache.kafka.clients.consumer.KafkaConsumer
> > >
> > > After digging the code I realized that the implementation for the same
> is
> > > not available. Only API is present.
> > > Could you please let me know by when we can expect the implementation
> of
> > > the same.
> > >
> > > Thanks & Regards
> > >
> > > Reeni
> > >
> >
>

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Hi, Jason,

I am not sure that we should delay the 0.8.2 release. The reasons are (1)
There are other features such as the new java producer and Kafka-based
offset management that are potentially useful to people. (2) It may take
some time to completely fix the issue with deleting topic since it involves
wire protocol changes.

What we can do is probably just to document the limitation of delete topic
in 0.8.2.0 in our site.

Thanks,

Jun

On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg  wrote:

> I think this shortcoming should probably delay release of 0.8.2 until
> resolved, no?
>
> On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang  wrote:
>
> > It then seems to me that delete-topic will not actually work "smoothly"
> > until create topic request is added since it is too much to require
> people
> > to turn off their clients while deleting topics. In this case shall we
> make
> > it clear in the release docs of in 0.8.2, or even still mark it as
> > not-supported?
> >
> > Guozhang
> >
> > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> >
> > > Jun,
> > >   I made an attempt at fixing that issue as part of this JIRA
> > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > As Jay pointed out there should be admin api if there is more info on
> > > this api I am interested in adding/fixing this issue.
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > TopicMetadataRequest, which can be issued from both the producer and
> > the
> > > > consumer. To prevent that, you would need to stop the producer and
> the
> > > > consumer before deleting a topic. We plan to address this issue once
> we
> > > > have a separate request for creating topics.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > >
> > > > > There could be another case where if you have
> > auto.create.topics.enable
> > > > > to set to true ( its true by default) . Any TopicMetadataRequest
> can
> > > > > recreate topics. So if you issued a delete topic command and you
> have
> > > > > producers running or consumers? too which is issuing a
> > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > -Harsha
> > > > >
> > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > cversion did change (incremented by 2) when I issue the delete
> > > command.
> > > > > >
> > > > > > From the logs on the conroller broker (also the leader for the
> > > topic), it
> > > > > > looks like the delete proceeds, and then the topic gets recreated
> > > > > > immediately (highlighted in yellow). It appears maybe it’s due
> to a
> > > > > > consumer client app trying to consume the topic. Also, the
> consumer
> > > is
> > > > > > not
> > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of
> > the
> > > > > > problem?
> > > > > >
> > > > > >
> > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > > mytopic
> > > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> Topic
> > > > > > deletion callback for mytopic
> > > > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > > > Partition deletion callback for [mytopic,0]
> > > > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > > > controller.ReplicaStateMachine - [Replica state machine on
> > controller
> > > > > > 6]: Invoking state change to OfflineReplica for replicas
> > > > > >
> > > > >
> > >
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > > > controller.KafkaController - [Controller 6]: New leader and ISR
> for
> > > > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > > > controller.KafkaController - [Controller 6]: New leader and ISR
> for
> > > > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > > > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > > > > controller.ReplicaStateMachine - [Replica state machine on
> > controller
>

Re: Regarding Kafka release 0.8.2-beta

[Joe Stein]

Matve wr should add to the documentation experimental so folks that don't
know understand.

/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
/
On Jan 26, 2015 11:56 AM, "Jason Rosenberg"  wrote:

> shouldn't the new consumer api be removed from the 0.8.2 code base then?
>
> On Fri, Jan 23, 2015 at 10:30 AM, Joe Stein  wrote:
>
> > The new consumer is scheduled for 0.9.0.
> >
> > Currently Kafka release candidate 2 for 0.8.2.0 is being voted on.
> >
> > There is an in progress patch to the new consumer that you can try out
> > https://issues.apache.org/jira/browse/KAFKA-1760
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
> > On Fri, Jan 23, 2015 at 1:55 AM, Reeni Mathew 
> > wrote:
> >
> > > Hi Team,
> > >
> > > I was playing around with your recent release 0.8.2-beta.
> > > Producer worked fine whereas new consumer did not.
> > >
> > > org.apache.kafka.clients.consumer.KafkaConsumer
> > >
> > > After digging the code I realized that the implementation for the same
> is
> > > not available. Only API is present.
> > > Could you please let me know by when we can expect the implementation
> of
> > > the same.
> > >
> > > Thanks & Regards
> > >
> > > Reeni
> > >
> >
>

Re: Regarding Kafka release 0.8.2-beta

[Jason Rosenberg]

shouldn't the new consumer api be removed from the 0.8.2 code base then?

On Fri, Jan 23, 2015 at 10:30 AM, Joe Stein  wrote:

> The new consumer is scheduled for 0.9.0.
>
> Currently Kafka release candidate 2 for 0.8.2.0 is being voted on.
>
> There is an in progress patch to the new consumer that you can try out
> https://issues.apache.org/jira/browse/KAFKA-1760
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
> On Fri, Jan 23, 2015 at 1:55 AM, Reeni Mathew 
> wrote:
>
> > Hi Team,
> >
> > I was playing around with your recent release 0.8.2-beta.
> > Producer worked fine whereas new consumer did not.
> >
> > org.apache.kafka.clients.consumer.KafkaConsumer
> >
> > After digging the code I realized that the implementation for the same is
> > not available. Only API is present.
> > Could you please let me know by when we can expect the implementation of
> > the same.
> >
> > Thanks & Regards
> >
> > Reeni
> >
>

Re: kafka deleted old logs but not released

[nitin sharma]

hi All,

I am facing the same issue.. i  have got Kafka_2.9.2-0.8.1.1.jar deployed
in my application.
My Operation team had reported significant increase in the directory where
Kafak-log folder exists . when i checked the size of parent directory is it
99GB (100%) utilized..  This was very confusing because when i calculated
the sizes of .log files of my partitions it came upto only 40GB...

Moreover, when i checked Kafka logs, i saw following error message, which
means something wrong with Kafka code..

Can anyone tell where can i find the old Log files which Kafka code still
pointing to..? are they in hidden folders?

 [2015-01-23 19:00:11,233] ERROR Uncaught exception in scheduled task
'kafka-log-retention' (kafka.utils.KafkaScheduler)

kafka.common.KafkaStorageException: Failed to change the log file suffix
from  to .deleted for log segment 449238458

at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)

at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:636)

at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:627)

at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:415)

at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:415)

Regards,
Nitin Kumar Sharma.


On Sun, Jan 25, 2015 at 10:57 PM, Yonghui Zhao 
wrote:

> I have fixed this issue like this patch
> https://reviews.apache.org/r/29755/diff/5/.
>
> I find rename failure still happens:
>
> server.log.2015-01-26-06:[2015-01-26 06:10:54,513] ERROR File rename
> failed, forcefully deleting file (kafka.log.Log)
> server.log.2015-01-26-06:[2015-01-26 06:10:54,600] ERROR File rename
> failed, forcefully deleting file (kafka.log.Log)
> server.log.2015-01-26-06:[2015-01-26 06:10:54,685] ERROR File rename
> failed, forcefully deleting file (kafka.log.Log)
> server.log.2015-01-26-06:[2015-01-26 06:10:54,797] ERROR File rename
> failed, forcefully deleting file (kafka.log.Log)
> 
>
> And use lsof I can still find some files opened by kafka but deleted,  but
> those files sizes are 0.
>
> java   3228 root   34uw REG  253,2 0   26084228
> /home/work/data/soft/kafka-0.8/data/.lock (deleted)
> java   3228 root   35u  REG  253,2 0   26084232
> /home/work/data/soft/kafka-0.8/data/cube-0/.log
> (deleted)
> java   3228 root   36u  REG  253,2 0   26869778
>
> /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_topic_misearch_appstore-search-0/3116.log
> (deleted)
> java   3228 root   37u  REG  253,2 0   26084234
>
> /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_topic_mishop-search_mishop_v1-0/.log
> (deleted)
>
>
>
> Here is my configuration:
>
> Binary: kafka_2.10-0.8.1.1
> Retention config:
>
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=168
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> #log.retention.bytes=1073741824
>
> # The maximum size of a log segment file. When this size is reached a new
> log segment will be created.
> log.segment.bytes=536870912
>
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.retention.check.interval.ms=6
>
> # By default the log cleaner is disabled and the log retention policy will
> default to just delete segments after their retention expires.
> # If log.cleaner.enable=true is set the cleaner will be enabled and
> individual logs can then be marked for log compaction.
> log.cleaner.enable=false
>
>
>
> OS:  CentOS release 6.4 (Final)
> JDK:
> *java version "1.6.0_37"*
> Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)
>
> JDK is too old, but not sure if this result in rename failure.
>
>
>
>
>
>
> 2015-01-26 0:42 GMT+08:00 Jay Kreps :
>
> > Also, what is the configuration for the servers? In particular it would
> be
> > good to know the retention and/or log compaction settings as those delete
> > files.
> >
> > -Jay
> >
> > On Sun, Jan 25, 2015 at 4:34 AM, Jaikiran Pai 
> > wrote:
> >
> >> Hi Yonghui,
> >>
> >> Do you still have this happening? If yes, can you tell us a bit more
> >> about your setup? Is there something else that accesses or maybe
> deleting
> >> these log files? For more context to this question, please read the
> >> discussion related to this here http://mail-archives.apache.
> >> org/mod_mbox/kafka-dev/201501.mbox/%3C54C47E9B.5060401%40gmail.com%3E
> >>
> >>
> >> -Jaikiran
> >>
> >>
> >>> On Thursday 08 January 2015 11:19 AM, Yonghui Zhao wrote:
> >>>
>  CentOS release 6.3 (Final)
> 
> 
>  2015-01-07 22:18 GMT+08:00 Harsha :
> 
>   Yonghui,
> > Which OS you are running.
> > -Harsha
> >
> > On Wed, Jan 7, 2015, at 01:38

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

I think this shortcoming should probably delay release of 0.8.2 until
resolved, no?

On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang  wrote:

> It then seems to me that delete-topic will not actually work "smoothly"
> until create topic request is added since it is too much to require people
> to turn off their clients while deleting topics. In this case shall we make
> it clear in the release docs of in 0.8.2, or even still mark it as
> not-supported?
>
> Guozhang
>
> On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
>
> > Jun,
> >   I made an attempt at fixing that issue as part of this JIRA
> >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > As Jay pointed out there should be admin api if there is more info on
> > this api I am interested in adding/fixing this issue.
> > Thanks,
> > Harsha
> >
> > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > Yes, that's the issue. Currently, topics can be auto-created on
> > > TopicMetadataRequest, which can be issued from both the producer and
> the
> > > consumer. To prevent that, you would need to stop the producer and the
> > > consumer before deleting a topic. We plan to address this issue once we
> > > have a separate request for creating topics.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > >
> > > > There could be another case where if you have
> auto.create.topics.enable
> > > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > > recreate topics. So if you issued a delete topic command and you have
> > > > producers running or consumers? too which is issuing a
> > > > TopicMetadataRequest than the topic will be recreated.
> > > > -Harsha
> > > >
> > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > cversion did change (incremented by 2) when I issue the delete
> > command.
> > > > >
> > > > > From the logs on the conroller broker (also the leader for the
> > topic), it
> > > > > looks like the delete proceeds, and then the topic gets recreated
> > > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > > consumer client app trying to consume the topic. Also, the consumer
> > is
> > > > > not
> > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of
> the
> > > > > problem?
> > > > >
> > > > >
> > > > > 2015-01-26 07:02:14,281  INFO
> > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > mytopic
> > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > > deletion callback for mytopic
> > > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > > Partition deletion callback for [mytopic,0]
> > > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > > controller.ReplicaStateMachine - [Replica state machine on
> controller
> > > > > 6]: Invoking state change to OfflineReplica for replicas
> > > > >
> > > >
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > > > controller.ReplicaStateMachine - [Replica state machine on
> controller
> > > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > > > >
> > > >
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > > Removed fetcher for partitions [mytopic,0]
> > > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > > Removed fetcher for partitions [mytopic,0]
> > > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > > log.OffsetIndex - Deleting index
> > > > > /mypath/mytopic-0/

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

So, this is rather disappointing, especially since topic deletion is really
the primary feature in 0.8.2 I'm interested in.  The topic I was trying to
delete above had no data for many months.  The consumer which is triggering
recreation of that topic has been restarted several times since that topic
stopped receiving data.

In our case, we have a large number of topics, that no longer receive
messages (many have not received messages in over 12 months).  However, we
have consumers that use a regex which matches multiple topics, some of
which are these empty zombie topics.  Thus, we have a chicken-and-egg
problem:

1.  Consumer uses a regex to discover matching topics.
2.  Consumer starts consuming 'topic.foo.which.is.empty'.  This results in
regular meta-data requests for data for that topic.
3.  'topic.foo.which.is.empty' is deleted.
4.  Consumer encounters an error trying to fetch this topic, so issues a
meta-data request to find the leader for this topic.
5.  Broker recreates the topic in response to this.

We can stop consumers for maintenance and do a batch delete, but it is
problematic, because there are realtime dependencies on those consumers
being up and running.

I expect this will necessarily result in my voting no for 0.8.2 RC2 (if I
have a vote :)).

Just read up on KAFKA-1507, and left my 2 cents:

"I think relegating topic creation to an admin client would be very
limitiing. It's extremely useful to have a self-service system where new
applications can just create a new topic on demand (with reasonable
defaults), without the need for an admin to come in and prepare topics
ahead of a code release (leave that to dba's managing transactional
databases!).

I do like the idea of an automatic create topic request from a producer, in
response to a topic not found exception, rather than auto-creating topics
from meta-data requests (which happens asynchronously and causes the
initial meta data request to fail usually!). Consumers should never create
a topic, I should think."

On Mon, Jan 26, 2015 at 11:14 AM, Harsha  wrote:

> Jun,
>   I made an attempt at fixing that issue as part of this JIRA
>   https://issues.apache.org/jira/browse/KAFKA-1507 .
> As Jay pointed out there should be admin api if there is more info on
> this api I am interested in adding/fixing this issue.
> Thanks,
> Harsha
>
> On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > Yes, that's the issue. Currently, topics can be auto-created on
> > TopicMetadataRequest, which can be issued from both the producer and the
> > consumer. To prevent that, you would need to stop the producer and the
> > consumer before deleting a topic. We plan to address this issue once we
> > have a separate request for creating topics.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> >
> > > There could be another case where if you have auto.create.topics.enable
> > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > recreate topics. So if you issued a delete topic command and you have
> > > producers running or consumers? too which is issuing a
> > > TopicMetadataRequest than the topic will be recreated.
> > > -Harsha
> > >
> > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > cversion did change (incremented by 2) when I issue the delete
> command.
> > > >
> > > > From the logs on the conroller broker (also the leader for the
> topic), it
> > > > looks like the delete proceeds, and then the topic gets recreated
> > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > consumer client app trying to consume the topic. Also, the consumer
> is
> > > > not
> > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > > problem?
> > > >
> > > >
> > > > 2015-01-26 07:02:14,281  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > mytopic
> > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > deletion callback for mytopic
> > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > Partition deletion callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to OfflineReplica f

Re: unable to delete topic with 0.8.2 rc2

[Guozhang Wang]

It then seems to me that delete-topic will not actually work "smoothly"
until create topic request is added since it is too much to require people
to turn off their clients while deleting topics. In this case shall we make
it clear in the release docs of in 0.8.2, or even still mark it as
not-supported?

Guozhang

On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:

> Jun,
>   I made an attempt at fixing that issue as part of this JIRA
>   https://issues.apache.org/jira/browse/KAFKA-1507 .
> As Jay pointed out there should be admin api if there is more info on
> this api I am interested in adding/fixing this issue.
> Thanks,
> Harsha
>
> On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > Yes, that's the issue. Currently, topics can be auto-created on
> > TopicMetadataRequest, which can be issued from both the producer and the
> > consumer. To prevent that, you would need to stop the producer and the
> > consumer before deleting a topic. We plan to address this issue once we
> > have a separate request for creating topics.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> >
> > > There could be another case where if you have auto.create.topics.enable
> > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > recreate topics. So if you issued a delete topic command and you have
> > > producers running or consumers? too which is issuing a
> > > TopicMetadataRequest than the topic will be recreated.
> > > -Harsha
> > >
> > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > cversion did change (incremented by 2) when I issue the delete
> command.
> > > >
> > > > From the logs on the conroller broker (also the leader for the
> topic), it
> > > > looks like the delete proceeds, and then the topic gets recreated
> > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > consumer client app trying to consume the topic. Also, the consumer
> is
> > > > not
> > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > > problem?
> > > >
> > > >
> > > > 2015-01-26 07:02:14,281  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > mytopic
> > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > deletion callback for mytopic
> > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > Partition deletion callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to OfflineReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > log.OffsetIndex - Deleting index
> > > > /mypath/mytopic-0/.index
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> log.LogManager
> > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > > > 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking stat

[DISCUSSION] Boot dependency in the new producer

[Guozhang Wang]

Hi all,

I am not sure if we have discussed about this before, but recently I
realized that we have introduced boot dependency of the kafka-server
specified by the "bootstrap.servers" config in the new producer. More
specifically, although in the old producer we also have a similar config
for specifying the broker list, the producer will not try to connect to
those brokers until the first message send call is triggered; whereas in
the new producer, it will try to talk to them in construction time via:

update(Cluster.bootstrap(addresses), time.milliseconds());


I personally am neutral to this change, as in most cases the corresponding
kafka server should be up and running before the producer clients are
deployed, but there are still some corner cases when it is not true, for
example some standalone deployment tests of the app embedded with some
clients, etc. So I would like to bring this up to people's attention if we
have not discussed about it before: do we think this is OK to introduce
this boot dependency in the new producer?

-- Guozhang

MQTT and Kafka Integration

[Su She]

Hello Everyone,

I recently watched this video by Tim Kellogg:
https://www.youtube.com/watch?v=VoTclkxSago.

1) Has anyone integrated MQTT and Kafka? If so how did the architecture
look like? I am confused as to how to get messages from the MQTT broker to
the Kafka broker.

2) Has anyone used this KafkaBridge?:
https://github.com/jacklund/mqttKafkaBridge

3) I understand the benefits of adding Kafka to MQTT, but can anyone
explain the benefits of adding MQTT to Kafka? I'm guessing security is one
of them.

Thank you for the help!

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Joe Stein]

+1 (binding)

artifacts and quick start look good. I ran in some client code, minor edits
from 0-8.2-beta https://github.com/stealthly/scala-kafka/pull/26

On Mon, Jan 26, 2015 at 3:38 AM, Manikumar Reddy 
wrote:

> +1 (Non-binding)
> Verified source package, unit tests, release build, topic deletion,
> compaction and random testing
>
> On Mon, Jan 26, 2015 at 6:14 AM, Neha Narkhede  wrote:
>
>> +1 (binding)
>> Verified keys, quick start, unit tests.
>>
>> On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein  wrote:
>>
>> > That makes sense, thanks!
>> >
>> > On Sat, Jan 24, 2015 at 7:00 PM, Jay Kreps  wrote:
>> >
>> > > But I think the flaw in trying to guess what kind of serializer they
>> will
>> > > use is when we get it wrong. Basically let's say we guess "String".
>> Say
>> > 30%
>> > > of the time we will be right and we will save the two configuration
>> > lines.
>> > > 70% of the time we will be wrong and the user gets a super cryptic
>> > > ClassCastException: "xyz cannot be cast to [B" (because [B is how java
>> > > chooses to display the byte array class just to up the pain), then
>> they
>> > > figure out how to subscribe to our mailing list and email us the
>> cryptic
>> > > exception, then we explain about how we helpfully set these properties
>> > for
>> > > them to save them time. :-)
>> > >
>> > >
>> https://www.google.com/?gws_rd=ssl#q=kafka+classcastexception+%22%5BB%22
>> > >
>> > > I think basically we did this experiment with the old clients and the
>> > > conclusion is that serialization is something you basically have to
>> think
>> > > about to use Kafka and trying to guess just makes things worse.
>> > >
>> > > -Jay
>> > >
>> > > On Sat, Jan 24, 2015 at 2:51 PM, Joe Stein 
>> wrote:
>> > >
>> > >> Maybe. I think the StringSerialzer could look more like a typical
>> type
>> > of
>> > >> message.  Instead of encoding being a property it would be more
>> > typically
>> > >> just written in the bytes.
>> > >>
>> > >> On Sat, Jan 24, 2015 at 12:12 AM, Jay Kreps 
>> > wrote:
>> > >>
>> > >> > I don't think so--see if you buy my explanation. We previously
>> > defaulted
>> > >> > to the byte array serializer and it was a source of unending
>> > frustration
>> > >> > and confusion. Since it wasn't a required config people just went
>> > along
>> > >> > plugging in whatever objects they had, and thinking that changing
>> the
>> > >> > parametric types would somehow help. Then they would get a class
>> case
>> > >> > exception and assume our stuff was somehow busted, not realizing we
>> > had
>> > >> > helpfully configured a type different from what they were passing
>> in
>> > >> under
>> > >> > the covers. So I think it is actually good for people to think: how
>> > am I
>> > >> > serializing my data, and getting that exception will make them ask
>> > that
>> > >> > question right?
>> > >> >
>> > >> > -Jay
>> > >> >
>> > >> > On Fri, Jan 23, 2015 at 9:06 PM, Joe Stein 
>> > >> wrote:
>> > >> >
>> > >> >> Should value.serializer in the new java producer be defaulted to
>> > >> >> Array[Byte] ?
>> > >> >>
>> > >> >> I was working on testing some upgrade paths and got this
>> > >> >>
>> > >> >> ! return exception in callback when buffer cannot accept
>> message
>> > >> >>
>> > >> >>   ConfigException: Missing required configuration
>> > >> "value.serializer"
>> > >> >> which has no default value. (ConfigDef.java:124)
>> > >> >>
>> > >> >>
>>  org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >>
>> >
>> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >>
>> >
>> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:235)
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >>
>> >
>> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:129)
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >>
>> >
>> ly.stealth.testing.BaseSpec$class.createNewKafkaProducer(BaseSpec.scala:42)
>> > >> >>
>> > >> >>
>> > >>
>> ly.stealth.testing.KafkaSpec.createNewKafkaProducer(KafkaSpec.scala:36)
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >>
>> >
>> ly.stealth.testing.KafkaSpec$$anonfun$3$$anonfun$apply$37.apply(KafkaSpec.scala:175)
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >>
>> >
>> ly.stealth.testing.KafkaSpec$$anonfun$3$$anonfun$apply$37.apply(KafkaSpec.scala:170)
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On Fri, Jan 23, 2015 at 5:55 PM, Jun Rao 
>> wrote:
>> > >> >>
>> > >> >> > This is a reminder that the deadline for the vote is this
>> Monday,
>> > Jan
>> > >> >> 26,
>> > >> >> > 7pm PT.
>> > >> >> >
>> > >> >> > Thanks,
>> > >> >> >
>> > >> >> > Jun
>> > >> >> >
>> > >> >> > On Wed, Jan 21, 2015 at 8:28 AM, Jun Rao 
>> wrote:
>> > >> >> >
>> > >> >> >> This is the second candidate for release of Apache Kafka
>> 0.8.2.0.
>> > >> There
>> > >> >> >> has been some changes since the 0.8.2 beta release, especially
>> in
>> > >> the
>>

Re: unable to delete topic with 0.8.2 rc2

[Ari Flink]

unsubscribe

On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:

> Jun,
>   I made an attempt at fixing that issue as part of this JIRA
>   https://issues.apache.org/jira/browse/KAFKA-1507 .
> As Jay pointed out there should be admin api if there is more info on
> this api I am interested in adding/fixing this issue.
> Thanks,
> Harsha
>
> On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > Yes, that's the issue. Currently, topics can be auto-created on
> > TopicMetadataRequest, which can be issued from both the producer and the
> > consumer. To prevent that, you would need to stop the producer and the
> > consumer before deleting a topic. We plan to address this issue once we
> > have a separate request for creating topics.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> >
> > > There could be another case where if you have auto.create.topics.enable
> > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > recreate topics. So if you issued a delete topic command and you have
> > > producers running or consumers? too which is issuing a
> > > TopicMetadataRequest than the topic will be recreated.
> > > -Harsha
> > >
> > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > cversion did change (incremented by 2) when I issue the delete
> command.
> > > >
> > > > From the logs on the conroller broker (also the leader for the
> topic), it
> > > > looks like the delete proceeds, and then the topic gets recreated
> > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > consumer client app trying to consume the topic. Also, the consumer
> is
> > > > not
> > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > > problem?
> > > >
> > > >
> > > > 2015-01-26 07:02:14,281  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > mytopic
> > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > deletion callback for mytopic
> > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > Partition deletion callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to OfflineReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > log.OffsetIndex - Deleting index
> > > > /mypath/mytopic-0/.index
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> log.LogManager
> > > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > > > 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> > > > [Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,314  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 201

Re: Increased CPU usage with 0.8.2-beta

[Mathias Söderberg]

Hi Neha,

I sent an e-mail earlier today, but noticed now that it didn't actually go
through.

Anyhow, I've attached two files, one with output from a 10 minute run and
one with output from a 30 minute run. Realized that maybe I should've done
one or two runs with 0.8.1.1 as well, but nevertheless.

I upgraded our staging cluster to 0.8.2.0-rc2, and I'm seeing the same CPU
usage as with the beta version (basically pegging all cores). If I manage
to find the time I'll do another run with hprof on the rc2 version later
today.

Best regards,
Mathias

On Tue Dec 09 2014 at 10:08:21 PM Neha Narkhede  wrote:

> The following should be sufficient
>
> java
> -agentlib:hprof=cpu=samples,depth=100,interval=20,lineno=
> y,thread=y,file=kafka.hprof
> 
>
> You would need to start the Kafka server with the settings above for
> sometime until you observe the problem.
>
> On Tue, Dec 9, 2014 at 3:47 AM, Mathias Söderberg <
> mathias.soederb...@gmail.com> wrote:
>
> > Hi Neha,
> >
> > Yeah sure. I'm not familiar with hprof, so any particular options I
> should
> > include or just run with defaults?
> >
> > Best regards,
> > Mathias
> >
> > On Mon Dec 08 2014 at 7:41:32 PM Neha Narkhede 
> wrote:
> >
> > > Thanks for reporting the issue. Would you mind running hprof and
> sending
> > > the output?
> > >
> > > On Mon, Dec 8, 2014 at 1:25 AM, Mathias Söderberg <
> > > mathias.soederb...@gmail.com> wrote:
> > >
> > > > Good day,
> > > >
> > > > I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed
> > that
> > > > the CPU usage on the broker machines went up by roughly 40%, from
> ~60%
> > to
> > > > ~100% and am wondering if anyone else has experienced something
> > similar?
> > > > The load average also went up by 2x-3x.
> > > >
> > > > We're running on EC2 and the cluster currently consists of four
> > > m1.xlarge,
> > > > with roughly 1100 topics / 4000 partitions. Using Java 7 (1.7.0_65 to
> > be
> > > > exact) and Scala 2.9.2. Configurations can be found over here:
> > > > https://gist.github.com/mthssdrbrg/7df34a795e07eef10262.
> > > >
> > > > I'm assuming that this is not expected behaviour for 0.8.2-beta?
> > > >
> > > > Best regards,
> > > > Mathias
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Neha
> > >
> >
>
>
>
> --
> Thanks,
> Neha
>


kafka-20141222-30min.hprof.gz
Description: GNU Zip compressed data


kafka-20141222-10min.hprof.gz
Description: GNU Zip compressed data

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Jun Rao]

+1 (binding) Verified quick start and unit tests.

Thanks,

Jun

On Wed, Jan 21, 2015 at 8:28 AM, Jun Rao  wrote:

> This is the second candidate for release of Apache Kafka 0.8.2.0. There
> has been some changes since the 0.8.2 beta release, especially in the new
> java producer api and jmx mbean names. It would be great if people can test
> this out thoroughly.
>
> Release Notes for the 0.8.2.0 release
>
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Jan 26h, 7pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2
> (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/javadoc/
>
> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=058d58adef2ab2787e49d8efeefd61bb3d32f99c
> (commit 0b312a6b9f0833d38eec434bfff4c647c1814564)
>
> /***
>
> Thanks,
>
> Jun
>
>

Re: unable to delete topic with 0.8.2 rc2

[Harsha]

Jun,
  I made an attempt at fixing that issue as part of this JIRA
  https://issues.apache.org/jira/browse/KAFKA-1507 . 
As Jay pointed out there should be admin api if there is more info on
this api I am interested in adding/fixing this issue.
Thanks,
Harsha

On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> Yes, that's the issue. Currently, topics can be auto-created on
> TopicMetadataRequest, which can be issued from both the producer and the
> consumer. To prevent that, you would need to stop the producer and the
> consumer before deleting a topic. We plan to address this issue once we
> have a separate request for creating topics.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> 
> > There could be another case where if you have auto.create.topics.enable
> > to set to true ( its true by default) . Any TopicMetadataRequest can
> > recreate topics. So if you issued a delete topic command and you have
> > producers running or consumers? too which is issuing a
> > TopicMetadataRequest than the topic will be recreated.
> > -Harsha
> >
> > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > cversion did change (incremented by 2) when I issue the delete command.
> > >
> > > From the logs on the conroller broker (also the leader for the topic), it
> > > looks like the delete proceeds, and then the topic gets recreated
> > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > consumer client app trying to consume the topic. Also, the consumer is
> > > not
> > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > problem?
> > >
> > >
> > > 2015-01-26 07:02:14,281  INFO
> > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > mytopic
> > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > deletion callback for mytopic
> > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > Partition deletion callback for [mytopic,0]
> > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > 6]: Invoking state change to OfflineReplica for replicas
> > >
> > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > >
> > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > Removed fetcher for partitions [mytopic,0]
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > Removed fetcher for partitions [mytopic,0]
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > log.OffsetIndex - Deleting index
> > > /mypath/mytopic-0/.index
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7] log.LogManager
> > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > > 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> > > [Topic=mytopic,Partition=0,Replica=6]
> > > 2015-01-26 07:02:14,314  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > [delete-topics-thread-6], Deletion for replicas 7 for partition
> > > [mytopic,0] of topic mytopic in progress
> > > 2015-01-26 07:02:1

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Yes, that's the issue. Currently, topics can be auto-created on
TopicMetadataRequest, which can be issued from both the producer and the
consumer. To prevent that, you would need to stop the producer and the
consumer before deleting a topic. We plan to address this issue once we
have a separate request for creating topics.

Thanks,

Jun

On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:

> There could be another case where if you have auto.create.topics.enable
> to set to true ( its true by default) . Any TopicMetadataRequest can
> recreate topics. So if you issued a delete topic command and you have
> producers running or consumers? too which is issuing a
> TopicMetadataRequest than the topic will be recreated.
> -Harsha
>
> On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > cversion did change (incremented by 2) when I issue the delete command.
> >
> > From the logs on the conroller broker (also the leader for the topic), it
> > looks like the delete proceeds, and then the topic gets recreated
> > immediately (highlighted in yellow). It appears maybe it’s due to a
> > consumer client app trying to consume the topic. Also, the consumer is
> > not
> > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > problem?
> >
> >
> > 2015-01-26 07:02:14,281  INFO
> > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > controller.PartitionStateMachine$DeleteTopicsListener -
> > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > mytopic
> > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Handling deletion for topics mytopic
> > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > deletion callback for mytopic
> > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > Partition deletion callback for [mytopic,0]
> > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > controller.ReplicaStateMachine - [Replica state machine on controller
> > 6]: Invoking state change to OfflineReplica for replicas
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > controller.KafkaController - [Controller 6]: New leader and ISR for
> > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > controller.KafkaController - [Controller 6]: New leader and ISR for
> > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > controller.ReplicaStateMachine - [Replica state machine on controller
> > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > Removed fetcher for partitions [mytopic,0]
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > Removed fetcher for partitions [mytopic,0]
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > log.OffsetIndex - Deleting index
> > /mypath/mytopic-0/.index
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7] log.LogManager
> > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> > controller.ReplicaStateMachine - [Replica state machine on controller
> > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> > [Topic=mytopic,Partition=0,Replica=6]
> > 2015-01-26 07:02:14,314  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Handling deletion for topics mytopic
> > 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Deletion for replicas 7 for partition
> > [mytopic,0] of topic mytopic in progress
> > 2015-01-26 07:02:14,316  INFO [Controller-6-to-broker-7-send-thread]
> > controller.ReplicaStateMachine - [Replica state machine on controller
> > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> > [Topic=mytopic,Partition=0,Replica=7]
> > 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Handling deletion for topics mytopic
> > 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> > controll

Re: unable to delete topic with 0.8.2 rc2

[Harsha]

There could be another case where if you have auto.create.topics.enable
to set to true ( its true by default) . Any TopicMetadataRequest can
recreate topics. So if you issued a delete topic command and you have
producers running or consumers? too which is issuing a
TopicMetadataRequest than the topic will be recreated.
-Harsha

On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> cversion did change (incremented by 2) when I issue the delete command.
> 
> From the logs on the conroller broker (also the leader for the topic), it
> looks like the delete proceeds, and then the topic gets recreated
> immediately (highlighted in yellow). It appears maybe it’s due to a
> consumer client app trying to consume the topic. Also, the consumer is
> not
> yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> problem?
> 
> 
> 2015-01-26 07:02:14,281  INFO
> [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> controller.PartitionStateMachine$DeleteTopicsListener -
> [DeleteTopicsListener on 6]: Starting topic deletion for topics
> mytopic
> 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Handling deletion for topics mytopic
> 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Deletion of topic mytopic (re)started
> 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> deletion callback for mytopic
> 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager - [Topic Deletion Manager 6],
> Partition deletion callback for [mytopic,0]
> 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to OfflineReplica for replicas
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> controller.KafkaController - [Controller 6]: New leader and ISR for
> partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> controller.KafkaController - [Controller 6]: New leader and ISR for
> partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to ReplicaDeletionStarted for replicas
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> Removed fetcher for partitions [mytopic,0]
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> Removed fetcher for partitions [mytopic,0]
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> log.OffsetIndex - Deleting index
> /mypath/mytopic-0/.index
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7] log.LogManager
> - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> [Topic=mytopic,Partition=0,Replica=6]
> 2015-01-26 07:02:14,314  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Handling deletion for topics mytopic
> 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Deletion for replicas 7 for partition
> [mytopic,0] of topic mytopic in progress
> 2015-01-26 07:02:14,316  INFO [Controller-6-to-broker-7-send-thread]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> [Topic=mytopic,Partition=0,Replica=7]
> 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Handling deletion for topics mytopic
> 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to NonExistentReplica for replicas
> [Topic=mytopic,Partition=0,Replica=6],[Topic=mytopic,Partition=0,Replica=7]
> 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> controller.PartitionStateMachine - [Partition state machine on
> Controller 6]: Invoking state change to OfflinePartition for
> partitions [mytopic,0]
> 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> controller.PartitionStateMachine - [Partition state machine on
> Controller 6]:

Re: no brokers found when trying to rebalance

[Jeff Holoman]

Yeah the issue was mostly that literally ALL of the docs use / as the
chroot.  People kept getting tripped up with the ZK URL with /kafka, it's a
bit funky eg:

host1.domain.com:2181,host2.domain.com:2181,host3.domain.com:2181/kafka. So
we opted to change it back to what everyone else is doing by default.

I checked the latest PDF and it has this change updated.
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-kafka/latest/PDF/cloudera-kafka.pdf

 Sorry about the confusion.

Jeff




On Sat, Jan 24, 2015 at 4:39 PM, Tim Smith  wrote:

> Yes, turns out that during the upgrade from CDH5.2 to CDH5.3, the namespace
> changed from "/kafka" to simply "/" on Zk. I spoke with the developers over
> at Cloudera and they said, using "/" seemed to be the convention. I am
> going to open a case with them to clearly document the change so other
> customers don't end up scratching their heads :)
>
> I think "/kafka" is just cleaner.
>
>
>
> On Fri, Jan 23, 2015 at 9:34 AM, Jun Rao  wrote:
>
> > Is the broker configured with the correct ZK url and the right namespace?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Jan 23, 2015 at 12:17 AM, Tim Smith  wrote:
> >
> > > Using kafka 0.8.1.1, the cluster had been healthy with producers and
> > > consumers being able to function well. After a restart of the cluster,
> it
> > > looks like consumers are locked out.
> > >
> > > When I try to consume from a topic, I get this warning:
> > > [2015-01-23 07:48:50,667] WARN
> > > [console-consumer-32626_kafka-node-1.abc.net-1421999330230-a3d2b9e1],
> no
> > > brokers found when trying to rebalance.
> > > (kafka.consumer.ZookeeperConsumerConnector)
> > >
> > > I don't see any errors in server.log on the kafka nodes and there
> aren't
> > > any firewalls between the hosts (brokers, consumers and producers).
> > >
> > > I can query the topic state:
> > > $ /opt/cloudera/parcels/CLABS_KAFKA/bin/kafka-topics --zookeeper
> > > zookeeper-node-1:2181/kafka --describe --topic rawunstruct
> > > Topic:rawunstruct PartitionCount:5 ReplicationFactor:3 Configs:
> > > Topic: rawunstruct Partition: 0 Leader: 328 Replicas: 328,327,329 Isr:
> > > 328,327
> > > Topic: rawunstruct Partition: 1 Leader: 328 Replicas: 329,328,327 Isr:
> > > 328,327
> > > Topic: rawunstruct Partition: 2 Leader: 328 Replicas: 327,329,328 Isr:
> > > 328,327
> > > Topic: rawunstruct Partition: 3 Leader: 328 Replicas: 328,329,327 Isr:
> > > 328,327
> > > Topic: rawunstruct Partition: 4 Leader: 328 Replicas: 329,327,328 Isr:
> > > 328,327
> > >
> > > However, when I list /kafka/brokers/ids on any of the ZK servers, I
> don't
> > > see any brokers ids.
> > >
> > > I did upgrade from CDH 5.2 to 5.3 but other components in the stack
> seem
> > to
> > > be able to talk to Zookeeper just fine.
> > >
> > > Any pointers for troubleshooting?
> > >
> > > Thanks
> > >
> >
>



-- 
Jeff Holoman
Systems Engineer
678-612-9519

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

[Manikumar Reddy]

+1 (Non-binding)
Verified source package, unit tests, release build, topic deletion,
compaction and random testing

On Mon, Jan 26, 2015 at 6:14 AM, Neha Narkhede  wrote:

> +1 (binding)
> Verified keys, quick start, unit tests.
>
> On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein  wrote:
>
> > That makes sense, thanks!
> >
> > On Sat, Jan 24, 2015 at 7:00 PM, Jay Kreps  wrote:
> >
> > > But I think the flaw in trying to guess what kind of serializer they
> will
> > > use is when we get it wrong. Basically let's say we guess "String". Say
> > 30%
> > > of the time we will be right and we will save the two configuration
> > lines.
> > > 70% of the time we will be wrong and the user gets a super cryptic
> > > ClassCastException: "xyz cannot be cast to [B" (because [B is how java
> > > chooses to display the byte array class just to up the pain), then they
> > > figure out how to subscribe to our mailing list and email us the
> cryptic
> > > exception, then we explain about how we helpfully set these properties
> > for
> > > them to save them time. :-)
> > >
> > >
> https://www.google.com/?gws_rd=ssl#q=kafka+classcastexception+%22%5BB%22
> > >
> > > I think basically we did this experiment with the old clients and the
> > > conclusion is that serialization is something you basically have to
> think
> > > about to use Kafka and trying to guess just makes things worse.
> > >
> > > -Jay
> > >
> > > On Sat, Jan 24, 2015 at 2:51 PM, Joe Stein 
> wrote:
> > >
> > >> Maybe. I think the StringSerialzer could look more like a typical type
> > of
> > >> message.  Instead of encoding being a property it would be more
> > typically
> > >> just written in the bytes.
> > >>
> > >> On Sat, Jan 24, 2015 at 12:12 AM, Jay Kreps 
> > wrote:
> > >>
> > >> > I don't think so--see if you buy my explanation. We previously
> > defaulted
> > >> > to the byte array serializer and it was a source of unending
> > frustration
> > >> > and confusion. Since it wasn't a required config people just went
> > along
> > >> > plugging in whatever objects they had, and thinking that changing
> the
> > >> > parametric types would somehow help. Then they would get a class
> case
> > >> > exception and assume our stuff was somehow busted, not realizing we
> > had
> > >> > helpfully configured a type different from what they were passing in
> > >> under
> > >> > the covers. So I think it is actually good for people to think: how
> > am I
> > >> > serializing my data, and getting that exception will make them ask
> > that
> > >> > question right?
> > >> >
> > >> > -Jay
> > >> >
> > >> > On Fri, Jan 23, 2015 at 9:06 PM, Joe Stein 
> > >> wrote:
> > >> >
> > >> >> Should value.serializer in the new java producer be defaulted to
> > >> >> Array[Byte] ?
> > >> >>
> > >> >> I was working on testing some upgrade paths and got this
> > >> >>
> > >> >> ! return exception in callback when buffer cannot accept
> message
> > >> >>
> > >> >>   ConfigException: Missing required configuration
> > >> "value.serializer"
> > >> >> which has no default value. (ConfigDef.java:124)
> > >> >>
> > >> >>
>  org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:235)
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:129)
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
> ly.stealth.testing.BaseSpec$class.createNewKafkaProducer(BaseSpec.scala:42)
> > >> >>
> > >> >>
> > >>
> ly.stealth.testing.KafkaSpec.createNewKafkaProducer(KafkaSpec.scala:36)
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
> ly.stealth.testing.KafkaSpec$$anonfun$3$$anonfun$apply$37.apply(KafkaSpec.scala:175)
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
> ly.stealth.testing.KafkaSpec$$anonfun$3$$anonfun$apply$37.apply(KafkaSpec.scala:170)
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, Jan 23, 2015 at 5:55 PM, Jun Rao  wrote:
> > >> >>
> > >> >> > This is a reminder that the deadline for the vote is this Monday,
> > Jan
> > >> >> 26,
> > >> >> > 7pm PT.
> > >> >> >
> > >> >> > Thanks,
> > >> >> >
> > >> >> > Jun
> > >> >> >
> > >> >> > On Wed, Jan 21, 2015 at 8:28 AM, Jun Rao 
> wrote:
> > >> >> >
> > >> >> >> This is the second candidate for release of Apache Kafka
> 0.8.2.0.
> > >> There
> > >> >> >> has been some changes since the 0.8.2 beta release, especially
> in
> > >> the
> > >> >> new
> > >> >> >> java producer api and jmx mbean names. It would be great if
> people
> > >> can
> > >> >> test
> > >> >> >> this out thoroughly.
> > >> >> >>
> > >> >> >> Release Notes for the 0.8.2.0 release
> > >> >> >>
> > >> >> >>
> > >> >>
> > >>
> >
> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/RELEASE_NOTES.html
> > >> >> >>
> > >> >> >> *** Please download, test