Re: Fault Injection

[Timothy Chen]

Hi Colin,

The Kibosh code is just a README for now, is it going to be published soon?

Tim

On Tue, Aug 22, 2017 at 11:44 AM, Colin McCabe  wrote:
> Hi all,
>
> I've been working on a fault injector for Apache Kafka.  The general
> idea is to create faults such as network partitions or disk failures,
> and see what happens in the cluster.  The fault injector can run as part
> of a ducktape system test, or standalone.
>
> The fault injector has two processes: a coordinator, and an agent.  The
> agent process is responsible for actually implementing the faults.  For
> example, it might run iptables, send signals to processes, generate a
> lot of load, or do something else to disrupt the computer it is running
> on.  We run an agent process on each node where we would like to
> potentially inject faults.  So it will run alongside the brokers,
> zookeeper nodes, etc.
>
> The coordinator process is responsible for communicating with the agent
> processes and for scheduling faults.  For example, the coordinator can
> be instructed to create a fault immediately on several nodes.  Or it can
> be instructed to create faults over time, based on a pseudorandom seed.
> Both the coordinator and the agent expose a REST interface that accepts
> objects serialized via JSON.
>
> I think two kinds of faults will be especially interesting: network
> faults, and disk errors.  Simulating network faults in a Linux
> environment is relatively straightforward using iptables.  Disk errors
> are tougher to simulate, but I have written a FUSE filesystem to do
> this.  The  filesystem essentially simulates a bind mount in most cases,
> but it can take a JSON specification telling it to inject certain
> faults.  (Disk errors seem especially relevant to the ongoing work on
> JBOD.)
>
> Although it's not a user-visible component, I think having a fault
> injector will be really great for Kafka users.  It will really help us
> stress test Kafka in more situations.  I'm going to post some patches in
> a day or two-- it would be great to get some feedback.  Check out
> https://cwiki.apache.org/confluence/display/KAFKA/Fault+Injection
>
> best,
> Colin

[jira] [Commented] (KAFKA-2339) broker becomes unavailable if bad data is passed through the protocol

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630981#comment-14630981
 ] 

Timothy Chen commented on KAFKA-2339:
-

Somehow I can't produce this with kafkacat and latest trunk:

Timothys-MacBook-Pro :: ~/src/kafka ‹trunk*› » kafkacat -P -p 3..5 -t test -b 
localhost:9092  
   1 ↵
abc
% ERROR: Failed to produce message (3 bytes): Local: Unknown partition

Which version of Kafka are you using?

> broker becomes unavailable if bad data is passed through the protocol
> -
>
> Key: KAFKA-2339
> URL: https://issues.apache.org/jira/browse/KAFKA-2339
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Timothy Chen
>Priority: Critical
> Fix For: 0.8.3
>
>
> I ran into a situation that a non integer value got past for the partition 
> and the brokers went bonkers.
> reproducible
> {code}
> ah="1..2"
> echo "don't do this in production"|kafkacat -b localhost:9092 -p $ah
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2339) broker becomes unavailable if bad data is passed through the protocol

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen reassigned KAFKA-2339:
---

Assignee: Timothy Chen

> broker becomes unavailable if bad data is passed through the protocol
> -
>
> Key: KAFKA-2339
> URL: https://issues.apache.org/jira/browse/KAFKA-2339
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>    Assignee: Timothy Chen
>Priority: Critical
> Fix For: 0.8.3
>
>
> I ran into a situation that a non integer value got past for the partition 
> and the brokers went bonkers.
> reproducible
> {code}
> ah="1..2"
> echo "don't do this in production"|kafkacat -b localhost:9092 -p $ah
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36474: Patch for KAFKA-2188

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36474/
---

(Updated July 15, 2015, 8:19 a.m.)


Review request for kafka and Joe Stein.


Bugs: KAFKA-2188
https://issues.apache.org/jira/browse/KAFKA-2188


Repository: kafka


Description
---

KAFKA-2188 - JBOD Support


Diffs (updated)
-

  core/src/main/scala/kafka/cluster/Partition.scala 2649090 
  core/src/main/scala/kafka/common/GenericKafkaStorageException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/controller/KafkaController.scala b4fc755 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala bb6b5c8 
  core/src/main/scala/kafka/log/Log.scala e5e8007 
  core/src/main/scala/kafka/log/LogManager.scala 69386c1 
  core/src/main/scala/kafka/log/LogSegment.scala 1377e8f 
  core/src/main/scala/kafka/server/KafkaApis.scala 18f5b5b 
  core/src/main/scala/kafka/server/KafkaConfig.scala dbe170f 
  core/src/main/scala/kafka/server/OffsetCheckpoint.scala 8c5b054 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala c89d00b 
  core/src/main/scala/kafka/server/ReplicaManager.scala 795220e 
  core/src/main/scala/kafka/utils/CoreUtils.scala 168a18d 
  core/src/main/scala/kafka/utils/ZkUtils.scala 166814c 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala a13f2be 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
60cd824 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 00d5933 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 17e9fe4 

Diff: https://reviews.apache.org/r/36474/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627708#comment-14627708
 ] 

Timothy Chen commented on KAFKA-2188:
-

Created reviewboard https://reviews.apache.org/r/36503/diff/
 against branch origin/trunk

> JBOD Support
> 
>
> Key: KAFKA-2188
> URL: https://issues.apache.org/jira/browse/KAFKA-2188
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2188.patch, KAFKA-2188.patch, KAFKA-2188.patch
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-2188:

Attachment: KAFKA-2188.patch

> JBOD Support
> 
>
> Key: KAFKA-2188
> URL: https://issues.apache.org/jira/browse/KAFKA-2188
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2188.patch, KAFKA-2188.patch, KAFKA-2188.patch
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 36503: Patch for KAFKA-2188

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36503/
---

Review request for kafka.


Bugs: KAFKA-2188
https://issues.apache.org/jira/browse/KAFKA-2188


Repository: kafka


Description
---

KAFKA-2188 - JBOD Support


Diffs
-

  core/src/main/scala/kafka/cluster/Partition.scala 
2649090b6cbf8d442649f19fd7113a30d62bca91 
  core/src/main/scala/kafka/common/GenericKafkaStorageException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/controller/KafkaController.scala 
b4fc755641b9bbe8a6bf9c221a9ffaec0b94d6e8 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
bb6b5c8764522e7947bb08998256ce1deb717c84 
  core/src/main/scala/kafka/log/Log.scala 
e5e80079645ce6e6fe7bb1c2696d3dd21a07761b 
  core/src/main/scala/kafka/log/LogManager.scala 
69386c17153e5ef08a24d4f14b915e4316b121d8 
  core/src/main/scala/kafka/log/LogSegment.scala 
1377e8f322a3fedc683d93feaf27c955de528a4b 
  core/src/main/scala/kafka/server/KafkaApis.scala 
18f5b5b895af1469876b2223841fd90a2dd255e0 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
dbe170f87331f43e2dc30165080d2cb7dfe5fdbf 
  core/src/main/scala/kafka/server/OffsetCheckpoint.scala 
8c5b0546908d3b3affb9f48e2ece9ed252518783 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
c89d00b5976ffa34cafdae261239934b1b917bfe 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
795220e7f63d163be90738b4c1a39687b44c1395 
  core/src/main/scala/kafka/utils/CoreUtils.scala 
168a18d380c200ee566eccb6988dd1ae85ed5b09 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
166814c2959a429e20f400d1c0e523090ce37d91 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
a13f2bef8ee8c3d42192c9a60df092023e4d2ff9 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
60cd8249e6ec03349e20bb0a7226ea9cd66e6b17 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
17e9fe4c159a29033fe9a287db6ced2fdc3fa9d1 

Diff: https://reviews.apache.org/r/36503/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625910#comment-14625910
 ] 

Timothy Chen commented on KAFKA-2188:
-

Created reviewboard https://reviews.apache.org/r/36474/diff/
 against branch origin/trunk

> JBOD Support
> 
>
> Key: KAFKA-2188
> URL: https://issues.apache.org/jira/browse/KAFKA-2188
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2188.patch, KAFKA-2188.patch
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2188) JBOD Support

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-2188:

Attachment: KAFKA-2188.patch

> JBOD Support
> 
>
> Key: KAFKA-2188
> URL: https://issues.apache.org/jira/browse/KAFKA-2188
> Project: Kafka
>  Issue Type: Bug
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2188.patch, KAFKA-2188.patch
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-18+-+JBOD+Support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 36474: Patch for KAFKA-2188

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36474/
---

Review request for kafka.


Bugs: KAFKA-2188
https://issues.apache.org/jira/browse/KAFKA-2188


Repository: kafka


Description
---

KAFKA-2188 - JBOD Support


Diffs
-

  core/src/main/scala/kafka/cluster/Partition.scala 
2649090b6cbf8d442649f19fd7113a30d62bca91 
  core/src/main/scala/kafka/common/GenericKafkaStorageException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/controller/KafkaController.scala 
b4fc755641b9bbe8a6bf9c221a9ffaec0b94d6e8 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
bb6b5c8764522e7947bb08998256ce1deb717c84 
  core/src/main/scala/kafka/log/Log.scala 
e5e80079645ce6e6fe7bb1c2696d3dd21a07761b 
  core/src/main/scala/kafka/log/LogManager.scala 
69386c17153e5ef08a24d4f14b915e4316b121d8 
  core/src/main/scala/kafka/log/LogSegment.scala 
1377e8f322a3fedc683d93feaf27c955de528a4b 
  core/src/main/scala/kafka/server/KafkaApis.scala 
18f5b5b895af1469876b2223841fd90a2dd255e0 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
dbe170f87331f43e2dc30165080d2cb7dfe5fdbf 
  core/src/main/scala/kafka/server/OffsetCheckpoint.scala 
8c5b0546908d3b3affb9f48e2ece9ed252518783 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
c89d00b5976ffa34cafdae261239934b1b917bfe 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
795220e7f63d163be90738b4c1a39687b44c1395 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
166814c2959a429e20f400d1c0e523090ce37d91 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
a13f2bef8ee8c3d42192c9a60df092023e4d2ff9 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
60cd8249e6ec03349e20bb0a7226ea9cd66e6b17 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
17e9fe4c159a29033fe9a287db6ced2fdc3fa9d1 

Diff: https://reviews.apache.org/r/36474/diff/


Testing
---


Thanks,

Timothy Chen

Re: Guozhang Wang elected to Kafka PMC

[Timothy Chen]

Congrats Guozhang!!

Tim

On Tue, Jun 16, 2015 at 8:19 AM, Aditya Auradkar
 wrote:
> Congrats Guozhang!
>
> 
> From: Ashish Singh [asi...@cloudera.com]
> Sent: Monday, June 15, 2015 10:53 PM
> To: dev@kafka.apache.org
> Cc: Jun Rao
> Subject: Re: Guozhang Wang elected to Kafka PMC
>
> Congrats Guozhang!
>
> On Mon, Jun 15, 2015 at 10:20 PM, Sriharsha Chintalapani 
> wrote:
>
>> Congrats Guozhang.
>>
>> --
>> Harsha
>>
>>
>> On June 15, 2015 at 9:59:37 PM, Jun Rao (j...@confluent.io) wrote:
>>
>> Hi, Everyone,
>>
>> Guozhang Wang has been active in the Kafka community since he became a
>> Kafka committer about 7 months ago. I am glad to announce that Guozhang is
>> now a member of Kafka PMC.
>>
>> Congratulations, Guozhang!
>>
>> Jun
>>
>
>
>
> --
>
> Regards,
> Ashish

Re: [DISCUSSION] Message Metadata

[Timothy Chen]

Hi Guozhang,

I don't think that is publically accessible, can you update it to the
Kafka wiki?

Tim

On Fri, Nov 21, 2014 at 5:24 PM, Guozhang Wang  wrote:
> Hi all,
>
> I have updated the wiki page (
> https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Kafka+Enriched+Message+Metadata)
> according to people's comments and discussions offline.
>
> Guozhang
>
> On Thu, Nov 13, 2014 at 9:43 AM, Guozhang Wang  wrote:
>
>> Hi Jun,
>>
>> Sorry for the delay on your comments in the wiki page as well as this
>> thread; quite swamped now. I will get back to you as soon as I find some
>> time.
>>
>> Guozhang
>>
>> On Tue, Nov 11, 2014 at 6:26 PM, Jun Rao  wrote:
>>
>>> Thinking about this a bit more. For adding the auditing support, I am not
>>> sure if we need to change the message format by adding the application
>>> tags. An alternative way to do that is to add it in the producer client.
>>> For example, for each message payload (doesn't matter what the
>>> serialization mechanism is) that a producer receives, the producer can
>>> just
>>> add a header before the original payload. The header will contain all
>>> needed fields (e.g. timestamp, host, etc) for the purpose of auditing.
>>> This
>>> way, we don't need to change the message format and the auditing info can
>>> be added independent of the serialization mechanism of the message. The
>>> header can use a different serialization mechanism for better efficiency.
>>> For example, if we use Avro to serialize the header, the encoded bytes
>>> won't include the field names in the header. This is potentially more
>>> efficient than representing those fields as application tags in the
>>> message
>>> where the tags have to be explicitly store in every message.
>>>
>>> To make it easier for the client to add and make use of this kind of
>>> auditing support, I was imagining that we can add a ProducerFactory in the
>>> new java client. The ProducerFactory will create an instance of Producer
>>> based on a config property. By default, the current KafkaProducer will be
>>> returned. However, a user can plug in a different implementation of
>>> Producer that does auditing. For example, an implementation of an
>>> AuditProducer.send() can take the original ProducerRecord, add the header
>>> to the value byte array and then forward the record to an underlying
>>> KafkaProducer. We can add a similar ConsumerFactory to the new consumer
>>> client. If a user plugs in an implementation of the AuditingConsumer, the
>>> consumer will then be audited automatically.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Tue, Oct 21, 2014 at 4:06 PM, Guozhang Wang 
>>> wrote:
>>>
>>> > Hi Jun,
>>> >
>>> > Regarding 4) in your comment, after thinking it for a while I cannot
>>> come
>>> > up a way to it along with log compaction without adding new fields into
>>> the
>>> > current format on message set. Do you have a better way that do not
>>> require
>>> > protocol changes?
>>> >
>>> > Guozhang
>>> >
>>> > On Mon, Oct 20, 2014 at 9:53 AM, Guozhang Wang 
>>> wrote:
>>> >
>>> > > I have updated the wiki page incorporating received comments. We can
>>> > > discuss some more details on:
>>> > >
>>> > > 1. How we want to do audit? Whether we want to have in-built auditing
>>> on
>>> > > brokers or even MMs or use  an audit consumer to fetch all messages
>>> from
>>> > > just brokers.
>>> > >
>>> > > 2. How we can avoid de-/re-compression on brokers and MMs with log
>>> > > compaction turned on.
>>> > >
>>> > > 3. How we can resolve unclean leader election resulted data
>>> inconsistency
>>> > > with control messages.
>>> > >
>>> > > Guozhang
>>> > >
>>> > > On Sun, Oct 19, 2014 at 11:41 PM, Guozhang Wang 
>>> > > wrote:
>>> > >
>>> > >> Thanks for the detailed comments Jun! Some replies inlined.
>>> > >>
>>> > >> On Sun, Oct 19, 2014 at 7:42 PM, Jun Rao  wrote:
>>> > >>
>>> > >>> Hi, Guozhang,
>>> > >>>
>>> > >>> Thanks for the writeup.
>>> > >>>
>>> > >>> A few high level comments.
>>> > >>>
>>> > >>> 1. Associating (versioned) schemas to a topic can be a good thing
>>> > >>> overall.
>>> > >>> Yes, this could add a bit more management overhead in Kafka.
>>> However,
>>> > it
>>> > >>> makes sure that the data format contract between a producer and a
>>> > >>> consumer
>>> > >>> is kept and managed in a central place, instead of in the
>>> application.
>>> > >>> The
>>> > >>> latter is probably easier to start with, but is likely to be
>>> brittle in
>>> > >>> the
>>> > >>> long run.
>>> > >>>
>>> > >>
>>> > >> I am actually not proposing to not support associated versioned
>>> schemas
>>> > >> for topics, but to not let some core Kafka functionalities like
>>> auditing
>>> > >> being depend on schemas. I think this alone can separate the schema
>>> > >> management from Kafka piping management (i.e. making sure every
>>> single
>>> > >> message is delivered, and within some latency, etc). Adding
>>> additional
>>> > >> auditing info into an existing schema will force Kafka to be aware of
>>> > th

Re: Welcome Kafka's newest committer

[Timothy Chen]

Congrats Guozhang!

Tim

On Wed, Nov 19, 2014 at 4:05 PM, Neha Narkhede  wrote:
> Hi everyone,
>
> I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> become a committer. Guozhang has made significant contributions to Kafka
> over the past year, along with being very active on code reviews and the
> mailing list.
>
> Please join me in welcoming him.
>
> Thanks,
> Neha (on behalf of the Kafka PMC)

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

> On Aug. 6, 2014, 5:32 a.m., Timothy Chen wrote:
> > core/src/main/scala/kafka/server/TransactionManager.scala, line 300
> > <https://reviews.apache.org/r/23568/diff/6/?file=653418#file653418line300>
> >
> > I'm just trying to think if there is a chance to have overlapping tx 
> > ids especially around failover. I'm not familiar with the tx changes so 
> > just wondering if the range size can cause issues. Seems like you're 
> > incrementing the zk version on each id fetch and allow a range of the batch 
> > size of them. I was thinking if I sent 999 messages and then failed over, 
> > and the new broker that took over has a small batch size now, even though 
> > the version has incremented the new range now overlaps then right? It might 
> > already been handled but just want to throw it out there.
> 
> Dong Lin wrote:
> Sure. Thanks for taking time to help think about it.
> 
> Here I assume the batch size should be the same across producers 
> regardless of when they start. If a producer sends 999 messages, say with 
> txid in range 1000 to 1999, and fails over, the next producer will get the 
> txid in range 2000 to 2999. The range is guaranteed to be disjoint from 
> ranges used by previous/existing producers. Does this make sense?
> 
> Dong Lin wrote:
> Correction: I actually mean broker where I say producer in my response.
> 
> Timothy Chen wrote:
> The next producer will do so by looking at the default batch size with 
> the version, which is what I was thinking there really isn't anything 
> guarding aganist the range being changed from the code.
> The safest thing to do is perhaps write the range into zk node data and 
> when the current range is smaller have a big warning.
> But minimally we will want to leave a comment in the code on the batch 
> size to have this warning so this isn't something to experiment with as it 
> can mess up existing transactions.
> 
> Dong Lin wrote:
> It is the broker who looks at the default batch size with the version, 
> and allocate unique txId to producer upon request. In other words, this code 
> on the server side and we don't need to worry about Kafka user changing this 
> code. Am I right?
> 
> Timothy Chen wrote:
> You're right we don't need to worry about it for the user, but we need to 
> worry about it when we change the default and do a rolling upgrade.
> 
> Dong Lin wrote:
> Oh, I see your point. Sure, I agree that we should add a comment in the 
> TransactionManger code so that any future developer should be aware of the 
> effect of changing DefaultTransactionsIdBatchSize. But since this is just one 
> of the many other configuration, I am not sure if it is worth saving this 
> default value in zk node.
> 
> Does that address your concern? Thanks!

Yes I don't think we need to do write it out, just a comment should be good IMO.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23568/
> ---
> 
> (Updated Aug. 6, 2014, 4:25 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1523
> https://issues.apache.org/jira/browse/KAFKA-1523
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1523 Transaction manager and its failover handling.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 003a09c6160618bc94858ebc0d806b2aa4158e0a 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 134aef9c88068443d4d465189f376dd78605b4f8 
>   core/src/main/scala/kafka/common/Topic.scala 
> ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
>   core/src/main/scala/kafka/message/Message.scala 
> d2a7293c7be4022af30884330924791340acc5c1 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1a45f8716ccc0398cf9395d91d66199d16882aae 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 897783cb756de548a8b634876f729b63ffe9925e 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
>   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
> 
> Diff: https://reviews.apache.org/r/23568/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

> On Aug. 6, 2014, 5:32 a.m., Timothy Chen wrote:
> > core/src/main/scala/kafka/server/TransactionManager.scala, line 300
> > <https://reviews.apache.org/r/23568/diff/6/?file=653418#file653418line300>
> >
> > I'm just trying to think if there is a chance to have overlapping tx 
> > ids especially around failover. I'm not familiar with the tx changes so 
> > just wondering if the range size can cause issues. Seems like you're 
> > incrementing the zk version on each id fetch and allow a range of the batch 
> > size of them. I was thinking if I sent 999 messages and then failed over, 
> > and the new broker that took over has a small batch size now, even though 
> > the version has incremented the new range now overlaps then right? It might 
> > already been handled but just want to throw it out there.
> 
> Dong Lin wrote:
> Sure. Thanks for taking time to help think about it.
> 
> Here I assume the batch size should be the same across producers 
> regardless of when they start. If a producer sends 999 messages, say with 
> txid in range 1000 to 1999, and fails over, the next producer will get the 
> txid in range 2000 to 2999. The range is guaranteed to be disjoint from 
> ranges used by previous/existing producers. Does this make sense?
> 
> Dong Lin wrote:
> Correction: I actually mean broker where I say producer in my response.
> 
> Timothy Chen wrote:
> The next producer will do so by looking at the default batch size with 
> the version, which is what I was thinking there really isn't anything 
> guarding aganist the range being changed from the code.
> The safest thing to do is perhaps write the range into zk node data and 
> when the current range is smaller have a big warning.
> But minimally we will want to leave a comment in the code on the batch 
> size to have this warning so this isn't something to experiment with as it 
> can mess up existing transactions.
> 
> Dong Lin wrote:
> It is the broker who looks at the default batch size with the version, 
> and allocate unique txId to producer upon request. In other words, this code 
> on the server side and we don't need to worry about Kafka user changing this 
> code. Am I right?

You're right we don't need to worry about it for the user, but we need to worry 
about it when we change the default and do a rolling upgrade.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23568/
> ---
> 
> (Updated Aug. 6, 2014, 4:25 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1523
> https://issues.apache.org/jira/browse/KAFKA-1523
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1523 Transaction manager and its failover handling.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 003a09c6160618bc94858ebc0d806b2aa4158e0a 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 134aef9c88068443d4d465189f376dd78605b4f8 
>   core/src/main/scala/kafka/common/Topic.scala 
> ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
>   core/src/main/scala/kafka/message/Message.scala 
> d2a7293c7be4022af30884330924791340acc5c1 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1a45f8716ccc0398cf9395d91d66199d16882aae 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 897783cb756de548a8b634876f729b63ffe9925e 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
>   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
> 
> Diff: https://reviews.apache.org/r/23568/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

> On Aug. 6, 2014, 5:32 a.m., Timothy Chen wrote:
> > core/src/main/scala/kafka/server/TransactionManager.scala, line 300
> > <https://reviews.apache.org/r/23568/diff/6/?file=653418#file653418line300>
> >
> > I'm just trying to think if there is a chance to have overlapping tx 
> > ids especially around failover. I'm not familiar with the tx changes so 
> > just wondering if the range size can cause issues. Seems like you're 
> > incrementing the zk version on each id fetch and allow a range of the batch 
> > size of them. I was thinking if I sent 999 messages and then failed over, 
> > and the new broker that took over has a small batch size now, even though 
> > the version has incremented the new range now overlaps then right? It might 
> > already been handled but just want to throw it out there.
> 
> Dong Lin wrote:
> Sure. Thanks for taking time to help think about it.
> 
> Here I assume the batch size should be the same across producers 
> regardless of when they start. If a producer sends 999 messages, say with 
> txid in range 1000 to 1999, and fails over, the next producer will get the 
> txid in range 2000 to 2999. The range is guaranteed to be disjoint from 
> ranges used by previous/existing producers. Does this make sense?
> 
> Dong Lin wrote:
> Correction: I actually mean broker where I say producer in my response.

The next producer will do so by looking at the default batch size with the 
version, which is what I was thinking there really isn't anything guarding 
aganist the range being changed from the code.
The safest thing to do is perhaps write the range into zk node data and when 
the current range is smaller have a big warning.
But minimally we will want to leave a comment in the code on the batch size to 
have this warning so this isn't something to experiment with as it can mess up 
existing transactions.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23568/
> ---
> 
> (Updated Aug. 6, 2014, 4:25 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1523
> https://issues.apache.org/jira/browse/KAFKA-1523
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1523 Transaction manager and its failover handling.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 003a09c6160618bc94858ebc0d806b2aa4158e0a 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 134aef9c88068443d4d465189f376dd78605b4f8 
>   core/src/main/scala/kafka/common/Topic.scala 
> ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
>   core/src/main/scala/kafka/message/Message.scala 
> d2a7293c7be4022af30884330924791340acc5c1 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1a45f8716ccc0398cf9395d91d66199d16882aae 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 897783cb756de548a8b634876f729b63ffe9925e 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
>   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
> 
> Diff: https://reviews.apache.org/r/23568/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49696
---



core/src/main/scala/kafka/server/TransactionManager.scala
<https://reviews.apache.org/r/23568/#comment86986>

I'm just trying to think if there is a chance to have overlapping tx ids 
especially around failover. I'm not familiar with the tx changes so just 
wondering if the range size can cause issues. Seems like you're incrementing 
the zk version on each id fetch and allow a range of the batch size of them. I 
was thinking if I sent 999 messages and then failed over, and the new broker 
that took over has a small batch size now, even though the version has 
incremented the new range now overlaps then right? It might already been 
handled but just want to throw it out there. 


- Timothy Chen


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23568/
> ---
> 
> (Updated Aug. 6, 2014, 4:25 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1523
> https://issues.apache.org/jira/browse/KAFKA-1523
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1523 Transaction manager and its failover handling.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 003a09c6160618bc94858ebc0d806b2aa4158e0a 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 134aef9c88068443d4d465189f376dd78605b4f8 
>   core/src/main/scala/kafka/common/Topic.scala 
> ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
>   core/src/main/scala/kafka/message/Message.scala 
> d2a7293c7be4022af30884330924791340acc5c1 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1a45f8716ccc0398cf9395d91d66199d16882aae 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 897783cb756de548a8b634876f729b63ffe9925e 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
>   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
> 
> Diff: https://reviews.apache.org/r/23568/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>

Re: Review Request 23568: Patch for KAFKA-1523

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23568/#review49692
---



core/src/main/scala/kafka/admin/TopicCommand.scala
<https://reviews.apache.org/r/23568/#comment86976>

Why not use the InternalTopics array here?



core/src/main/scala/kafka/server/KafkaApis.scala
<https://reviews.apache.org/r/23568/#comment86978>

that is quite a mouthful, should it be called just 
producerTxRequestToCoordinator?



core/src/main/scala/kafka/server/TransactionManager.scala
<https://reviews.apache.org/r/23568/#comment86979>

If the batch size changes to a smaller number does the tx Id range overlap 
then?


- Timothy Chen


On Aug. 6, 2014, 4:25 a.m., Dong Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23568/
> ---
> 
> (Updated Aug. 6, 2014, 4:25 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1523
> https://issues.apache.org/jira/browse/KAFKA-1523
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1523 Transaction manager and its failover handling.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 003a09c6160618bc94858ebc0d806b2aa4158e0a 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 134aef9c88068443d4d465189f376dd78605b4f8 
>   core/src/main/scala/kafka/common/Topic.scala 
> ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> ecbfa0f328ba6a652a758ab20cacef324a8b2fb8 
>   core/src/main/scala/kafka/message/Message.scala 
> d2a7293c7be4022af30884330924791340acc5c1 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> fd5f12ee31e78cdcda8e24a0ab3e1740b3928884 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1a45f8716ccc0398cf9395d91d66199d16882aae 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 897783cb756de548a8b634876f729b63ffe9925e 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> 3d0ff1e2dbd6a5c3587cffa283db70415af83a7b 
>   core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
> 
> Diff: https://reviews.apache.org/r/23568/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dong Lin
> 
>

Re: Issues getting IntelliJ set up

[Timothy Chen]

Hi Johnathan,

Did you update your scala version before you run gradle idea?

Also try cleaning up all the artifacts and try it again, as perhaps
your intellij is not picking up the right version and from the right
build folder.

Tim

On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins  wrote:
> Hi,
>
> I've been having some issues getting IntelliJ set up...I followed all the
> instructions on the wiki, and I've successfully imported the project, and
> run the jar Gradle target successfully. However, when I try to run a test
> in the IDE, I get a number of errors:
>
> /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
> Error:(21, 30) java: package kafka.javaapi.producer does not exist
> Error:(22, 22) java: package kafka.producer does not exist
> Error:(23, 22) java: package kafka.producer does not exist
> Error:(24, 19) java: cannot find symbol
>   symbol:   class Utils
>   location: package kafka.utils
> Error:(303, 39) java: cannot find symbol
>   symbol:   class KeyedMessage
>   location: class kafka.tools.KafkaMigrationTool.MigrationThread
>
> And so on.
>
> The two classes that seem to be causing trouble are KafkaMigrationTool and
> ConsumerConnector. Has anybody run into this? Anyone know how to get around
> this issue?
>
> Thanks a lot,
> Natty

Re: Review Request 23702: Patch for KAFKA-1070

[Timothy Chen]

> On July 19, 2014, 11:28 p.m., Timothy Chen wrote:
> > core/src/main/scala/kafka/server/KafkaConfig.scala, line 64
> > <https://reviews.apache.org/r/23702/diff/2/?file=636339#file636339line64>
> >
> > Why reduce the range to 1000? 
> >
> 
> Sriharsha Chintalapani wrote:
> As we go with auto-generating broker id we want to have a backward 
> compatibility where a user can continue to provide brokerId as part of 
> server.properties we will give preference to the user provided brokerId if it 
> presents instead a of generating a new seqId from zookeeper. 
> 1000 here is more of a reserved number for user provided config and to 
> avoid conflict with zookeeper generated seqId. For more details
> please check comments by Jay Kreps here 
> https://issues.apache.org/jira/browse/KAFKA-1070. Thanks.

I see, a comment and a constant will be helpful for what it is.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23702/#review48184
---


On July 19, 2014, 11:06 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23702/
> ---
> 
> (Updated July 19, 2014, 11:06 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1070
> https://issues.apache.org/jira/browse/KAFKA-1070
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1070. Auto assign node id.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/common/InconsistentBrokerIdException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 50b09edb73af1b45f88f919ac8c46ae056878c8e 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> def1dc2a5818d45d9ee0881137ff989cec4eb9b1 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
>   core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 3faa884f8eb83c7c00baab416d0acfb488dc39c1 
> 
> Diff: https://reviews.apache.org/r/23702/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

Re: Review Request 23702: Patch for KAFKA-1070

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23702/#review48184
---



core/src/main/scala/kafka/server/KafkaConfig.scala
<https://reviews.apache.org/r/23702/#comment84538>

Why reduce the range to 1000? 




core/src/main/scala/kafka/server/KafkaServer.scala
<https://reviews.apache.org/r/23702/#comment84539>

You can just use an Option


- Timothy Chen


On July 19, 2014, 11:06 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23702/
> ---
> 
> (Updated July 19, 2014, 11:06 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1070
> https://issues.apache.org/jira/browse/KAFKA-1070
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1070. Auto assign node id.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/common/InconsistentBrokerIdException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 50b09edb73af1b45f88f919ac8c46ae056878c8e 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> def1dc2a5818d45d9ee0881137ff989cec4eb9b1 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
>   core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 3faa884f8eb83c7c00baab416d0acfb488dc39c1 
> 
> Diff: https://reviews.apache.org/r/23702/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

Re: Improving the Kafka client ecosystem

[Timothy Chen]

The certified client test suite really will benefit all the client developers, 
as writing a Kafka client often is not just talking protocol but to be able to 
handle correctly all the cases, errors and situations, but also performance.

From my experience writing a C# client definitely feel that a lot of test 
scenarios could be generalized and used for all clients.

I was reviewing some other client implementation and there are errors and cases 
it didn't handle and having a suite that exposes that will allow users to not 
run knot those problems and try to determine its a client or server bug as it's 
sometimes hard to figure out.

Tim

> On Jul 18, 2014, at 3:57 PM, Jay Kreps  wrote:
> 
> Basically my thought with getting a separate mailing list was to have
> a place specifically to discuss issues around clients. I don't see a
> lot of discussion about them on the main list. I thought perhaps this
> was because people don't like to ask questions which are about
> adjacent projects/code bases. But basically whatever will lead to a
> robust discussion, bug tracking, etc on clients.
> 
> -Jay
> 
>> On Fri, Jul 18, 2014 at 3:49 PM, Jun Rao  wrote:
>> Another important part of eco-system could be around the adaptors of
>> getting data from other systems into Kafka and vice versa. So, for the
>> ingestion part, this can include things like getting data from mysql,
>> syslog, apache server log, etc. For the egress part, this can include
>> putting Kafka data into HDFS, S3, etc.
>> 
>> Will a separate mailing list be convenient? Could we just use the Kafka
>> mailing list?
>> 
>> Thanks,
>> 
>> Jun
>> 
>> 
>>> On Fri, Jul 18, 2014 at 2:34 PM, Jay Kreps  wrote:
>>> 
>>> A question was asked in another thread about what was an effective way
>>> to contribute to the Kafka project for people who weren't very
>>> enthusiastic about writing Java/Scala code.
>>> 
>>> I wanted to kind of advocate for an area I think is really important
>>> and not as good as it could be--the client ecosystem. I think our goal
>>> is to make Kafka effective as a general purpose, centralized, data
>>> subscription system. This vision only really works if all your
>>> applications, are able to integrate easily, whatever language they are
>>> in.
>>> 
>>> We have a number of pretty good non-java producers. We have been
>>> lacking the features on the server-side to make writing non-java
>>> consumers easy. We are fixing that right now as part of the consumer
>>> work going on right now (which moves a lot of the functionality in the
>>> java consumer to the server side).
>>> 
>>> But apart from this I think there may be a lot more we can do to make
>>> the client ecosystem better.
>>> 
>>> Here are some concrete ideas. If anyone has additional ideas please
>>> reply to this thread and share them. If you are interested in picking
>>> any of these up, please do.
>>> 
>>> 1. The most obvious way to improve the ecosystem is to help work on
>>> clients. This doesn't necessarily mean writing new clients, since in
>>> many cases we already have a client in a given language. I think any
>>> way we can incentivize fewer, better clients rather than many
>>> half-working clients we should do. However we are working now on the
>>> server-side consumer co-ordination so it should now be possible to
>>> write much simpler consumers.
>>> 
>>> 2. It would be great if someone put together a mailing list just for
>>> client developers to share tips, tricks, problems, and so on. We can
>>> make sure all the main contributors on this too. I think this could be
>>> a forum for kind of directing improvements in this area.
>>> 
>>> 3. Help improve the documentation on how to implement a client. We
>>> have tried to make the protocol spec not just a dry document but also
>>> have it share best practices, rationale, and intentions. I think this
>>> could potentially be even better as there is really a range of options
>>> from a very simple quick implementation to a more complex highly
>>> optimized version. It would be good to really document some of the
>>> options and tradeoffs.
>>> 
>>> 4. Come up with a standard way of documenting the features of clients.
>>> In an ideal world it would be possible to get the same information
>>> (author, language, feature set, download link, source code, etc) for
>>> all clients. It would be great to standardize the documentation for
>>> the client as well. For example having one or two basic examples that
>>> are repeated for every client in a standardized way. This would let
>>> someone come to the Kafka site who is not a java developer, and click
>>> on the link for their language and view examples of interacting with
>>> Kafka in the language they know using the client they would eventually
>>> use.
>>> 
>>> 5. Build a Kafka Client Compatibility Kit (KCCK) :-) The idea is this:
>>> anyone who wants to implement a client would implement a simple
>>> command line program with a set of standardized options. The
>>> compatibil

Re: Review Request 21272: Patch for KAFKA-1443

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21272/#review44851
---


I removed the DeleteTopicCommand from my latest patch now.

- Timothy Chen


On June 5, 2014, 3:51 a.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21272/
> ---
> 
> (Updated June 5, 2014, 3:51 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1443
> https://issues.apache.org/jira/browse/KAFKA-1443
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1443 Add delete topic option to topic commands
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 
> 804b331a7e4418219bc21ec920b98a0c3a21fac9 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> bdc72eaa9052b6f46bac510ab53c57d3283cdc25 
> 
> Diff: https://reviews.apache.org/r/21272/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Updated] (KAFKA-1443) Add delete topic to topic commands and update DeleteTopicCommand

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1443:


Attachment: KAFKA-1443_2014-06-04_20:51:44.patch

> Add delete topic to topic commands and update DeleteTopicCommand
> 
>
> Key: KAFKA-1443
> URL: https://issues.apache.org/jira/browse/KAFKA-1443
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
> Attachments: KAFKA-1443.patch, KAFKA-1443.patch, 
> KAFKA-1443_2014-06-04_20:51:44.patch
>
>
> Add delete topic option to current topic commands



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1443) Add delete topic to topic commands and update DeleteTopicCommand

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018454#comment-14018454
 ] 

Timothy Chen commented on KAFKA-1443:
-

Updated reviewboard https://reviews.apache.org/r/21272/diff/
 against branch origin/trunk

> Add delete topic to topic commands and update DeleteTopicCommand
> 
>
> Key: KAFKA-1443
> URL: https://issues.apache.org/jira/browse/KAFKA-1443
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
> Attachments: KAFKA-1443.patch, KAFKA-1443.patch, 
> KAFKA-1443_2014-06-04_20:51:44.patch
>
>
> Add delete topic option to current topic commands



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21272: Patch for KAFKA-1443

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21272/
---

(Updated June 5, 2014, 3:51 a.m.)


Review request for kafka.


Bugs: KAFKA-1443
https://issues.apache.org/jira/browse/KAFKA-1443


Repository: kafka


Description (updated)
---

KAFKA-1443 Add delete topic option to topic commands


Diffs (updated)
-

  core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 
804b331a7e4418219bc21ec920b98a0c3a21fac9 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
bdc72eaa9052b6f46bac510ab53c57d3283cdc25 

Diff: https://reviews.apache.org/r/21272/diff/


Testing
---


Thanks,

Timothy Chen

Re: deleting topics

[Timothy Chen]

On the same note, perhaps someone can review my patch for the delete
topic command?

https://reviews.apache.org/r/21272/

Thanks!

Tim

On Wed, Jun 4, 2014 at 9:54 AM, Guozhang Wang  wrote:
> The delete topic utility should be workable in the next release 0.8.2.
>
> Guozhang
>
>
> On Wed, Jun 4, 2014 at 1:29 AM, Piotr Jagielski  wrote:
>
>> Hi all,
>>
>> Can anyone explain the current state of deleting topics? We've
>> found delete.topic.enable config property and after setting it to true on
>> 0.8.1.1 deletion seems to work ok.
>>
>> There is also a fix of https://issues.apache.org/jira/browse/KAFKA-1397 in
>> the trunk is it possible that 0.8.2 could have delete topics enabled by
>> default?
>>
>> Regards
>>
>
>
>
> --
> -- Guozhang

[jira] [Commented] (KAFKA-1443) Add delete topic to topic commands and update DeleteTopicCommand

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993760#comment-13993760
 ] 

Timothy Chen commented on KAFKA-1443:
-

Created reviewboard https://reviews.apache.org/r/21272/
 against branch origin/trunk

> Add delete topic to topic commands and update DeleteTopicCommand
> 
>
> Key: KAFKA-1443
> URL: https://issues.apache.org/jira/browse/KAFKA-1443
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
> Attachments: KAFKA-1443.patch, KAFKA-1443.patch
>
>
> Add delete topic option to current topic commands



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 21272: Patch for KAFKA-1443

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21272/
---

Review request for kafka.


Bugs: KAFKA-1443
https://issues.apache.org/jira/browse/KAFKA-1443


Repository: kafka


Description
---

Add delete topic option to topic commands


Diffs
-

  core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 
804b331a7e4418219bc21ec920b98a0c3a21fac9 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
bdc72eaa9052b6f46bac510ab53c57d3283cdc25 

Diff: https://reviews.apache.org/r/21272/diff/


Testing
---


Thanks,

Timothy Chen

Re: Failed to build Kafka using IntelliJ

[Timothy Chen]

It seems like that you generated your idea files you were on a
different scala version than what you configured it to be.

Can you try to set your scala version to 2.9.2 in your
gradle.properties, and do a ./gradlew clean idea, and try again?


Tim

On Tue, May 13, 2014 at 11:26 PM, Sheng Wang  wrote:
> Hi,
>
> I am trying to use IntelliJ to build Kafka.  I use java 1.7.0_55-b13 on Mac
> OS X Mavericks. The "gradlew jar" commands etc. worked fine.  However, I
> cannot use IntelliJ to build the project. I got some errors when I used
> IntelliJ 13.1 to build the project. Could anyone tell how I can solve this
> problem?
>
> Thanks!
>
> Information:Compilation completed with 38 errors and 7 warnings in 22 sec
> Error:scalac: Error:
> org.jetbrains.jps.incremental.scala.remote.ServerException
> Error compiling sbt component 'compiler-interface-2.8.0.final-51.0'
> at
> sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:145)
> at
> sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:142)
> at sbt.IO$.withTemporaryDirectory(IO.scala:285)
> at
> sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:142)
> at
> sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:139)
> at sbt.IO$.withTemporaryDirectory(IO.scala:285)
> at
> sbt.compiler.AnalyzingCompiler$.compileSources(AnalyzingCompiler.scala:139)
> at sbt.compiler.IC$.compileInterfaceJar(IncrementalCompiler.scala:33)
> at
> org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl$.org$jetbrains$jps$incremental$scala$local$CompilerFactoryImpl$$getOrCompileInterfaceJar(CompilerFactoryImpl.scala:86)
> at
> org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl$$anonfun$getScalac$1.apply(CompilerFactoryImpl.scala:43)
> at
> org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl$$anonfun$getScalac$1.apply(CompilerFactoryImpl.scala:42)
> at scala.Option.map(Option.scala:145)
> at
> org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl.getScalac(CompilerFactoryImpl.scala:42)
> at
> org.jetbrains.jps.incremental.scala.local.CompilerFactoryImpl.createCompiler(CompilerFactoryImpl.scala:21)
> at
> org.jetbrains.jps.incremental.scala.local.CachingFactory$$anonfun$createCompiler$1.apply(CachingFactory.scala:23)
> at
> org.jetbrains.jps.incremental.scala.local.CachingFactory$$anonfun$createCompiler$1.apply(CachingFactory.scala:23)
> at
> org.jetbrains.jps.incremental.scala.local.Cache$$anonfun$getOrUpdate$2.apply(Cache.scala:20)
> at scala.Option.getOrElse(Option.scala:120)
> at
> org.jetbrains.jps.incremental.scala.local.Cache.getOrUpdate(Cache.scala:19)
> at
> org.jetbrains.jps.incremental.scala.local.CachingFactory.createCompiler(CachingFactory.scala:22)
> at
> org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:21)
> at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.scala:64)
> at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:22)
> at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)
> Warning:scalac:
> /var/folders/k2/_ksrnkxx6p502x6sb3v4jsvhgp/T/sbt_8d7d63cd/CompilerInterface.scala:161:
> error: object creation impossible, since method registerTopLevelSym in
> trait GlobalCompat of type (sym: this.Symbol)Unit is not defined
> Warning:scalac: new Compiler() with RangePositions // unnecessary in 2.11
> Warning:scalac: ^
> Warning:scalac:
> /var/folders/k2/_ksrnkxx6p502x6sb3v4jsvhgp/T/sbt_8d7d63cd/CompilerInterface.scala:165:
> error: class Compiler needs to be abstract, since method
> registerTopLevelSym in trait GlobalCompat of type (sym:
> Compiler.this.Symbol)Unit is not defined
> Warning:scalac: class Compiler extends CallbackGlobal(command.settings,
> dreporter, output)
> Warning:scalac:   ^
> Warning:scalac: two errors found
> /Users/aaa/git/kafka2/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
> Error:(21, 30) java: package kafka.javaapi.producer does not exist
> Error:(22, 22) java: package kafka.producer does not exist
> Error:(23, 22) java: package kafka.producer does not exist
> Error:(24, 19) java: cannot find symbol
>   symbol:   class Utils
>   location: package kafka.utils
> Error:(303, 39) java: cannot find symbol
>   symbol:   class KeyedMessage
>   location: class kafka.tools.KafkaMigrationTool.MigrationThread
> Error:(310, 57) java: cannot find symbol
>   symbol:   class KeyedMessage
>   location: class kafka.tools.KafkaMigrationTool.MigrationThread
> Error:(366, 39

Re: Review Request 21398: Fix KAFKA-1445 v2

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21398/#review42982
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
<https://reviews.apache.org/r/21398/#comment76966>

Got it!


- Timothy Chen


On May 13, 2014, 6:25 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21398/
> ---
> 
> (Updated May 13, 2014, 6:25 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1445
> https://issues.apache.org/jira/browse/KAFKA-1445
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 0. Add the partitionsForNode index in Cluster;\n 1. Ready would return a list 
> of ready nodes instead of partitions;\n 2. Ready would also check if there is 
> any ready partitions with unknown leader, if yes indicate the 
> processReadyNode to force metadata refresh;\n 3. Drain would take a list of 
> nodes and drain the batches per node until the max request size is reached;\n 
> 4. Collocate would not be just tranform batches per node into a producer 
> request;\n 5. Corresponding unit test changes; \n 6. One minor compilation 
> warning fix
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
>  fbb732a57522109ac0e0eaf5c87b50cbd3a7f767 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  2d7e52d430fa267ee3689a06f8a621ce5dfd1e33 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> f0152fabbdd44e9f1a24291e84c17edf8379f4fc 
>   clients/src/main/java/org/apache/kafka/common/Cluster.java 
> 426bd1eec708979149cbd6fa3959e6f9e73c7e0e 
>   clients/src/main/java/org/apache/kafka/common/Node.java 
> 0e47ff3ff0e055823ec5a5aa4839d25b0fac8374 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  f37ab770b1794830154f9908a0156e7e99b4a458 
>   
> clients/src/test/java/org/apache/kafka/common/utils/AbstractIteratorTest.java 
> 1df226606fad29da47d81d0b8ff36209c3536c06 
> 
> Diff: https://reviews.apache.org/r/21398/diff/
> 
> 
> Testing
> ---
> 
> unit tests
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: How can I step through the Kafka code using a debugger

[Timothy Chen]

Hi Sheng Wang,

If you look at the Kafka website there is already a IDE setup page you
can follow.

Once your IDE is setup you can simply run Kafka via the Kafka main
method and set breakpoints in the broker.

Tim

On Tue, May 13, 2014 at 11:44 PM, Sheng Wang  wrote:
> Hi,
>
> I want to learn more about the Kafka code base. One of the easiest ways
> that I can think of is to walk through the code with a debugger. Could
> anyone tell how I can do that? Can I do it using any IDE?
>
> Thanks!
>
> -Sheng

Re: Review Request 21398: Fix KAFKA-1445 v2

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21398/#review42869
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
<https://reviews.apache.org/r/21398/#comment76775>

Should we only call forceUpdate once if we get multiple unknown nodes?


- Timothy Chen


On May 13, 2014, 6:25 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21398/
> ---
> 
> (Updated May 13, 2014, 6:25 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1445
> https://issues.apache.org/jira/browse/KAFKA-1445
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 0. Add the partitionsForNode index in Cluster;\n 1. Ready would return a list 
> of ready nodes instead of partitions;\n 2. Ready would also check if there is 
> any ready partitions with unknown leader, if yes indicate the 
> processReadyNode to force metadata refresh;\n 3. Drain would take a list of 
> nodes and drain the batches per node until the max request size is reached;\n 
> 4. Collocate would not be just tranform batches per node into a producer 
> request;\n 5. Corresponding unit test changes; \n 6. One minor compilation 
> warning fix
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
>  fbb732a57522109ac0e0eaf5c87b50cbd3a7f767 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  2d7e52d430fa267ee3689a06f8a621ce5dfd1e33 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> f0152fabbdd44e9f1a24291e84c17edf8379f4fc 
>   clients/src/main/java/org/apache/kafka/common/Cluster.java 
> 426bd1eec708979149cbd6fa3959e6f9e73c7e0e 
>   clients/src/main/java/org/apache/kafka/common/Node.java 
> 0e47ff3ff0e055823ec5a5aa4839d25b0fac8374 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  f37ab770b1794830154f9908a0156e7e99b4a458 
>   
> clients/src/test/java/org/apache/kafka/common/utils/AbstractIteratorTest.java 
> 1df226606fad29da47d81d0b8ff36209c3536c06 
> 
> Diff: https://reviews.apache.org/r/21398/diff/
> 
> 
> Testing
> ---
> 
> unit tests
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Updated] (KAFKA-1443) Add delete topic to topic commands and update DeleteTopicCommand

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1443:


Attachment: KAFKA-1443.patch

> Add delete topic to topic commands and update DeleteTopicCommand
> 
>
> Key: KAFKA-1443
> URL: https://issues.apache.org/jira/browse/KAFKA-1443
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
> Attachments: KAFKA-1443.patch, KAFKA-1443.patch
>
>
> Add delete topic option to current topic commands



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13991080#comment-13991080
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
> KAFKA-1384_2014-05-05_17:14:57.patch, KAFKA-1384_2014-05-06_13:21:14.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 6, 2014, 8:21 p.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
401bf1ea353d30353f9bae4a396302b15aa059c6 
  core/src/main/scala/kafka/log/LogManager.scala 
ab72cffd439c867051247e955dde2b6ade6836b8 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
be1a1ee3ac5c1dd8d658deabcfc740e88e053288 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
a78f7cfa025b0c5d968381b6d5dc7aec894a612b 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
41ebc7acae2efdd8a58964aa746c90fa71ac9ba4 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-06_13:21:14.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
> KAFKA-1384_2014-05-05_17:14:57.patch, KAFKA-1384_2014-05-06_13:21:14.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-05_17:14:57.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
> KAFKA-1384_2014-05-05_17:14:57.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 6, 2014, 12:15 a.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
be1a1ee3ac5c1dd8d658deabcfc740e88e053288 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
a78f7cfa025b0c5d968381b6d5dc7aec894a612b 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
41ebc7acae2efdd8a58964aa746c90fa71ac9ba4 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990123#comment-13990123
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch, 
> KAFKA-1384_2014-05-05_17:14:57.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989994#comment-13989994
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-05_14:25:25.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch, KAFKA-1384_2014-05-05_14:25:25.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

> On May 5, 2014, 8:51 p.m., Jun Rao wrote:
> > Compilation error when trying to run the unit tests.
> > 
> > kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:35:
> >  not enough arguments for constructor LogManager: (logDirs: 
> > Array[java.io.File],topicConfigs: 
> > scala.collection.Map[String,kafka.log.LogConfig],defaultConfig: 
> > kafka.log.LogConfig,cleanerConfig: kafka.log.CleanerConfig,flushCheckMs: 
> > Long,flushCheckpointMs: Long,retentionCheckMs: Long,scheduler: 
> > kafka.utils.Scheduler,brokerState: kafka.server.BrokerState,time: 
> > kafka.utils.Time)kafka.log.LogManager.
> > Unspecified value parameter brokerState.
> >   val logManagers = configs.map(config => new LogManager(logDirs = 
> > config.logDirs.map(new File(_)).toArray,
> >   ^
> > /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:47:
> >  value logDirs is not a member of Nothing
> > for(manager <- logManagers; dir <- manager.logDirs)
> >^
> > /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:67:
> >  value createLog is not a member of Nothing
> > val log0 = logManagers(0).createLog(TopicAndPartition(topic, 0), 
> > LogConfig())
> >   ^
> > /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:106:
> >  value createLog is not a member of Nothing
> > val topic1Log0 = logManagers(0).createLog(TopicAndPartition(topic1, 0), 
> > LogConfig())
> > ^
> > /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:122:
> >  value createLog is not a member of Nothing
> > val topic2Log0 = logManagers(0).createLog(TopicAndPartition(topic2, 0), 
> > LogConfig())
> > ^
> > /Users/jrao/Intellij/kafka_gradle/core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:64:
> >  not enough arguments for constructor LogManager: (logDirs: 
> > Array[java.io.File],topicConfigs: 
> > scala.collection.Map[String,kafka.log.LogConfig],defaultConfig: 
> > kafka.log.LogConfig,cleanerConfig: kafka.log.CleanerConfig,flushCheckMs: 
> > Long,flushCheckpointMs: Long,retentionCheckMs: Long,scheduler: 
> > kafka.utils.Scheduler,brokerState: kafka.server.BrokerState,time: 
> > kafka.utils.Time)kafka.log.LogManager.
> > Unspecified value parameter brokerState.
> > return new LogManager(logDirs,
> >

Ah sorry, oddly I was able to run it but didn't try out all the tests. Fixed it 
in the next patch.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review42197
---


On May 5, 2014, 9:25 p.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20718/
> ---
> 
> (Updated May 5, 2014, 9:25 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1384
> https://issues.apache.org/jira/browse/KAFKA-1384
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1384: Logging kafka state metric
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/log/Log.scala 
> b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c208f83bed7fb91f07fae42f2b66892e6d46fecc 
>   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
> acda52b801714bcc182edc0ced925f0e4b493fc1 
>   core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
> be1a1ee3ac5c1dd8d658deabcfc740e88e053288 
>   core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
> a78f7cfa025b0c5d968381b6d5dc7aec894a612b 
>   core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
> 41ebc7acae2efdd8a58964aa746c90fa71ac9ba4 
> 
> Diff: https://reviews.apache.org/r/20718/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 5, 2014, 9:25 p.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
be1a1ee3ac5c1dd8d658deabcfc740e88e053288 
  core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala 
a78f7cfa025b0c5d968381b6d5dc7aec894a612b 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
41ebc7acae2efdd8a58964aa746c90fa71ac9ba4 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-05_14:00:29.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
> KAFKA-1397_2014-05-02_13:38:02.patch, KAFKA-1397_2014-05-05_11:17:59.patch, 
> KAFKA-1397_2014-05-05_14:00:29.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989968#comment-13989968
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
> KAFKA-1397_2014-05-02_13:38:02.patch, KAFKA-1397_2014-05-05_11:17:59.patch, 
> KAFKA-1397_2014-05-05_14:00:29.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 5, 2014, 9 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
014e9644dba1d65142f6a9abc745858870a46230 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
00bfba41fe008fe72c17fb788e3cada3daf8eb30 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review42199
---



core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala
<https://reviews.apache.org/r/20745/#comment75941>

Makes sense!


- Timothy Chen


On May 5, 2014, 9 p.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20745/
> ---
> 
> (Updated May 5, 2014, 9 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1397
> https://issues.apache.org/jira/browse/KAFKA-1397
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1397: Fix delete topic tests and deadlock
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 36ddeb44490e8343a4e8056c45726ac660e4b2f9 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 0e47dac8cbf65a86d053a3371a18af467afd70ae 
>   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
> e4bc2439ce1933c7c7571d255464ee678226a6cb 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 11c20cee83fda9a492156674d351a0096b13fd99 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 9c29e144bba2c9bafa91941b6ca5c263490693b3 
>   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
> 5c487968014b56490eb2bc876cef1c52efd1cdad 
>   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
> 014e9644dba1d65142f6a9abc745858870a46230 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 00bfba41fe008fe72c17fb788e3cada3daf8eb30 
> 
> Diff: https://reviews.apache.org/r/20745/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-05_11:17:59.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
> KAFKA-1397_2014-05-02_13:38:02.patch, KAFKA-1397_2014-05-05_11:17:59.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989793#comment-13989793
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
> KAFKA-1397_2014-05-02_13:38:02.patch, KAFKA-1397_2014-05-05_11:17:59.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 5, 2014, 6:17 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
014e9644dba1d65142f6a9abc745858870a46230 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
00bfba41fe008fe72c17fb788e3cada3daf8eb30 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

> On May 5, 2014, 5:12 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/utils/ShutdownableThread.scala, lines 28-33
> > <https://reviews.apache.org/r/20745/diff/6/?file=573313#file573313line28>
> >
> > I thought you plan to remove isShuttingDown?

Odd I don't have this anymore in my box, but end up in my patch


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review42058
-------


On May 2, 2014, 8:38 p.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20745/
> ---
> 
> (Updated May 2, 2014, 8:38 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1397
> https://issues.apache.org/jira/browse/KAFKA-1397
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1397: Fix delete topic tests and deadlock
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 36ddeb44490e8343a4e8056c45726ac660e4b2f9 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 0e47dac8cbf65a86d053a3371a18af467afd70ae 
>   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
> e4bc2439ce1933c7c7571d255464ee678226a6cb 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 11c20cee83fda9a492156674d351a0096b13fd99 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 9c29e144bba2c9bafa91941b6ca5c263490693b3 
>   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
> 5c487968014b56490eb2bc876cef1c52efd1cdad 
>   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
> 014e9644dba1d65142f6a9abc745858870a46230 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 00bfba41fe008fe72c17fb788e3cada3daf8eb30 
> 
> Diff: https://reviews.apache.org/r/20745/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-05_11:05:15.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989785#comment-13989785
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch, 
> KAFKA-1384_2014-05-05_11:05:15.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 5, 2014, 6:05 p.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988206#comment-13988206
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
> KAFKA-1397_2014-05-02_13:38:02.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-02_13:38:02.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch, 
> KAFKA-1397_2014-05-02_13:38:02.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 2, 2014, 8:38 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
014e9644dba1d65142f6a9abc745858870a46230 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
00bfba41fe008fe72c17fb788e3cada3daf8eb30 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

> On May 2, 2014, 12:26 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/utils/ShutdownableThread.scala, lines 35-36
> > <https://reviews.apache.org/r/20745/diff/5/?file=573213#file573213line35>
> >
> > We probably should check the return value of initiateShutdown().

I was thinking that even with concurrent calls since all calls end up just 
waiting on the shutdownLatch it should be safe to not check it.
Therefore shutdown call regardless of what state will end up waiting until the 
thread shutdown is completed.


> On May 2, 2014, 12:26 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/utils/ShutdownableThread.scala, lines 40-41
> > <https://reviews.apache.org/r/20745/diff/5/?file=573213#file573213line40>
> >
> > Instead of introducing isShuttingDown, could we just to compareAndSet 
> > on isRunning?

Good point, consolidated it.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41979
-------


On May 2, 2014, 1:12 a.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20745/
> ---
> 
> (Updated May 2, 2014, 1:12 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1397
> https://issues.apache.org/jira/browse/KAFKA-1397
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1397: Fix delete topic tests and deadlock
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 36ddeb44490e8343a4e8056c45726ac660e4b2f9 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 0e47dac8cbf65a86d053a3371a18af467afd70ae 
>   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
> e4bc2439ce1933c7c7571d255464ee678226a6cb 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 11c20cee83fda9a492156674d351a0096b13fd99 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 9c29e144bba2c9bafa91941b6ca5c263490693b3 
>   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
> 5c487968014b56490eb2bc876cef1c52efd1cdad 
>   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
> 014e9644dba1d65142f6a9abc745858870a46230 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 49c7790c995bb2e79322eb148ab80d0dcccefed4 
> 
> Diff: https://reviews.apache.org/r/20745/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988111#comment-13988111
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 2, 2014, 7:13 p.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-02_12:13:24.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch, KAFKA-1384_2014-05-02_12:13:24.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987259#comment-13987259
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-05-01_18:47:03.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch, 
> KAFKA-1384_2014-05-01_18:47:03.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated May 2, 2014, 1:47 a.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
b7bc5ffdecba7221b3d2e4bca2b0c703bc74fa12 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

> On May 2, 2014, 1:09 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/KafkaServer.scala, lines 108-109
> > <https://reviews.apache.org/r/20718/diff/2/?file=569132#file569132line108>
> >
> > Will that override the state to broker even though it's set to 
> > controller?

The controller only sets the state when it is elected, and from the server 
startup steps it will set as broker state first, and once the controller starts 
up and the elect happens then it will be set to controller state.


> On May 2, 2014, 1:09 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/KafkaServerStartable.scala, lines 55-58
> > <https://reviews.apache.org/r/20718/diff/2/?file=569133#file569133line55>
> >
> > What's the intention of this method?

This is to allow custom state that is not defined in the enums. One example is 
in our custom kafka server startable we can inject more states that is specific 
to different use cases.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review41993
-------


On April 26, 2014, 7:09 a.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20718/
> ---
> 
> (Updated April 26, 2014, 7:09 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1384
> https://issues.apache.org/jira/browse/KAFKA-1384
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1384: Logging kafka state metric
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/log/Log.scala 
> f20c232e2daa63a10d91b965af52801af656477c 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c208f83bed7fb91f07fae42f2b66892e6d46fecc 
>   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
> acda52b801714bcc182edc0ced925f0e4b493fc1 
> 
> Diff: https://reviews.apache.org/r/20718/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987239#comment-13987239
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-01_18:12:24.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch, KAFKA-1397_2014-05-01_18:12:24.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 2, 2014, 1:12 a.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
014e9644dba1d65142f6a9abc745858870a46230 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
49c7790c995bb2e79322eb148ab80d0dcccefed4 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987127#comment-13987127
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-05-01_15:53:57.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch, 
> KAFKA-1397_2014-05-01_15:53:57.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated May 1, 2014, 10:54 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
36ddeb44490e8343a4e8056c45726ac660e4b2f9 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
cf8adc9f468f4d6f01d1303efe39a3ec6f3d9b53 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
49c7790c995bb2e79322eb148ab80d0dcccefed4 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

> On May 1, 2014, 6:26 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/controller/TopicDeletionManager.scala, lines 
> > 394-396
> > <https://reviews.apache.org/r/20745/diff/4/?file=571881#file571881line394>
> >
> > It seems this test is not necessary since the outer loop will detect 
> > that too.

It's actually necessary since isRunning might been set to false while it woke 
up from the condition, in that case we don't want it to still try to acquire 
the controller lock but to bail early.


> On May 1, 2014, 6:26 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/server/ReplicaManager.scala, lines 128-136
> > <https://reviews.apache.org/r/20745/diff/4/?file=571885#file571885line128>
> >
> > Do we still need to do this now that we make sure that if a replica is 
> > deleted, the controller will never send a LeaderAndIsrRequest to this 
> > replica until the topic is deleted?

We need this as described in the comment that if a broker that was shutdown 
during the topic deletion and brought back up, the 
stopReplicaRequest(deletePartition=true) without this extra log manager cleanup 
will never delete the log and the folders associated with as ReplicaManager 
doesn't hold that information anymore as we don't send the leader and isr 
request again as part of the retry.


> On May 1, 2014, 6:26 p.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala, lines 86-94
> > <https://reviews.apache.org/r/20745/diff/4/?file=571886#file571886line86>
> >
> > This is not going to be reliable since we can't be sure whether the 
> > controller has completed the deletion before the failover.

Ok removing the test for now.


> On May 1, 2014, 6:26 p.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala, lines 239-240
> > <https://reviews.apache.org/r/20745/diff/4/?file=571886#file571886line239>
> >
> > This test seems to be the same as 
> > testPartitionReassignmentDuringDeleteTopic().

Just the order of requests, but as well cannot gurantee what is processed first 
as they're just asynchronous. Will remove.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41904
---


On April 30, 2014, 9:55 p.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20745/
> ---
> 
> (Updated April 30, 2014, 9:55 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1397
> https://issues.apache.org/jira/browse/KAFKA-1397
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1397: Fix delete topic tests and deadlock
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 0e47dac8cbf65a86d053a3371a18af467afd70ae 
>   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
> e4bc2439ce1933c7c7571d255464ee678226a6cb 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 0b668f230c8556fdf08654ce522a11847d0bf39b 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c208f83bed7fb91f07fae42f2b66892e6d46fecc 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 11c20cee83fda9a492156674d351a0096b13fd99 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 9c29e144bba2c9bafa91941b6ca5c263490693b3 
>   core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
> 5c487968014b56490eb2bc876cef1c52efd1cdad 
>   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
> b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 384c74e7c3985abff864e61dea5e530dbd189d5d 
> 
> Diff: https://reviews.apache.org/r/20745/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986142#comment-13986142
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-04-30_14:55:28.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch, KAFKA-1397_2014-04-30_14:55:28.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated April 30, 2014, 9:55 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description (updated)
---

KAFKA-1397: Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
11c20cee83fda9a492156674d351a0096b13fd99 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
5c487968014b56490eb2bc876cef1c52efd1cdad 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
384c74e7c3985abff864e61dea5e530dbd189d5d 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

> On April 29, 2014, 5:02 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/controller/ControllerChannelManager.scala, lines 
> > 209-212
> > <https://reviews.apache.org/r/20745/diff/3/?file=569935#file569935line209>
> >
> > Is it better to do the check here or in the caller?

Hmm, at first glance not sure why it makes sense to propagate 
LeaderAndIsrRequest when a partition is being deleted, but moving it just to 
the delete topic path is probably the least impact. I think I'll move it to the 
caller for now.


> On April 29, 2014, 5:02 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/controller/TopicDeletionManager.scala, lines 
> > 385-386
> > <https://reviews.apache.org/r/20745/diff/3/?file=569938#file569938line385>
> >
> > Do we need to check here at all since the caller of doWork() already 
> > does the check?

Good point, since isRunning must be set to false before it can shut down this 
is redudant.


> On April 29, 2014, 5:02 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/controller/TopicDeletionManager.scala, lines 
> > 412-413
> > <https://reviews.apache.org/r/20745/diff/3/?file=569938#file569938line412>
> >
> > Do we need to introduce isAnyReplicaInState? replicasInState() seems 
> > more general and we can just check the output to implement 
> > isAnyReplicaInState.

I added this because the logic here only want to see if there is any replica 
that is in a specific state, and thought returning the whole list is a waste as 
I see internally it's constructing a new set. From profiling other Scala code I 
see that any set constructing is quite expensive as it's using immutable sets. 
It is more clean to only expose one API, but I thought the trade off might be 
worth it. More thoughts?


> On April 29, 2014, 5:02 p.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala, lines 86-91
> > <https://reviews.apache.org/r/20745/diff/3/?file=569942#file569942line86>
> >
> > There seems to be no guarantee that the delete topic process is 
> > completed before the controller was shutdown. So, I am not sure how 
> > reliable the test is.

It's actually the reverse, where it tries to shutdown controller while the 
delete topic process is still in progress. Currently there is no way to 
gurantee that the delete topic can halt until some condition happens, so it 
might be not guranteed.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41714
---


On April 29, 2014, 12:08 a.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20745/
> ---
> 
> (Updated April 29, 2014, 12:08 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1397
> https://issues.apache.org/jira/browse/KAFKA-1397
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Fix delete topic tests and deadlock
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> 919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 0e47dac8cbf65a86d053a3371a18af467afd70ae 
>   core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
> e4bc2439ce1933c7c7571d255464ee678226a6cb 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 0b668f230c8556fdf08654ce522a11847d0bf39b 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c208f83bed7fb91f07fae42f2b66892e6d46fecc 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 9c29e144bba2c9bafa91941b6ca5c263490693b3 
>   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
> b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 
> 
> Diff: https://reviews.apache.org/r/20745/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983759#comment-13983759
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-04-28_17:08:49.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch, 
> KAFKA-1397_2014-04-28_17:08:49.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated April 29, 2014, 12:08 a.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397_2014-04-28_14:48:32.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983619#comment-13983619
 ] 

Timothy Chen commented on KAFKA-1397:
-

Updated reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch, KAFKA-1397_2014-04-28_14:48:32.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

(Updated April 28, 2014, 9:48 p.m.)


Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

Fix delete topic tests and deadlock


Diffs (updated)
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

> On April 28, 2014, 7:43 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/controller/ControllerChannelManager.scala, line 
> > 210
> > <https://reviews.apache.org/r/20745/diff/1/?file=569160#file569160line210>
> >
> > better to not leak the logic from delete topic manager here. It is 
> > worth exposing the check as an API in DeleteTopicManager. In any case, can 
> > you explain the motivation behind your change?

Sorry forgot to add more comments to the rb, have been chatting mostly with Jun 
about my fixes.
One test that was failing in delete topic test was a test that was handling 
request while topic being deleted, and it made sure it can never finish by 
shutting down one of the follower broker. Sending a fetch request to a topic 
being deleted returns a NotLeaderForPartition error code instead of expected 
UnknownTopicOrPartition.
 
The cause of this is that as part of the metadata cache changes in Kafka api, 
we removed the check if a topic exists in the cache when serving fetching 
request and now it tries to read message set from the replica manager, and even 
though a stop replica request with delete partition is already been sent and 
processed, the partition was somehow still in replica manager and it returns 
NotLeaderForPartition exception since there is no leader for this topic.

I found out that the partition was being recreated in the replica manager, 
because in the process of retrying deleting the same topic it change the failed 
replica state into OfflineReplica again that triggers a LeaderAndIsrRequest to 
all brokers for shrinking the Isr for that replica. 

The brokers that received the LeaderAndIsrRequest recreates the partition with 
getOrCreatePartition, and therefore future fetch requests hits it and throws 
that error code.

I think the correct thing to do is to not send LeaderAndIsrRequest if the topic 
is being deleted when bring replica to an offline state, as it is not necessary 
anymore.


> On April 28, 2014, 7:43 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/controller/KafkaController.scala, line 652
> > <https://reviews.apache.org/r/20745/diff/1/?file=569161#file569161line652>
> >
> > did you make this change since onControllerResignation already holds 
> > the same lock? 
> > 
> > In general, for tricky bugs like deadlock, it is a good idea to explain 
> > all the significant changes, so it is easier to review.

The latest fix for the deadlock decouples the await topic deletion with a new 
lock internal to delete topic manager, so that during the shutdown the await on 
the topic deletion condition and resume since it's not based on the controller 
lock anymore.

We're still seeing occasional deadlock hanging problem, and with investigation 
Jun and I believe it's because if the topic deletion resumes and right before 
it acquires the controller lock, if a shutdown happens and acquired the lock, 
the delete topic thread will be waiting forever as the shutdown also blocks and 
waits.

There are a variety of fixes, but I felt one of the cleaner fix is to shutdown 
the delete topic manager outside of the controller lock as it doesn't really 
require it anymore. Once delete topic thread shuts down then we can proceed 
holding the controller lock and finish the broker shutdown. 


> On April 28, 2014, 7:43 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/controller/TopicDeletionManager.scala, line 403
> > <https://reviews.apache.org/r/20745/diff/1/?file=569163#file569163line403>
> >
> > * delete topic wasn't initiated for the given topic
> > 
> > Nevertheless, it seems like the loop goes over topics that are queued 
> > up for deletion, so is this comment right?

There isn't really actual bug here, but I was trying to fix the comments and 
logic as if a topic was queued to be deleted and it's the first time calling 
delete topic thread, the current code and comment seems to suggest all topics 
that are queued that is not in completed has one failed replica. But it could 
be that it is just queued up to be deleted the first time.

The effect is that in the logs you will see that the current code will log that 
it attempts to retry topic deletion again, while it's just being deleted the 
first time. 

I was trying to say that this is the first time the topic has been queued for 
deletion and not resumed to be completed or retried. I'll try to rewrite it.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/#review41626
---


On April 27, 2014, 6:43 a.m., Timothy Chen wrote:
> 
> ---
> Th

[jira] [Updated] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1397:


Attachment: KAFKA-1397.patch

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1397) delete topic is not working

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13982206#comment-13982206
 ] 

Timothy Chen commented on KAFKA-1397:
-

Created reviewboard https://reviews.apache.org/r/20745/
 against branch origin/trunk

> delete topic is not working 
> 
>
> Key: KAFKA-1397
> URL: https://issues.apache.org/jira/browse/KAFKA-1397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Jun Rao
> Attachments: KAFKA-1397.patch
>
>
> All unit tests are disabled since they hang transiently (see details in 
> KAFKA-1391).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20745: Patch for KAFKA-1397

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20745/
---

Review request for kafka.


Bugs: KAFKA-1397
https://issues.apache.org/jira/browse/KAFKA-1397


Repository: kafka


Description
---

Fix delete topic tests and deadlock


Diffs
-

  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
919aeb26f93d2fc34d873cfb3dbfab7235a9d635 
  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
0e47dac8cbf65a86d053a3371a18af467afd70ae 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
e4bc2439ce1933c7c7571d255464ee678226a6cb 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/KafkaApis.scala 
0b668f230c8556fdf08654ce522a11847d0bf39b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
9c29e144bba2c9bafa91941b6ca5c263490693b3 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
b1c4ce9f66aa86c68f2e6988f67546fa63ed1f9b 

Diff: https://reviews.apache.org/r/20745/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384_2014-04-26_00:09:56.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981909#comment-13981909
 ] 

Timothy Chen commented on KAFKA-1384:
-

Updated reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch, KAFKA-1384_2014-04-26_00:09:56.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

(Updated April 26, 2014, 7:09 a.m.)


Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs (updated)
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
f20c232e2daa63a10d91b965af52801af656477c 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

> On April 25, 2014, 6:35 p.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/server/KafkaServerStartable.scala, line 55
> > <https://reviews.apache.org/r/20718/diff/1/?file=568568#file568568line55>
> >
> > Unused

This is actually exposed setting custom state that is not one of the defined 
enums. One example is for our internal custom kafka server startable override 
can allow more states that is not defined in BrokerStates. The alternative is 
to just come up with all possible enums and let that be the only option 
available, but I was thinking that might be a bit too limiting. I'm not sure if 
this approach is too permissive either, you have more thoughts?


> On April 25, 2014, 6:35 p.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/server/BrokerStates.scala, line 28
> > <https://reviews.apache.org/r/20718/diff/1/?file=568566#file568566line28>
> >
> > Just a thought: I'm wondering if it is better to have a bit-vector 
> > approach for states; although that will limit the number of possible states 
> > - but I think that is fine.
> > 
> > The main reason for this is that it will enable composing simultaneous 
> > states. E.g., in this approach you cannot distinguish state 5 from state 3 
> > (if a shutting down broker is the controller). Although we can probably 
> > infer that from the fact that the other brokers are likely in state (2) or 
> > by looking at the active controller count separately. Still, we currently 
> > allow more than one broker to shut down.
> > 
> > It also _might_ help catch erroneous dual states (due to bugs).
> > 
> > What you have is probably fine for lifecycle states (except for the 
> > above caveat). However, if we ever want to allow more-than-lifecycle states 
> > (e.g., under-replicated is a state we _might_ want to include on this - 
> > even though we have a separate URP mbean and it's not a lifecycle state; 
> > another example is "loading consumer offsets").

Erroneous dual states is something I thought of, especially when state changes 
rapidly it will be harder to detect.
The down side of having a bit field is also that you need to then also have to 
unset that state when you're going into another lifecycle state.

I think coming back to the high level of what we want for broker states, since 
really a broker should really just be in a state at one point of time, setting 
a particular state I think makes more sense.

Composite states probably makes more sense for different component states. 


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review41501
---


On April 25, 2014, 5:25 p.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20718/
> ---
> 
> (Updated April 25, 2014, 5:25 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1384
> https://issues.apache.org/jira/browse/KAFKA-1384
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1384: Logging kafka state metric
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/log/Log.scala 
> 46df8d99d977a3b010a9b9f4698187fa9bfb2498 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c208f83bed7fb91f07fae42f2b66892e6d46fecc 
>   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
> acda52b801714bcc182edc0ced925f0e4b493fc1 
> 
> Diff: https://reviews.apache.org/r/20718/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981275#comment-13981275
 ] 

Timothy Chen commented on KAFKA-1384:
-

Created reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>    Reporter: Timothy Chen
>    Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981272#comment-13981272
 ] 

Timothy Chen commented on KAFKA-1384:
-

I'm planning just to expose it via JMX, as we don't have a need to write it to 
zookeeper as other brokers doesn't have a need to query it.

Less contention to ZK as well, you have anything in mind why you thought 
putting it in ZK would be good?


> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Timothy Chen
>Assignee: Timothy Chen
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
46df8d99d977a3b010a9b9f4698187fa9bfb2498 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

Re: Review Request 20713: Patch for KAFKA-1424

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20713/#review41473
---

Ship it!


Ship It!

- Timothy Chen


On April 25, 2014, 2:54 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20713/
> ---
> 
> (Updated April 25, 2014, 2:54 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1424
> https://issues.apache.org/jira/browse/KAFKA-1424
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Wait metadata is propagated before fetching.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/producer/ProducerTest.scala 
> dc6a5ea4abcdff370f3b61c774bbde4e4157f412 
> 
> Diff: https://reviews.apache.org/r/20713/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: [VOTE] Apache Kafka 0.8.1.1 Release Candidate 1

[Timothy Chen]

+1 (Non-binding).

Tim

On Thu, Apr 24, 2014 at 5:32 PM, Joel Koshy  wrote:
> +1
>
> One (non-critical) issue is that I think our packaging isn't complete
> for kafka-perf (we are only building for 2.8.0).
>
> On Tue, Apr 22, 2014 at 07:18:38PM -0400, Joe Stein wrote:
>> This is the first candidate for release of Apache Kafka 0.8.1.1
>>
>> Release Notes for the 0.8.1.1 release
>> https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Friday, April 25th, 4pm PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5, sha1
>> and sha2 (SHA256) checksum.
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/
>>
>> * Maven artifacts to be voted upon prior to release:
>> https://repository.apache.org/content/groups/staging/
>>
>> * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1.1 tag
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=34237371b662f617dd80c75cd6282f97e047ef56
>>
>>
>> /***
>>  Joe Stein
>>  Founder, Principal Consultant
>>  Big Data Open Source Security LLC
>>  http://www.stealth.ly
>>  Twitter: @allthingshadoop 
>> /
>

Re: Review Request 20540: Patch for KAFKA-1410

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20540/#review41067
---

Ship it!


Ship It!

- Timothy Chen


On April 22, 2014, 8:51 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20540/
> ---
> 
> (Updated April 22, 2014, 8:51 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1410
> https://issues.apache.org/jira/browse/KAFKA-1410
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 1. Handle the creation of OffsetTopic properly.
> 
> 
> 1. Move MetadataCache to its own class. 2. Avoid updating metadata cache if 
> the controller epoch is old.
> 
> 
> 1. Remove ensureTopicExists(). 2. Move the update of controller epoc in topic 
> metadata request to ReplicaManager.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 1a4ffcea37e32739b8360956b4a40c0101b97822 
>   core/src/main/scala/kafka/server/MetadataCache.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 5588f59783d5cd8f074263c207045625282af12d 
>   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
> 17b08e14f5e9bfec1233c80d74b50bc2ec38aa25 
> 
> Diff: https://reviews.apache.org/r/20540/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 20540: Patch for KAFKA-1410

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20540/#review41039
---



core/src/main/scala/kafka/server/KafkaApis.scala
<https://reviews.apache.org/r/20540/#comment74401>

Do we no longer need to check for offset management topic?


- Timothy Chen


On April 22, 2014, 4:37 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20540/
> ---
> 
> (Updated April 22, 2014, 4:37 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1410
> https://issues.apache.org/jira/browse/KAFKA-1410
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 1. Move MetadataCache to its own class. 2. Avoid updating metadata cache if 
> the controller epoch is old.
> 
> 
> 1. Remove ensureTopicExists(). 2. Move the update of controller epoc in topic 
> metadata request to ReplicaManager.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 1a4ffcea37e32739b8360956b4a40c0101b97822 
>   core/src/main/scala/kafka/server/MetadataCache.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 5588f59783d5cd8f074263c207045625282af12d 
>   core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
> 17b08e14f5e9bfec1233c80d74b50bc2ec38aa25 
> 
> Diff: https://reviews.apache.org/r/20540/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 20534: Patch for KAFKA-1380

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20534/#review40968
---

Ship it!


Ship It!

- Timothy Chen


On April 21, 2014, 9:47 p.m., Joel Koshy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20534/
> ---
> 
> (Updated April 21, 2014, 9:47 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1380
> https://issues.apache.org/jira/browse/KAFKA-1380
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1356; follow-up - return unknown topic partition on non-existent topic 
> if auto.create is off
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 9e569eb8c3586b2fdb3b5904c5fa5f938cffada1 
> 
> Diff: https://reviews.apache.org/r/20534/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Joel Koshy
> 
>