Re: Request for permission to edit pages in the wiki

[Ismael Juma]

On Thu, May 21, 2015 at 11:51 PM, Joel Koshy  wrote:

> Sorry about that - should work now.
>

Thank you. It does indeed.

Best,
Ismael

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

[Mayuresh Gharat]

Hi Jun,

Thanks a lot. I get it now.
 Point 4) will actually enable clients to who don't want to create a topic
with default partitions, if it does not exist and then can manually create
the topic with their own configs(#partitions).

Thanks,

Mayuresh

On Thu, May 21, 2015 at 6:16 PM, Jun Rao  wrote:

> Mayuresh,
>
> The current plan is the following.
>
> 1. Add TMR v1, which still triggers auto topic creation.
> 2. Change the consumer client to TMR v1. Change the producer client to use
> TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly
> create the topic with the default server side partitions and replicas.
> 3. At some later time after the new clients are released and deployed,
> disable auto topic creation in TMR v1. This will make sure consumers never
> create new topics.
> 4. If needed, we can add a new config in the producer to control whether
> TopicCreateRequest should be issued or not on UnknownTopicException. If
> this is disabled and the topic doesn't exist, send will fail and the user
> is expected to create the topic manually.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi,
> > I had a question about TopicMetadata Request.
> > Currently the way it works is :
> >
> > 1) Suppose a topic T1 does not exist.
> > 2) Client wants to produce data to T1 using producer P1.
> > 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka.
> > This in turn creates the default number of partition. The number of
> > partitions is a cluster wide config.
> > 4) Same goes for a consumer. If the topic does not exist and new topic
> will
> > be created when the consumer issues TopicMetadata request.
> >
> > Here are 2 use cases where it might not be suited :
> >
> > The auto creation flag for topics  is turned  ON.
> >
> > a) Some clients might not want to create topic with default number of
> > partitions but with lower number of partitions. Currently in a
> multi-tenant
> > environment this is not possible without changing the cluster wide
> default
> > config.
> >
> > b) Some clients might want to just check if the topic exist or not but
> > currently the topic gets created automatically using default number of
> > partitions.
> >
> > Here are some ideas to address this :
> >
> > 1) The way this can be  addressed is that TopicMetadata request should
> have
> > a way to specify whether it should only check if the topic exist or check
> > and create a topic with given number of partitions. If the number of
> > partitions is not specified use the default cluster wide config.
> >
> > OR
> >
> > 2) We should only allow TopicMetadata Request to get the metadata
> > explicitly and not allow it to create a new topic. We should have another
> > Request that takes in config parameters from the user regarding how
> he/she
> > wants the topic to be created. This request can be used if we get an
> empty
> > TopicMetadata Response.
> >
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> > On Thu, May 14, 2015 at 10:22 AM, Jun Rao  wrote:
> >
> > > For ListTopics, we decided not to add a ListTopics request for now and
> > just
> > > rely on passing in an empty list to TMR. We can revisit this in the
> > future
> > > if it becomes an issue.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy 
> wrote:
> > >
> > > > Just had a few minor questions before I join the vote thread.
> > > > Apologies if these have been discussed:
> > > >
> > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return
> > > >   InvalidPartitions instead?
> > > > - AdminClient.listTopics: should we allow listing all partitions? Or
> > > >   do you intend for the client to issue listTopics followed by
> > > >   describeTopics?
> > > > - On returning future for partition reassignments: do we need
> to
> > > >   return any future especially since you have the
> > > >   verifyReassignPartitions method? For e.g., what happens if the
> > > >   controller moves? The get should fail right? The client will then
> > > >   need to connect to the new controller and reissue the request but
> > > >   will then get ReassignPartitionsInProgress. So in that case the
> > > >   client any way needs to rely in verifyReassignPartitions.
> > > > - In past hangouts I think either you/Joe were mentioning the need to
> > > >   locate the controller (and possibly other cluster metadata). It
> > > >   appears we decided to defer this for a future KIP. Correct?
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote:
> > > > > Guys,
> > > > >
> > > > > I've updated the wiki to reflect all previously discussed items
> > > > > (regarding the schema only - this is included to phase 1).
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > >
> > > > > I think w

[jira] [Commented] (KAFKA-2201) Open file handle leak

[jaikiran pai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555458#comment-14555458
 ] 

jaikiran pai commented on KAFKA-2201:
-

This might be related 
http://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3c54f40dea.4020...@gmail.com%3E


> Open file handle leak
> -
>
> Key: KAFKA-2201
> URL: https://issues.apache.org/jira/browse/KAFKA-2201
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.1
> Environment: Debian Linux 7, 64 bit
> Oracle JDK 1.7.0u40, 64-bit
>Reporter: Albert Visagie
>
> The kafka broker crashes with the following stack trace from the server.log 
> roughly every 18 hours:
> [2015-05-19 07:39:22,924] FATAL [KafkaApi-0] Halting due to unrecoverable I/O 
> error while handling produce request:  (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: I/O exception in append to log 'nnn-1'
> at kafka.log.Log.append(Log.scala:266)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:379)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:365)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inReadLock(Utils.scala:541)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365)
> at 
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:291)
> at 
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:282)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:282)
> at 
> kafka.server.KafkaApis.handleProducerOrOffsetCommitRequest(KafkaApis.scala:204)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
> at 
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(OffsetIndex.scala:265)
> at 
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:265)
> at 
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:265)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:264)
> at kafka.log.Log.roll(Log.scala:563)
> at kafka.log.Log.maybeRoll(Log.scala:539)
> at kafka.log.Log.append(Log.scala:306)
> ... 21 more
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
> ... 33 more
> The Kafka broker's open filehandles as seen by
> lsof | grep pid | wc -l 
> grows steadily as it runs. Under our load it lasts about 18 hours before 
> crashing with the stack trace above.
> We were experimenting with settings under Log Retention Policy in 
> server.properties:
> log.retention.hours=168
> log.retention.bytes=107374182
> log.segment.bytes=1073741
> log.retention.check.interval.ms=3000
> The result is that the broker rolls over segments quite rapidly. We don't 
> have to run it that way of course.
> We are running only one broker at the moment.
> lsof shows many open files without size and absent from ls in the log 
> directory with the suffix ".deleted"
> This is kafka 0.8.2.1 with scala 2.10.4 as downloaded from the website last 
> week.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34450: Fix KAFKA-2017

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34450/
---

(Updated May 22, 2015, 2:03 a.m.)


Review request for kafka.


Bugs: KAFKA-2017
https://issues.apache.org/jira/browse/KAFKA-2017


Repository: kafka


Description (updated)
---

1. Upon receiving join-group, if the group metadata cannot be found in the 
local cache try to read it from ZK; 
2. Upon completing rebalance, update the ZK with new group registry or delete 
the registry if the group becomes empty.

Address Jun's comments


Diffs
-

  core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 
af06ad45cdc46ac3bc27898ebc1a5bd5b1c7b19e 
  core/src/main/scala/kafka/coordinator/ConsumerGroupMetadata.scala 
47bdfa7cc86fd4e841e2b1d6bfd40f1508e643bd 
  core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala 
c39e6de34ee531c6dfa9107b830752bd7f8fbe59 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
2618dd39b925b979ad6e4c0abd5c6eaafb3db5d5 

Diff: https://reviews.apache.org/r/34450/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2017:
-
Attachment: KAFKA-2017_2015-05-21_19:02:47.patch

> Persist Coordinator State for Coordinator Failover
> --
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch, 
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to 
> a new coordinator without forcing all the consumers rejoin their groups. This 
> is possible if the coordinator persists its state so that the state can be 
> transferred during coordinator failover. This state consists of most of the 
> information in GroupRegistry and ConsumerRegistry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555452#comment-14555452
 ] 

Guozhang Wang commented on KAFKA-2017:
--

Updated reviewboard https://reviews.apache.org/r/34450/diff/
 against branch origin/trunk

> Persist Coordinator State for Coordinator Failover
> --
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch, 
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to 
> a new coordinator without forcing all the consumers rejoin their groups. This 
> is possible if the coordinator persists its state so that the state can be 
> transferred during coordinator failover. This state consists of most of the 
> information in GroupRegistry and ConsumerRegistry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34450: Fix KAFKA-2017

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34450/
---

(Updated May 22, 2015, 2:02 a.m.)


Review request for kafka.


Summary (updated)
-

Fix KAFKA-2017


Bugs: KAFKA-2017
https://issues.apache.org/jira/browse/KAFKA-2017


Repository: kafka


Description (updated)
---

1. Upon receiving join-group, if the group metadata cannot be found in the 
local cache try to read it from ZK; 2. Upon completing rebalance, update the ZK 
with new group registry or delete the registry if the group becomes empty; \n 
Address Jun's comments


Diffs (updated)
-

  core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 
af06ad45cdc46ac3bc27898ebc1a5bd5b1c7b19e 
  core/src/main/scala/kafka/coordinator/ConsumerGroupMetadata.scala 
47bdfa7cc86fd4e841e2b1d6bfd40f1508e643bd 
  core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala 
c39e6de34ee531c6dfa9107b830752bd7f8fbe59 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
2618dd39b925b979ad6e4c0abd5c6eaafb3db5d5 

Diff: https://reviews.apache.org/r/34450/diff/


Testing
---


Thanks,

Guozhang Wang

Re: Review Request 34450: Fix KAFKA-2017; rebased

[Guozhang Wang]

> On May 21, 2015, 12:16 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala, lines 
> > 102-106
> > 
> >
> > Another way to do this is to only load from ZK on the becoming leader 
> > event for an offsetTopic partition. Then, we don't have to read from ZK 
> > during join group, which will introduce unnecessary overhead when joining a 
> > new group.

I thought about this while working on the patch. The reason I feel it may not 
worth doing the loading thing upon become-leader is that:

1. When we are loading from ZK, we probably need to still reject any join-group 
request which is not loaded yet, like what we did in offset manager; this will 
introduce two more round trips (one for rediscover coordinator and one for 
another join-group, unless we introduce a separate "loading in progress" error 
code, then we can reduce it to one) compared with loading from ZK on the fly, 
which is just one ZK read.

2. It is likely that we only need to load from ZK once for each group, upon the 
first join-group request received (when two join requests are received at the 
same time we may need to unnecessarily read twice). And hence the latency 
overhead is not much compared with loading-all-at-once. The only concern is 
that it will slow down all handler threads a little bit when coordinator 
migration happens instead of taking one thread for reading all the ZK paths, 
which I feel is OK.


> On May 21, 2015, 12:16 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala, lines 
> > 369-381
> > 
> >
> > I was thinking whether it's worth including the leader epoch (of the 
> > corresponding offset topic partition) in the ZK value as we did for 
> > leaderAndIsr to prevent a zombie consumer coordinator from overwriting the 
> > value, during a soft failure. I am not sure if it's worth doing this 
> > immediately because
> > 
> > 1. When this happens, consumers can still recover after the heartbeat 
> > fails.
> > 2. It seems that doing this right is a bit more complicated. We need to 
> > keep the leader epoch in the ZK value. However, during a leader change, we 
> > probably need to update the values in ZK with the new leader epoch as well, 
> > in order to truely prevent the zombie coordinator from overwriting the 
> > value.
> > 
> > So, I think for now, we can just use the simple approach in this patch.

I think this is handled by the generation id, which is ever increasing, and 
coordinator writing to ZK must have its generation id = ZK value + 1.

One caveat though, is that when a group is empty we will remove it from ZK and 
when it appears again we will take it as a new group with generation id 
resetting to 1. Then a zombie coordinator happen to hold the "right" generation 
id after resetting maybe able to override. For this case we can create another 
JIRA.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34450/#review84604
---


On May 20, 2015, 4:13 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34450/
> ---
> 
> (Updated May 20, 2015, 4:13 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2017
> https://issues.apache.org/jira/browse/KAFKA-2017
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 1. Upon receiving join-group, if the group metadata cannot be found in the 
> local cache try to read it from ZK; 2. Upon completing rebalance, update the 
> ZK with new group registry or delete the registry if the group becomes empty
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 
> af06ad45cdc46ac3bc27898ebc1a5bd5b1c7b19e 
>   core/src/main/scala/kafka/coordinator/ConsumerGroupMetadata.scala 
> 47bdfa7cc86fd4e841e2b1d6bfd40f1508e643bd 
>   core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala 
> c39e6de34ee531c6dfa9107b830752bd7f8fbe59 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> 2618dd39b925b979ad6e4c0abd5c6eaafb3db5d5 
> 
> Diff: https://reviews.apache.org/r/34450/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Created] (KAFKA-2217) Refactor Client Selectable Interface for Better Concurrency Options

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2217:
--

 Summary: Refactor Client Selectable Interface for Better 
Concurrency Options
 Key: KAFKA-2217
 URL: https://issues.apache.org/jira/browse/KAFKA-2217
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson


The current Selectable interface makes thread-safe usage without external 
locking unlikely. In particular, the interface requires implementations to 
store internal lists containing the results from an invocation of poll. This 
makes dealing with issues such as KAFKA-2168 more difficult since it adds state 
which must be synchronized. Here are the offending methods:

{code:java}
interface Selectable {
  void poll(long timeout);
  List completedSends();
  List completedReceives();
  List disconnected();
  List connected();
  // rest excluded
}
{code}

The user is required to invoke poll, then extract the results from the 
corresponding methods. In order to avoid missing events, the caller must hold 
an external lock while they access the results of the poll. 

Instead, we can return the results directly from poll call using a container 
class. For example:

{code:java}
class PollResult {
  List completedSends;
  List completedReceives;
  List disconnected;
  List connected;
}

interface Selectable {
  PollResult poll(long timeout);
}
{code}

This should get us closer to a thread-safe NetworkClient, which would enable a 
more granular approach to synchronizing the KafkaConsumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

[Jun Rao]

Mayuresh,

The current plan is the following.

1. Add TMR v1, which still triggers auto topic creation.
2. Change the consumer client to TMR v1. Change the producer client to use
TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly
create the topic with the default server side partitions and replicas.
3. At some later time after the new clients are released and deployed,
disable auto topic creation in TMR v1. This will make sure consumers never
create new topics.
4. If needed, we can add a new config in the producer to control whether
TopicCreateRequest should be issued or not on UnknownTopicException. If
this is disabled and the topic doesn't exist, send will fail and the user
is expected to create the topic manually.

Thanks,

Jun


On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat  wrote:

> Hi,
> I had a question about TopicMetadata Request.
> Currently the way it works is :
>
> 1) Suppose a topic T1 does not exist.
> 2) Client wants to produce data to T1 using producer P1.
> 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka.
> This in turn creates the default number of partition. The number of
> partitions is a cluster wide config.
> 4) Same goes for a consumer. If the topic does not exist and new topic will
> be created when the consumer issues TopicMetadata request.
>
> Here are 2 use cases where it might not be suited :
>
> The auto creation flag for topics  is turned  ON.
>
> a) Some clients might not want to create topic with default number of
> partitions but with lower number of partitions. Currently in a multi-tenant
> environment this is not possible without changing the cluster wide default
> config.
>
> b) Some clients might want to just check if the topic exist or not but
> currently the topic gets created automatically using default number of
> partitions.
>
> Here are some ideas to address this :
>
> 1) The way this can be  addressed is that TopicMetadata request should have
> a way to specify whether it should only check if the topic exist or check
> and create a topic with given number of partitions. If the number of
> partitions is not specified use the default cluster wide config.
>
> OR
>
> 2) We should only allow TopicMetadata Request to get the metadata
> explicitly and not allow it to create a new topic. We should have another
> Request that takes in config parameters from the user regarding how he/she
> wants the topic to be created. This request can be used if we get an empty
> TopicMetadata Response.
>
>
> Thanks,
>
> Mayuresh
>
>
> On Thu, May 14, 2015 at 10:22 AM, Jun Rao  wrote:
>
> > For ListTopics, we decided not to add a ListTopics request for now and
> just
> > rely on passing in an empty list to TMR. We can revisit this in the
> future
> > if it becomes an issue.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy  wrote:
> >
> > > Just had a few minor questions before I join the vote thread.
> > > Apologies if these have been discussed:
> > >
> > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return
> > >   InvalidPartitions instead?
> > > - AdminClient.listTopics: should we allow listing all partitions? Or
> > >   do you intend for the client to issue listTopics followed by
> > >   describeTopics?
> > > - On returning future for partition reassignments: do we need to
> > >   return any future especially since you have the
> > >   verifyReassignPartitions method? For e.g., what happens if the
> > >   controller moves? The get should fail right? The client will then
> > >   need to connect to the new controller and reissue the request but
> > >   will then get ReassignPartitionsInProgress. So in that case the
> > >   client any way needs to rely in verifyReassignPartitions.
> > > - In past hangouts I think either you/Joe were mentioning the need to
> > >   locate the controller (and possibly other cluster metadata). It
> > >   appears we decided to defer this for a future KIP. Correct?
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote:
> > > > Guys,
> > > >
> > > > I've updated the wiki to reflect all previously discussed items
> > > > (regarding the schema only - this is included to phase 1).
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > >
> > > > I think we can have the final discussion today (for phase 1) and
> > > > in case no new remarks I will start the voting thread.
> > > >
> > > > With regards to AlterTopicRequest semantics. I agree with Jun,
> > > > and I think it's my bad I focused on "multiple topics in one
> request".
> > > > The same situation is possible in ProduceRequest, Fetch,
> TopicMetadata
> > > > and we handle it naturally and in the most transparent way - we
> > > > put all separate instructions into a map and thus silently ignore
> > > > duplicates.
> > > > This also makes Response part simple too - it's j

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555396#comment-14555396
 ] 

Guozhang Wang commented on KAFKA-2168:
--

Currently the NetworkClient's blocking calls: "poll" and "completeAll" (which 
triggers "poll" as well) are used in different layers as for:

1. KafkaConsumer.poll() naturally triggers client.poll().
2. Coordinator.ensureCoordinatorReady() to find a node to ask for consumer 
coordinator; 
3. Coordinator.sendAndReceive() for blocking requests as sync commit offsets, 
fetch offsets, join group, and fetch coordinator.
4. Fetcher.awaitMetadataUpdate() for blocking on metadata update.
5. Fetcher.offsetBefore() to blocking until successfully get list-offset 
responses.

Besides 1), all other four uses of NetworkClient's blocking calls are actually 
breaking the API declaration that consumer.poll(timeout) will block for as long 
as timeout period (for 3) actually sync commit offsets is OK since it is by 
definition blocking calls). In addition, 4) is a duplicate function as there is 
another KafkaConsumer.awaitMetadataUpdate.

So maybe we could consider combing the fix of KAFKA-1894 together and see if 
the following is possible:

1. Move all "poll" and "completeAll" from Coordinator and Fetcher into 
KafkaConsumer to make sure none of Coordiantor / Fetcher functions are 
blocking. For example, if the coordinator is not known, instead of blocking on 
ensureCoordinatorReady we should just call 
client.send(consumer-metadata-request) and return, and depend on the callback 
to handle coordinator discovery. This is mainly for KAFKA-1894 and will of 
course complicated the logic of Coordinator and Fetcher since now they need to 
maintain some more internal state.

2. Move Fetcher.resetOffset/OffsetBefore into subscription, and remove 
Fetcher.awaitMetadataUpdate. After this metadata becomes read-only to fetcher.

3. After the first two steps the finer synchronization approach would be 
simpler since now we only need to synchronize on subscriptions.

A side-effect though would be that calls to consumer.poll(timemout) may now 
returns much quicker with no returned data under partition re-assignment / sync 
offset commit, etc, and people need to call it multiple times before getting 
data. But as long as we document the usage clear I think this is fine.

> New consumer poll() can block other calls like position(), commit(), and 
> close() indefinitely
> -
>
> Key: KAFKA-2168
> URL: https://issues.apache.org/jira/browse/KAFKA-2168
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The new consumer is currently using very coarse-grained synchronization. For 
> most methods this isn't a problem since they finish quickly once the lock is 
> acquired, but poll() might run for a long time (and commonly will since 
> polling with long timeouts is a normal use case). This means any operations 
> invoked from another thread may block until the poll() call completes.
> Some example use cases where this can be a problem:
> * A shutdown hook is registered to trigger shutdown and invokes close(). It 
> gets invoked from another thread and blocks indefinitely.
> * User wants to manage offset commit themselves in a background thread. If 
> the commit policy is not purely time based, it's not currently possibly to 
> make sure the call to commit() will be processed promptly.
> Two possible solutions to this:
> 1. Make sure a lock is not held during the actual select call. Since we have 
> multiple layers (KafkaConsumer -> NetworkClient -> Selector -> nio Selector) 
> this is probably hard to make work cleanly since locking is currently only 
> performed at the KafkaConsumer level and we'd want it unlocked around a 
> single line of code in Selector.
> 2. Wake up the selector before synchronizing for certain operations. This 
> would require some additional coordination to make sure the caller of 
> wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() 
> thread being woken up and then promptly reacquiring the lock with a 
> subsequent long poll() call).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33620: Patch for KAFKA-1690

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review84845
---



clients/src/main/java/org/apache/kafka/common/network/SSLFactory.java


This doesn't seem to exist in java 6.


- Jun Rao


On May 21, 2015, 5:37 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33620/
> ---
> 
> (Updated May 21, 2015, 5:37 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1690
> https://issues.apache.org/jira/browse/KAFKA-1690
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1690. new java producer needs ssl support as a client.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. SSLFactory tests.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Added 
> PrincipalBuilder.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Addressing 
> reviews.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Addressing 
> reviews.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Addressing 
> reviews.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Fixed minor 
> issues with the patch.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Fixed minor 
> issues with the patch.
> 
> 
> Diffs
> -
> 
>   build.gradle cd2aa838fd53e8124f308979b1d70efe0c5725a6 
>   checkstyle/import-control.xml f2e6cec267e67ce8e261341e373718e14a8e8e03 
>   clients/src/main/java/org/apache/kafka/clients/ClientUtils.java 
> 0d68bf1e1e90fe9d5d4397ddf817b9a9af8d9f7a 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> bdff518b732105823058e6182f445248b45dc388 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> d301be4709f7b112e1f3a39f3c04cfa65f00fa60 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 8e336a3aa96c73f52beaeb56b931baf4b026cf21 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 187d0004c8c46b6664ddaffecc6166d4b47351e5 
>   clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java 
> c4fa058692f50abb4f47bd344119d805c60123f5 
>   clients/src/main/java/org/apache/kafka/common/config/SecurityConfigs.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/network/Authenticator.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/network/Channel.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/network/ChannelBuilder.java 
> PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/network/DefaultAuthenticator.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/network/PlainTextChannelBuilder.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/network/PlainTextTransportLayer.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/network/SSLChannelBuilder.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/network/SSLFactory.java 
> PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> b5f8d83e89f9026dc0853e5f92c00b2d7f043e22 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 57de0585e5e9a53eb9dcd99cac1ab3eb2086a302 
>   clients/src/main/java/org/apache/kafka/common/network/TransportLayer.java 
> PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/protocol/SecurityProtocol.java 
> dab1a94dd29563688b6ecf4eeb0e180b06049d3f 
>   
> clients/src/main/java/org/apache/kafka/common/security/auth/DefaultPrincipalBuilder.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/security/auth/PrincipalBuilder.java
>  PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> f73eedb030987f018d8446bb1dcd98d19fa97331 
>   clients/src/test/java/org/apache/kafka/common/network/EchoServer.java 
> PRE-CREATION 
>   clients/src/test/java/org/apache/kafka/common/network/SSLFactoryTest.java 
> PRE-CREATION 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> PRE-CREATION 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> d5b306b026e788b4e5479f3419805aa49ae889f3 
>   clients/src/test/java/org/apache/kafka/

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

[Jiangjie Qin]

Based on the discussion we have, I just updated the KIP with the following
proposal and want to see if there is further comments.

The proposal is to have the following four timeout as end state.

1. max.buffer.full.block.ms   - To replace block.on.buffer.full. The max
time to block when buffer is full.
2. metadata.fetch.timeout.ms  - reuse metadata timeout as batch.timeout.ms
because it is essentially metadata not available.
3. replication.timeout.ms - It defines how long a server will wait for
the records to be replicated to followers.
4. network.request.timeout.ms - This timeout is used when producer sends
request to brokers through TCP connections. It specifies how long the
producer should wait for the response.

With the above approach, we can achieve the following.
* We can have bounded blocking time for send() = (1) + (2).
* The time after send() until response got received is generally bounded
by linger.ms + (2) + (4), not taking retries into consideration.

So from user’s perspective. Send() depends on metadata of a topic and
buffer space. I am not sure if user would really care about how long it
takes to receive the response because it is async anyway and we have so
many things to consider (retries, linger.ms, retry backoff time, request
timeout, etc).

I think these configurations are clear enough to let user understand at
the first glance. Please let me know what do you think.

Thanks.

Jiangjie (Becket) Qin



On 5/20/15, 9:55 AM, "Joel Koshy"  wrote:

>> The fact that I understand the producer internals and am still
>>struggling
>> to understand the implications of the different settings, how I would
>>set
>> them, and how they potentially interact such that I could set invalid
>> combinations seems like a red flag to me... Being able to say "I want
>> produce requests to timeout in 5s" shouldn't require adjusting 3 or 4
>> configs if the defaults would normally timeout out in something like
>>30s.
>> 
>> Setting aside compatibility issues and focusing on the best set of
>>configs,
>> I agree with Jay that there are two things I actually want out of the
>>API.
>> The key thing is a per-request timeout, which should be enforced client
>> side. I would just expect this to follow the request through any
>>internals
>> so it can be enforced no matter where in the pipeline the request is.
>> Within each component in the pipeline we might have to compute how much
>> time we have left for the request in order to create a timeout within
>>that
>> setting. The second setting is to bound the amount of time spent
>>blocking
>> on send(). This is really an implementation detail, but one that people
>>are
>> complaining about enough that it seems worthwhile to provide control
>>over
>> it (and fixing it would just make that setting superfluous, not break
>> anything).
>>
>> Exposing a lot more settings also exposes a lot about the implementation
>> and makes it harder to improve the implementation in the future, but I
>> don't think we have listed good use cases for setting each of them
>> individually. Why would the user specifically care about how much time
>>the
>> request spends in the accumulator vs. some other component (assuming
>>they
>> have the overall timeout)? Same for requests in flight, as long as I
>>have
>> that client side timeout? And if they care about what component is the
>> bottleneck, could that be better exposed by the exceptions that are
>> returned rather than a ton of different settings?
>
>Agreed with the above. I'm also extremely wary of configs that are
>inherently unintuitive, or can interact to yield unintuitive behavior.
>OTOH I think it is okay if a config is categorized as "advanced" or if
>it requires deeper knowledge of the internals of the producer (or the
>configured system in general). i.e., as long as we think long and hard
>and agree on necessity (driven by clear use cases) before adding such
>configs. We should also consider how we can simplify or even eliminate
>existing configs.
>
>Re: requests in flight may be a good example: Becket had given a valid
>use-case i.e., support strict ordering. Maybe we can replace it with a
>"enable.strict.ordering" config which is clearer in intent and would
>internally ensure only one in-flight request per partition and default
>to a fixed in-flight requests (say, five or 10) if set to false. If we
>implement idempotence then we won't even need that.
>
>> On Tue, May 19, 2015 at 7:13 PM, Jiangjie Qin
>>
>> wrote:
>> 
>> > Hi Jay,
>> >
>> > I updated what I think int KIP wiki. Just a short summary here.
>>Because we
>> > need timeout for:
>> > 1. Send()
>> > 2. Batches in accumulator
>> > 3. Requests in flight.
>> > That means we need to have at least three configurations if we do not
>> > reuse configurations.
>> >
>> > I think we probably want to also separate the configurations for
>>exception
>> > handling and SLA purposes as well.
>> > My understanding of the configurations we are discussing here is they
>>are
>> > for 

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

[Joel Koshy]

> Here are some ideas to address this :
> 
> 1) The way this can be  addressed is that TopicMetadata request should have
> a way to specify whether it should only check if the topic exist or check
> and create a topic with given number of partitions. If the number of
> partitions is not specified use the default cluster wide config.
> 
> OR
> 
> 2) We should only allow TopicMetadata Request to get the metadata
> explicitly and not allow it to create a new topic. We should have another
> Request that takes in config parameters from the user regarding how he/she
> wants the topic to be created. This request can be used if we get an empty
> TopicMetadata Response.

I may be misunderstanding your points, but I think these are already
addressed - can you look at the
CreateTopicRequest/TopicMetadataRequestv1 section and verify?

> 
> 
> Thanks,
> 
> Mayuresh
> 
> 
> On Thu, May 14, 2015 at 10:22 AM, Jun Rao  wrote:
> 
> > For ListTopics, we decided not to add a ListTopics request for now and just
> > rely on passing in an empty list to TMR. We can revisit this in the future
> > if it becomes an issue.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy  wrote:
> >
> > > Just had a few minor questions before I join the vote thread.
> > > Apologies if these have been discussed:
> > >
> > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return
> > >   InvalidPartitions instead?
> > > - AdminClient.listTopics: should we allow listing all partitions? Or
> > >   do you intend for the client to issue listTopics followed by
> > >   describeTopics?
> > > - On returning future for partition reassignments: do we need to
> > >   return any future especially since you have the
> > >   verifyReassignPartitions method? For e.g., what happens if the
> > >   controller moves? The get should fail right? The client will then
> > >   need to connect to the new controller and reissue the request but
> > >   will then get ReassignPartitionsInProgress. So in that case the
> > >   client any way needs to rely in verifyReassignPartitions.
> > > - In past hangouts I think either you/Joe were mentioning the need to
> > >   locate the controller (and possibly other cluster metadata). It
> > >   appears we decided to defer this for a future KIP. Correct?
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote:
> > > > Guys,
> > > >
> > > > I've updated the wiki to reflect all previously discussed items
> > > > (regarding the schema only - this is included to phase 1).
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > >
> > > > I think we can have the final discussion today (for phase 1) and
> > > > in case no new remarks I will start the voting thread.
> > > >
> > > > With regards to AlterTopicRequest semantics. I agree with Jun,
> > > > and I think it's my bad I focused on "multiple topics in one request".
> > > > The same situation is possible in ProduceRequest, Fetch, TopicMetadata
> > > > and we handle it naturally and in the most transparent way - we
> > > > put all separate instructions into a map and thus silently ignore
> > > > duplicates.
> > > > This also makes Response part simple too - it's just a map
> > > Topic->ErrorCode.
> > > > I think we need to follow the same approach for Alter (and Create,
> > > Delete)
> > > > request. With this we add nothing new in terms of batch requests
> > > > semantics.
> > > >
> > > > Thanks,
> > > > Andrii Biletskyi
> > >
> >
> 
> 
> 
> -- 
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125

Re: Review Request 33620: Patch for KAFKA-1690

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review84232
---


Thanks for the patch. A few more comments below.


clients/src/main/java/org/apache/kafka/clients/ClientUtils.java


Since this property has a default, the key will always be available. So, we 
don't need the check on containsKey().



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


Do we need config.values? Could we just pass in config.originals() as we do 
for the serializer?



clients/src/main/java/org/apache/kafka/common/config/SecurityConfigs.java


Since all the configs are SSL specific, should we rename it to 
SslSecurityConfigs? Then, later on, we can add a SaslSecurityConfigs.



clients/src/main/java/org/apache/kafka/common/config/SecurityConfigs.java


This seems to be client specific. Perhaps it should be part of 
ClientCommonConfigs.



clients/src/main/java/org/apache/kafka/common/network/Channel.java


These don't seem to be used. Are they useful?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Could you describe the typical handshake flow in the comment?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Don't we need to set the interest ops for key depending on the 
handshakeStatus?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


If we are re-getting the handshake state on every call, can handshakeStatus 
be local?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Should the test be >= ? There are a few other places where we check for > 
instead of >=.



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


available data => available data size



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


It seems the flush() here needs to check if the socket is writable.



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


I am concerned about the concurrency here. The interest ops can be set in 
the task thread and the selector thread. How do we guarantee that the right 
interest bit is set in the end?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


It seems that the return result can be indeterministic, depending on 
whether the task completes or not.



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Would it be better to just throw the exception in the NOT_HANDSHAKING case 
in handshake()?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Should we add a check and throw an IllegalStateException if netWriteBuffer 
is not empty?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Why do we need to handle NEED_TASK here instead of just in handshake()?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Not sure why we are looping back here. If the HandshakeStatus is 
NEED_UNWRAP, we loop back w/o reading more data from the socket, will we get 
into infinite loop?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Hmm, shoudn't we check for remaining() > 0 here?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Could we rename unwrap as sth like result?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


When the buffer overflow, what can happen is that we read all bytes for a 
response from netwrok. Then we hit the buffer overflow. In this case, we want 
to drain all bytes from the app buffer into dst in the read() call. Otherwise, 
the next selector.poll() can take arbitrarily long since there is no more 

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

[Mayuresh Gharat]

Hi,
I had a question about TopicMetadata Request.
Currently the way it works is :

1) Suppose a topic T1 does not exist.
2) Client wants to produce data to T1 using producer P1.
3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka.
This in turn creates the default number of partition. The number of
partitions is a cluster wide config.
4) Same goes for a consumer. If the topic does not exist and new topic will
be created when the consumer issues TopicMetadata request.

Here are 2 use cases where it might not be suited :

The auto creation flag for topics  is turned  ON.

a) Some clients might not want to create topic with default number of
partitions but with lower number of partitions. Currently in a multi-tenant
environment this is not possible without changing the cluster wide default
config.

b) Some clients might want to just check if the topic exist or not but
currently the topic gets created automatically using default number of
partitions.

Here are some ideas to address this :

1) The way this can be  addressed is that TopicMetadata request should have
a way to specify whether it should only check if the topic exist or check
and create a topic with given number of partitions. If the number of
partitions is not specified use the default cluster wide config.

OR

2) We should only allow TopicMetadata Request to get the metadata
explicitly and not allow it to create a new topic. We should have another
Request that takes in config parameters from the user regarding how he/she
wants the topic to be created. This request can be used if we get an empty
TopicMetadata Response.


Thanks,

Mayuresh


On Thu, May 14, 2015 at 10:22 AM, Jun Rao  wrote:

> For ListTopics, we decided not to add a ListTopics request for now and just
> rely on passing in an empty list to TMR. We can revisit this in the future
> if it becomes an issue.
>
> Thanks,
>
> Jun
>
> On Wed, May 13, 2015 at 3:31 PM, Joel Koshy  wrote:
>
> > Just had a few minor questions before I join the vote thread.
> > Apologies if these have been discussed:
> >
> > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return
> >   InvalidPartitions instead?
> > - AdminClient.listTopics: should we allow listing all partitions? Or
> >   do you intend for the client to issue listTopics followed by
> >   describeTopics?
> > - On returning future for partition reassignments: do we need to
> >   return any future especially since you have the
> >   verifyReassignPartitions method? For e.g., what happens if the
> >   controller moves? The get should fail right? The client will then
> >   need to connect to the new controller and reissue the request but
> >   will then get ReassignPartitionsInProgress. So in that case the
> >   client any way needs to rely in verifyReassignPartitions.
> > - In past hangouts I think either you/Joe were mentioning the need to
> >   locate the controller (and possibly other cluster metadata). It
> >   appears we decided to defer this for a future KIP. Correct?
> >
> > Thanks,
> >
> > Joel
> >
> > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote:
> > > Guys,
> > >
> > > I've updated the wiki to reflect all previously discussed items
> > > (regarding the schema only - this is included to phase 1).
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > >
> > > I think we can have the final discussion today (for phase 1) and
> > > in case no new remarks I will start the voting thread.
> > >
> > > With regards to AlterTopicRequest semantics. I agree with Jun,
> > > and I think it's my bad I focused on "multiple topics in one request".
> > > The same situation is possible in ProduceRequest, Fetch, TopicMetadata
> > > and we handle it naturally and in the most transparent way - we
> > > put all separate instructions into a map and thus silently ignore
> > > duplicates.
> > > This also makes Response part simple too - it's just a map
> > Topic->ErrorCode.
> > > I think we need to follow the same approach for Alter (and Create,
> > Delete)
> > > request. With this we add nothing new in terms of batch requests
> > > semantics.
> > >
> > > Thanks,
> > > Andrii Biletskyi
> >
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: Review Request 33620: Patch for KAFKA-1690

[Sriharsha Chintalapani]

> On May 22, 2015, 12:14 a.m., Michael Herstine wrote:
> > clients/src/main/java/org/apache/kafka/common/security/auth/PrincipalBuilder.java,
> >  line 44
> > 
> >
> > I'm trying to imagine implementing `buildPrincipal` in such a way as to 
> > make a Principal from information found in the peer's certifcate: how could 
> > I obtain that? I don't see (immediately) a way to navigate from either the 
> > `TransportLayer` or the `Authenticator` to the current `SSLSession`...

The reason I added bothe TransportLayer and Authenticator to the buildPrincipal 
to give user option to use one or both of them to construct a principal. It 
depends on the which TransportLayer or Authenticator they might be using. 
in the case you pointed out, one can do this in buildPrincipal

if (transportLayer is instanceOf SSLTransportLayer)
SSLSession = ((SSLTransportLayer) sslTransportLayer).sslSession();


> On May 22, 2015, 12:14 a.m., Michael Herstine wrote:
> > clients/src/main/java/org/apache/kafka/common/network/SSLChannelBuilder.java,
> >  line 37
> > 
> >
> > More a question than an issue: do you want `SSLChannelBuilder` to 
> > always build a client channel?

Sorry I should've added a comment for this. I left this as a more of a question 
and wanted to get feedback from the reviewers. 
I was thinking injecting a key value pair into the configs from producer and 
client that says "ssl.mode" "SSLFactory.Mode.CLIENT" for producer/client
and on the broker side "ssl.mode" "SSLFactory.Mode.SERVER".


> On May 22, 2015, 12:14 a.m., Michael Herstine wrote:
> > clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java,
> >  line 65
> > 
> >
> > Two questions:
> > 
> >   - is this initialized somewhere?
> >   - AFAICT it is a copy of the interest operations in the key... so why 
> > does it exist?
> >   -

The reason behind this is while handshaking is going on we don't want external 
selector.send or reads set the interestBits. So while handshake happening we 
are storing the pending interest bits into this variable and once handshake is 
done we do key.interestOps(interestOps)


- Sriharsha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review84235
---


On May 21, 2015, 5:37 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33620/
> ---
> 
> (Updated May 21, 2015, 5:37 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1690
> https://issues.apache.org/jira/browse/KAFKA-1690
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1690. new java producer needs ssl support as a client.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. SSLFactory tests.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Added 
> PrincipalBuilder.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Addressing 
> reviews.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Addressing 
> reviews.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Addressing 
> reviews.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Fixed minor 
> issues with the patch.
> 
> 
> KAFKA-1690. new java producer needs ssl support as a client. Fixed minor 
> issues with the patch.
> 
> 
> Diffs
> -
> 
>   build.gradle cd2aa838fd53e8124f308979b1d70efe0c5725a6 
>   checkstyle/import-control.xml f2e6cec267e67ce8e261341e373718e14a8e8e03 
>   clients/src/main/java/org/apache/kafka/clients/ClientUtils.java 
> 0d68bf1e1e90fe9d5d4397ddf817b9a9af8d9f7a 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> bdff518b732105823058e6182f445248b45dc388 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> d301be4709f7b112e1f3a39f3c04cfa65f00fa60 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 8e336a3aa96c73f52beaeb56b931baf4b026cf21 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 187d0004c8c46b6664ddaffecc6166d4b47351e5 
>   clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java 
> c4fa058692f50abb4f47bd344119d805c60123f5 
>   clients/src/main/java/org/apache/kafka/common/config/SecurityConfigs.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache

Re: Review Request 33620: Patch for KAFKA-1690

[Michael Herstine]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review84235
---



clients/src/main/java/org/apache/kafka/common/network/SSLFactory.java


On the server, there are actually three settings:

  - require client authentication/certificate (wherein the handshake will 
fail if the client doesn't authenticate)
  - want client authentication/certificate (wherein the server will request 
client authentication, but allow the handhsake go through if the client doesn't 
comply)
  - no client certificate/authentication requested (wherein the server 
won't even ask)

Regrettably, the Java Secure Sockets Extensions represents this with two 
boolean parameters: `SSLEngine.setWantClientAuth` and 
`SSLEngine.setNeedClientAuth`. You seem to have adopted their "need client 
authentication" parameter, but not their "want client authentication" 
parameter; is this intentional?



clients/src/main/java/org/apache/kafka/common/network/SSLChannelBuilder.java


More a question than an issue: do you want `SSLChannelBuilder` to always 
build a client channel?



clients/src/main/java/org/apache/kafka/common/network/SSLFactory.java


There's a subtle bug here, and I only know about it because we got burned 
by the same thing. Here it is: suppose `needClientAuth` is true, and 
`wantClientAuth` is false. In this case, the code as written will create an 
`SSLEngine` instance with no client auth desired.

Crazy, I know. From the `SSLEngine` javadoc: "An engine's client 
authentication setting is one of the following:

  - client authentication required
  - client authentication requested
  - no client authentication desired

...Calling this method overrides any previous setting made by this method 
or setNeedClientAuth(boolean)."

IOW, the interface to `SSLEngine` models a parameter with three possible 
settings using two booleans. This is a great example of why interface design is 
so important. Anyway, I'd suggest code like this:

if (needClientAuth) {
sslEngine.setNeedClientAuth(true);
} else {
sslEngine.setWantClientAuth(wantClientAuth);
}

This covers the four cases:

```
 "wants"   
  True   False
 +-+-+
True | client auth | client auth |
 | required| required|
"needs"  +-+-+
False| client auth | no client   |
 | requested   | auth desired|
 +-+-+

```

Check me, but I think that's what we want here.



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Two questions:

  - is this initialized somewhere?
  - AFAICT it is a copy of the interest operations in the key... so why 
does it exist?
  -



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Does `appReadBuffer` need to be initialized, too?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


I'm a bit confused: does line 98 not duplicate the actions of line 96?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Typo: than => then



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Do you ever clear out `appReadBuffer`? As I read through the logic, it 
seems that you keep accumulating unwrapped messages therein during the 
handshake?



clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java


Do you want to call `compact` on the source buffer?



clients/src/main/java/org/apache/kafka/common/security/auth/PrincipalBuilder.java


I'm trying to imagine implementing `buildPrincipal` in such a way as to 
make a Principal from information found in the peer's certifcate: how could I 
obtain that? I don't see (immediately) a way to navigate from either the 
`TransportLayer` or the `Authenticator` to the current `SSLSession`...


Wow... this has turned into a major piece of code, hasn't it? Thanks for your 
attention to my comments. I tried to walk through the logic imagining partial 
reads & partial writes, but I thin

Re: Request for permission to edit pages in the wiki

[Joel Koshy]

Sorry about that - should work now.

On Thu, May 21, 2015 at 11:22:40PM +0100, Ismael Juma wrote:
> On Wed, May 20, 2015 at 11:54 PM, Joel Koshy  wrote:
> 
> > Done
> >
> 
> Thank you Joel, but the "Edit" link is still not available to me.
> 
> Best,
> Ismael
> 
> 
> >
> > On Wed, May 20, 2015 at 11:45:48PM +0100, Ismael Juma wrote:
> > > Hi,
> > >
> > > In order to edit the "Patch submission and review" page with information
> > on
> > > how to merge GitHub pull requests, it would be helpful to have edit
> > > permission for Kafka's Confluence pages. My Confluence account id is
> > > `ijuma`[1].
> > >
> > > Thanks,
> > > Ismael
> > >
> > > [1] https://cwiki.apache.org/confluence/display/~ijuma
> >
> >

Jenkins build is back to normal : Kafka-trunk #495

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-25 System test improvements

[Geoffrey Anderson]

Hi Ashish,

Looks like Ewen already hit the main points, but a few additions:

1. ducktape repo is here: https://github.com/confluentinc/ducktape
ducktape itself will be pip installable in the near future, and Kafka
system tests will be able to depend on a particular version of ducktape.

2.  The reporting is nothing fancy. We're definitely open to feedback, but
it consists of:
- top level summary of the test run (simple PASS/FAIL for each test)
- top level info and debug logs
- per-test info and debug logs
- per-test "service" logs gathered from each service used in the test. For
example, if your test pulls up a Kafka cluster with 5 brokers, the end
result will have the Kafka logs from each of those 5 machines.

Cheers,
Geoff

On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava 
wrote:

> Ashish,
>
> 1. That was the plan. We put some effort into cleanly separating the
> framework so it would be reusable across many projects.
> 2. I think you're seeing a test in progress where the final report hasn't
> been created yet. If you visit one of the older ones you'll see it has a
> landing page with links:
> http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently
> we need to adjust when we update the 'latest' symlink. The logs that are
> collected for tests are configurable, and service implementations include
> sane defaults (so, e.g., you will always get the normal log file for Kafka,
> but only get the data files if the test asks for them).
> 3. No code coverage support. Haven't looked into it, so I couldn't comment
> on how hard it would be to add.
>
> -Ewen
>
> On Thu, May 21, 2015 at 2:38 PM, Ashish Singh  wrote:
>
> > Geoffrey,
> >
> > This looks great!
> >
> > A few questions.
> > 1. Will ducktape be maintained separately as a github repo?
> > 2. How easy is viewing the test results and logs. The link in KIP,
> > http://testing.confluent.io/confluent_platform/latest/, lists a bunch of
> > files and dirs. Could you add to KIP how the result and logs for the
> tests
> > will be organized.
> > 3. Does it support code coverage? If not, how easy/ difficult would it
> be?
> >
> > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson 
> > wrote:
> >
> > > Great, I'll work on putting together a more detailed map of this
> > > replacement process.
> > >
> > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira 
> > > wrote:
> > >
> > > > Love this idea :)
> > > >
> > > > I took a look at Ducktape API and it looks like a good fit - clean
> API,
> > > > extensible, easy to use and powerful enough for our use-case.
> > > >
> > > > Something I'd like to see as part of the KIP is a map of what
> > system-test
> > > > currently tests, which ones we want to replace and a JIRA for
> replacing
> > > > (possibly one for each group of tests).
> > > > Basically, I know we all want to use the new system for new test
> cases
> > > > (upgrades, etc), but I really want to make sure we don't get stuck
> with
> > > > both systems forever.
> > > >
> > > > Gwen
> > > >
> > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
> ge...@confluent.io
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Just kicking off the discussion thread on KIP-25
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements
> > > > >
> > > > > Thanks,
> > > > > Geoff
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>
>
>
> --
> Thanks,
> Ewen
>

Re: Request for permission to edit pages in the wiki

[Ismael Juma]

On Wed, May 20, 2015 at 11:54 PM, Joel Koshy  wrote:

> Done
>

Thank you Joel, but the "Edit" link is still not available to me.

Best,
Ismael


>
> On Wed, May 20, 2015 at 11:45:48PM +0100, Ismael Juma wrote:
> > Hi,
> >
> > In order to edit the "Patch submission and review" page with information
> on
> > how to merge GitHub pull requests, it would be helpful to have edit
> > permission for Kafka's Confluence pages. My Confluence account id is
> > `ijuma`[1].
> >
> > Thanks,
> > Ismael
> >
> > [1] https://cwiki.apache.org/confluence/display/~ijuma
>
>

Re: [DISCUSS] KIP-25 System test improvements

[Ewen Cheslack-Postava]

Ashish,

1. That was the plan. We put some effort into cleanly separating the
framework so it would be reusable across many projects.
2. I think you're seeing a test in progress where the final report hasn't
been created yet. If you visit one of the older ones you'll see it has a
landing page with links:
http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently
we need to adjust when we update the 'latest' symlink. The logs that are
collected for tests are configurable, and service implementations include
sane defaults (so, e.g., you will always get the normal log file for Kafka,
but only get the data files if the test asks for them).
3. No code coverage support. Haven't looked into it, so I couldn't comment
on how hard it would be to add.

-Ewen

On Thu, May 21, 2015 at 2:38 PM, Ashish Singh  wrote:

> Geoffrey,
>
> This looks great!
>
> A few questions.
> 1. Will ducktape be maintained separately as a github repo?
> 2. How easy is viewing the test results and logs. The link in KIP,
> http://testing.confluent.io/confluent_platform/latest/, lists a bunch of
> files and dirs. Could you add to KIP how the result and logs for the tests
> will be organized.
> 3. Does it support code coverage? If not, how easy/ difficult would it be?
>
> On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson 
> wrote:
>
> > Great, I'll work on putting together a more detailed map of this
> > replacement process.
> >
> > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira 
> > wrote:
> >
> > > Love this idea :)
> > >
> > > I took a look at Ducktape API and it looks like a good fit - clean API,
> > > extensible, easy to use and powerful enough for our use-case.
> > >
> > > Something I'd like to see as part of the KIP is a map of what
> system-test
> > > currently tests, which ones we want to replace and a JIRA for replacing
> > > (possibly one for each group of tests).
> > > Basically, I know we all want to use the new system for new test cases
> > > (upgrades, etc), but I really want to make sure we don't get stuck with
> > > both systems forever.
> > >
> > > Gwen
> > >
> > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson  >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Just kicking off the discussion thread on KIP-25
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements
> > > >
> > > > Thanks,
> > > > Geoff
> > > >
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
>



-- 
Thanks,
Ewen

Jenkins build is back to normal : KafkaPreCommit #107

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-2186) Follow-up patch of KAFKA-1650

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555145#comment-14555145
 ] 

Jiangjie Qin commented on KAFKA-2186:
-

Created reviewboard https://reviews.apache.org/r/34569/diff/
 against branch origin/trunk

> Follow-up patch of KAFKA-1650
> -
>
> Key: KAFKA-2186
> URL: https://issues.apache.org/jira/browse/KAFKA-2186
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-2186.patch, KAFKA-2186.patch
>
>
> Offsets commit with a map was added in KAFKA-1650. It should be added to 
> consumer connector java API also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 34569: Patch for KAFKA-2186

[Jiangjie Qin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34569/
---

Review request for kafka.


Bugs: KAFKA-2186
https://issues.apache.org/jira/browse/KAFKA-2186


Repository: kafka


Description
---

rebased on trunk


Diffs
-

  core/src/main/scala/kafka/javaapi/consumer/ConsumerConnector.java 
cc3400ff81fc0db69b5129ad7b440f20a211a79d 

Diff: https://reviews.apache.org/r/34569/diff/


Testing
---


Thanks,

Jiangjie Qin

[jira] [Updated] (KAFKA-2186) Follow-up patch of KAFKA-1650

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2186:

Status: Patch Available  (was: Reopened)

> Follow-up patch of KAFKA-1650
> -
>
> Key: KAFKA-2186
> URL: https://issues.apache.org/jira/browse/KAFKA-2186
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-2186.patch, KAFKA-2186.patch
>
>
> Offsets commit with a map was added in KAFKA-1650. It should be added to 
> consumer connector java API also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2186) Follow-up patch of KAFKA-1650

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2186:

Attachment: KAFKA-2186.patch

> Follow-up patch of KAFKA-1650
> -
>
> Key: KAFKA-2186
> URL: https://issues.apache.org/jira/browse/KAFKA-2186
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-2186.patch, KAFKA-2186.patch
>
>
> Offsets commit with a map was added in KAFKA-1650. It should be added to 
> consumer connector java API also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-25 System test improvements

[Ashish Singh]

Geoffrey,

This looks great!

A few questions.
1. Will ducktape be maintained separately as a github repo?
2. How easy is viewing the test results and logs. The link in KIP,
http://testing.confluent.io/confluent_platform/latest/, lists a bunch of
files and dirs. Could you add to KIP how the result and logs for the tests
will be organized.
3. Does it support code coverage? If not, how easy/ difficult would it be?

On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson 
wrote:

> Great, I'll work on putting together a more detailed map of this
> replacement process.
>
> On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira 
> wrote:
>
> > Love this idea :)
> >
> > I took a look at Ducktape API and it looks like a good fit - clean API,
> > extensible, easy to use and powerful enough for our use-case.
> >
> > Something I'd like to see as part of the KIP is a map of what system-test
> > currently tests, which ones we want to replace and a JIRA for replacing
> > (possibly one for each group of tests).
> > Basically, I know we all want to use the new system for new test cases
> > (upgrades, etc), but I really want to make sure we don't get stuck with
> > both systems forever.
> >
> > Gwen
> >
> > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson 
> > wrote:
> >
> > > Hi,
> > >
> > > Just kicking off the discussion thread on KIP-25
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements
> > >
> > > Thanks,
> > > Geoff
> > >
> >
>



-- 

Regards,
Ashish

[jira] [Reopened] (KAFKA-2186) Follow-up patch of KAFKA-1650

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy reopened KAFKA-2186:
---

Reverted. Reopening due to compilation failure.

> Follow-up patch of KAFKA-1650
> -
>
> Key: KAFKA-2186
> URL: https://issues.apache.org/jira/browse/KAFKA-2186
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-2186.patch
>
>
> Offsets commit with a map was added in KAFKA-1650. It should be added to 
> consumer connector java API also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-25 System test improvements

[Geoffrey Anderson]

Great, I'll work on putting together a more detailed map of this
replacement process.

On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira 
wrote:

> Love this idea :)
>
> I took a look at Ducktape API and it looks like a good fit - clean API,
> extensible, easy to use and powerful enough for our use-case.
>
> Something I'd like to see as part of the KIP is a map of what system-test
> currently tests, which ones we want to replace and a JIRA for replacing
> (possibly one for each group of tests).
> Basically, I know we all want to use the new system for new test cases
> (upgrades, etc), but I really want to make sure we don't get stuck with
> both systems forever.
>
> Gwen
>
> On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson 
> wrote:
>
> > Hi,
> >
> > Just kicking off the discussion thread on KIP-25
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements
> >
> > Thanks,
> > Geoff
> >
>

Build failed in Jenkins: KafkaPreCommit #106

[Apache Jenkins Server]

See 

Changes:

[jjkoshy] KAFKA-2190; Flush mirror maker before commiting offsets; abort the

[jjkoshy] KAFKA-2186; Follow-up to KAFKA-1650 - add selective offset commit to

--
Started by an SCM change
Building remotely on ubuntu-5 (docker Ubuntu ubuntu5 ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 23ff851f30bb55e794aefd2fae5367845d9230ee 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 23ff851f30bb55e794aefd2fae5367845d9230ee
 > git rev-list bb133c63b0645c22f2d9b76393886fb506b14a93 # timeout=10
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
[KafkaPreCommit] $ /bin/bash -xe /tmp/hudson95350337355476673.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.5
:downloadWrapper UP-TO-DATE

BUILD SUCCESSFUL

Total time: 9.943 secs
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
[KafkaPreCommit] $ /bin/bash -xe /tmp/hudson8955856312388100753.sh
+ ./gradlew -PscalaVersion=2.10.1 test
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.0/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.1
:clients:compileJava UP-TO-DATE
:clients:processResources UP-TO-DATE
:clients:classes UP-TO-DATE
:clients:checkstyleMain UP-TO-DATE
:clients:compileTestJava UP-TO-DATE
:clients:processTestResources UP-TO-DATE
:clients:testClasses UP-TO-DATE
:clients:checkstyleTest UP-TO-DATE
:clients:test UP-TO-DATE
:contrib:compileJava UP-TO-DATE
:contrib:processResources UP-TO-DATE
:contrib:classes UP-TO-DATE
:contrib:compileTestJava UP-TO-DATE
:contrib:processTestResources UP-TO-DATE
:contrib:testClasses UP-TO-DATE
:contrib:test UP-TO-DATE
:clients:jar UP-TO-DATE
:core:compileJava UP-TO-DATE
:core:compileScala:61:
 class ZookeeperConsumerConnector needs to be abstract, since method 
commitOffsets in trait ConsumerConnector of type (offsetsToCommit: 
java.util.Map[org.apache.kafka.common.TopicPartition,kafka.common.OffsetAndMetadata],
 retryOnFailure: Boolean)Unit is not defined
(Note that 
java.util.Map[org.apache.kafka.common.TopicPartition,kafka.common.OffsetAndMetadata]
 does not match 
java.util.Map[kafka.common.TopicAndPartition,kafka.common.OffsetAndMetadata]: 
their type parameters differ)
private[kafka] class ZookeeperConsumerConnector(val config: ConsumerConfig,
 ^
one error found
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:compileScala'.
> Compilation failed

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 26.347 secs
Build step 'Execute shell' marked build as failure
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1

Build failed in Jenkins: Kafka-trunk #494

[Apache Jenkins Server]

See 

Changes:

[jjkoshy] KAFKA-2190; Flush mirror maker before commiting offsets; abort the

[jjkoshy] KAFKA-2186; Follow-up to KAFKA-1650 - add selective offset commit to

--
[...truncated 446 lines...]

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[2] PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[3] PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testSimple 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testNulls 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testDefault 
PASSED

org.apache.kafka.common.metrics.MetricsTest > testMetricName PASSED

org.apache.kafka.common.metrics.MetricsTest > testSimpleStats PASSED

org.apache.kafka.common.metrics.MetricsTest > testHierarchicalSensors PASSED

org.apache.kafka.common.metrics.MetricsTest > testBadSensorHiearchy PASSED

org.apache.kafka.common.metrics.MetricsTest > testEventWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testTimeWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testOldDataHasNoEffect PASSED

org.apache.kafka.common.metrics.MetricsTest > testDuplicateMetricName PASSED

org.apache.kafka.common.metrics.MetricsTest > testQuotas PASSED

org.apache.kafka.common.metrics.MetricsTest > testPercentiles PASSED

org.apache.kafka.common.metrics.JmxReporterTest > testJmxRegistration PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testHistogram PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testConstantBinScheme 
PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testLinearBinScheme PASSED

org.apache.kafka.common.config.ConfigDefTest > testBasicTypes PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired PASSED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice PASSED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultRange PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidators PASSED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
PASSED

org.apache.kafka.common.serialization.SerializationTest > testStringSerializer 
PASSED

org.apache.kafka.common.serialization.SerializationTest > testIntegerSerializer 
PASSED

org.apache.kafka.common.network.SelectorTest > testServerDisconnect PASSED

org.apache.kafka.common.network.SelectorTest > testClientDisconnect PASSED

org.apache.kafka.common.network.SelectorTest > testCantSendWithInProgress PASSED

org.apache.kafka.common.network.SelectorTest > testCantSendWithoutConnecting 
PASSED

org.apache.kafka.common.network.SelectorTest > testNoRouteToHost PASSED

org.apache.kafka.common.network.SelectorTest > testConnectionRefused PASSED

org.apache.kafka.common.network.SelectorTest > testNormalOperation PASSED

org.apache.kafka.common.network.SelectorTest > testSendLargeRequest PASSED

org.apache.kafka.common.network.SelectorTest > testEmptyRequest PASSED

org.apache.kafka.common.network.SelectorTest > testExistingConnectionId PASSED

org.apache.kafka.common.network.SelectorTest > testMute PASSED

org.apache.kafka.common.requests.RequestResponseTest > testSerialization PASSED

org.apache.kafka.clients.NetworkClientTest > testReadyAndDisconnect PASSED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode PASSED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse PASSED

org.apache.kafka.clients.MetadataTest > testMetadata PASSED

org.apache.kafka.clients.MetadataTest > testMetadataUpdateWaitTime PASSED

org.apache.kafka.clients.ClientUtilsTest > testParseAndValidateAddresses PASSED

org.apache.kafka.clients.ClientUtilsTest > testNoPort PASSED

org.apache.kafka.clients.consumer.MockConsumerTest > testSimpleMock PASSED

org.apache.kafka.clients.consumer.KafkaConsumerTest > testConstructorClose 
PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > 
testNormalHeartbeat PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > 
testCoordinatorNotAvailable PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > 
testNotCoordinator PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > 
testIllegalGeneration PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > 
testCoordinatorDisconnect PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > 
testNormalJoinGroup PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > testReJoinGroup 
PASSED

org.apache.kafka.clients.consumer.internals.CoordinatorTest > 
testCommitOffsetNo

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554867#comment-14554867
 ] 

Jason Gustafson commented on KAFKA-2168:


Yes, that is the tradeoff, but at least it would be confined to the 
KafkaConsumer class.

> New consumer poll() can block other calls like position(), commit(), and 
> close() indefinitely
> -
>
> Key: KAFKA-2168
> URL: https://issues.apache.org/jira/browse/KAFKA-2168
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The new consumer is currently using very coarse-grained synchronization. For 
> most methods this isn't a problem since they finish quickly once the lock is 
> acquired, but poll() might run for a long time (and commonly will since 
> polling with long timeouts is a normal use case). This means any operations 
> invoked from another thread may block until the poll() call completes.
> Some example use cases where this can be a problem:
> * A shutdown hook is registered to trigger shutdown and invokes close(). It 
> gets invoked from another thread and blocks indefinitely.
> * User wants to manage offset commit themselves in a background thread. If 
> the commit policy is not purely time based, it's not currently possibly to 
> make sure the call to commit() will be processed promptly.
> Two possible solutions to this:
> 1. Make sure a lock is not held during the actual select call. Since we have 
> multiple layers (KafkaConsumer -> NetworkClient -> Selector -> nio Selector) 
> this is probably hard to make work cleanly since locking is currently only 
> performed at the KafkaConsumer level and we'd want it unlocked around a 
> single line of code in Selector.
> 2. Wake up the selector before synchronizing for certain operations. This 
> would require some additional coordination to make sure the caller of 
> wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() 
> thread being woken up and then promptly reacquiring the lock with a 
> subsequent long poll() call).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2186) Follow-up patch of KAFKA-1650

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-2186:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the patch - committed to trunk.

> Follow-up patch of KAFKA-1650
> -
>
> Key: KAFKA-2186
> URL: https://issues.apache.org/jira/browse/KAFKA-2186
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-2186.patch
>
>
> Offsets commit with a map was added in KAFKA-1650. It should be added to 
> consumer connector java API also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2190) Incorporate close(timeout) to Mirror Maker

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-2190:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the patch - committed to trunk.

> Incorporate close(timeout) to Mirror Maker
> --
>
> Key: KAFKA-2190
> URL: https://issues.apache.org/jira/browse/KAFKA-2190
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-2190.patch, KAFKA-2190_2015-05-15_19:50:42.patch
>
>
> Use close(0) when mirror maker exits accidentally to avoid reordering.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

I think we should remove the config part in TopicMetadataResponse. It's 
probably cleaner if Alter and Describe are the only way to view and modify 
configs but I don't feel very strongly about it. 

Re-summarizing the proposed changes to KIP-4:
- Change AlterTopic to not allow setting configs. Config changes will flow 
through AlterConfig. CreateTopic will still allow setting configs as it is nice 
to be able to specify configs while creating the topic.
- TopicMetadataResponse shoudn't return config for the topic. DescribeConfig is 
the way to go.
- Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig" as 
proposed in KIP-21.

Aditya


From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 21, 2015 10:50 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

What about TopicMetadataResponse in KIP-4? Do we remove the config part in
it?

Thanks,

Jun

On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey Jun,
>
> I've added a section on error codes on the KIP-21 wiki.
>
> Here are the proposed changes to KIP-4. I'll update the wiki shortly.
> - Change AlterTopic to not allow setting configs. Config changes will flow
> through AlterConfig. CreateTopic will still allow setting configs as it is
> nice to be able to specify configs while creating the topic.
> - Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig"
> as proposed in KIP-21.
>
>
> Thanks,
> Aditya
>
> 
> From: Jun Rao [j...@confluent.io]
> Sent: Thursday, May 21, 2015 8:41 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Aditya,
>
> For completeness, could you list the set of error codes in the wiki? Also,
> could you summarize the changes that are needed for the requests listed in
> KIP-4 and update the wiki accordingly?
>
> Thanks,
>
> Jun
>
> On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Thanks Andrii. I'll make the changes.
> >
> > I've also updated KIP-21 to include the new config requests. Take a look
> > and vote.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Aditya
> > 
> > From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> > Sent: Tuesday, May 19, 2015 2:26 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hi,
> >
> > Sorry I wasn't able to participate. I don't have objections about
> removing
> > config changes from AlterTopic (as I understand both AddedConfig and
> > DeletedConfig) - you are welcome to update the KIP page.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Updating the discussion with the latest comments.
> > >
> > > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig).
> I'll
> > > update KIP-21 with details on these.
> > > 2. Discussed during the KIP hangout. We are in agreement.
> > >
> > > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> > KIP
> > > can be implemented independently. Any concerns if we tackle it as two
> > > separate work items implementation wise?
> > >
> > > We also discussed changing the AlterTopic command in KIP-4 to not
> include
> > > config changes. Instead, all config changes will pass through the newly
> > > proposed AlterConfig. If no-one objects, I can make some changes to
> KIP-4
> > > to reflect this.
> > >
> > > Aditya
> > >
> > > 
> > > From: Jay Kreps [jay.kr...@gmail.com]
> > > Sent: Tuesday, May 19, 2015 10:51 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Hey Aditya,
> > >
> > > Two comments:
> > >
> > > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think it
> does
> > > make sense to allow setting config during topic creation. I agree with
> > your
> > > summary that having alter topic and alter config may be confusing, but
> > > there are also some non-config changes such as replication factor and
> > > partition count that alter topic can carry out. What is the final state
> > you
> > > are proposing?
> > >
> > > 2. This is implementation related so probably can be removed from the
> KIP
> > > entirely, but you seem to be proposing a separate config manager for
> each
> > > config override type. Should we just generalize TopicConfigManager to
> be
> > > ConfigOverrideManager and have it handle all the override types we will
> > > have? I think I may just be unclear on what you are proposing...
> > >
> > > -Jay
> > >
> > > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar <
> > > aaurad...@linkedin.com.invalid> wrote:
> > >
> > > > Yeah, that was just a typo. I've fixed it. Thanks for calling it out.
> > > >
> > > > In KIP-4, I believe w

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554799#comment-14554799
 ] 

Ewen Cheslack-Postava commented on KAFKA-2168:
--

But isn't that going to make a mess of all the methods in Kafka consumer since 
we need to do this everywhere we currently synchronize? And I don't see a good 
way of providing it as a generic utility since you need the body of the method 
within the finally block.

> New consumer poll() can block other calls like position(), commit(), and 
> close() indefinitely
> -
>
> Key: KAFKA-2168
> URL: https://issues.apache.org/jira/browse/KAFKA-2168
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The new consumer is currently using very coarse-grained synchronization. For 
> most methods this isn't a problem since they finish quickly once the lock is 
> acquired, but poll() might run for a long time (and commonly will since 
> polling with long timeouts is a normal use case). This means any operations 
> invoked from another thread may block until the poll() call completes.
> Some example use cases where this can be a problem:
> * A shutdown hook is registered to trigger shutdown and invokes close(). It 
> gets invoked from another thread and blocks indefinitely.
> * User wants to manage offset commit themselves in a background thread. If 
> the commit policy is not purely time based, it's not currently possibly to 
> make sure the call to commit() will be processed promptly.
> Two possible solutions to this:
> 1. Make sure a lock is not held during the actual select call. Since we have 
> multiple layers (KafkaConsumer -> NetworkClient -> Selector -> nio Selector) 
> this is probably hard to make work cleanly since locking is currently only 
> performed at the KafkaConsumer level and we'd want it unlocked around a 
> single line of code in Selector.
> 2. Wake up the selector before synchronizing for certain operations. This 
> would require some additional coordination to make sure the caller of 
> wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() 
> thread being woken up and then promptly reacquiring the lock with a 
> subsequent long poll() call).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-25 System test improvements

[Gwen Shapira]

Love this idea :)

I took a look at Ducktape API and it looks like a good fit - clean API,
extensible, easy to use and powerful enough for our use-case.

Something I'd like to see as part of the KIP is a map of what system-test
currently tests, which ones we want to replace and a JIRA for replacing
(possibly one for each group of tests).
Basically, I know we all want to use the new system for new test cases
(upgrades, etc), but I really want to make sure we don't get stuck with
both systems forever.

Gwen

On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson 
wrote:

> Hi,
>
> Just kicking off the discussion thread on KIP-25
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements
>
> Thanks,
> Geoff
>

[DISCUSS] KIP-25 System test improvements

[Geoffrey Anderson]

Hi,

Just kicking off the discussion thread on KIP-25

https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements

Thanks,
Geoff

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554743#comment-14554743
 ] 

Jason Gustafson commented on KAFKA-2168:


Note that the coordinator has a couple cases where poll is called in a loop. 
There's a separate issue to fix this: KAFKA-1894. Might want to hold off on 
this until that is resolved.

> New consumer poll() can block other calls like position(), commit(), and 
> close() indefinitely
> -
>
> Key: KAFKA-2168
> URL: https://issues.apache.org/jira/browse/KAFKA-2168
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The new consumer is currently using very coarse-grained synchronization. For 
> most methods this isn't a problem since they finish quickly once the lock is 
> acquired, but poll() might run for a long time (and commonly will since 
> polling with long timeouts is a normal use case). This means any operations 
> invoked from another thread may block until the poll() call completes.
> Some example use cases where this can be a problem:
> * A shutdown hook is registered to trigger shutdown and invokes close(). It 
> gets invoked from another thread and blocks indefinitely.
> * User wants to manage offset commit themselves in a background thread. If 
> the commit policy is not purely time based, it's not currently possibly to 
> make sure the call to commit() will be processed promptly.
> Two possible solutions to this:
> 1. Make sure a lock is not held during the actual select call. Since we have 
> multiple layers (KafkaConsumer -> NetworkClient -> Selector -> nio Selector) 
> this is probably hard to make work cleanly since locking is currently only 
> performed at the KafkaConsumer level and we'd want it unlocked around a 
> single line of code in Selector.
> 2. Wake up the selector before synchronizing for certain operations. This 
> would require some additional coordination to make sure the caller of 
> wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() 
> thread being woken up and then promptly reacquiring the lock with a 
> subsequent long poll() call).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34554: Patch for KAFKA-2205

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/
---

(Updated May 21, 2015, 5:55 p.m.)


Review request for kafka.


Bugs: KAFKA-2205
https://issues.apache.org/jira/browse/KAFKA-2205


Repository: kafka


Description (updated)
---

KAFKA-2205. Summary of changes:

1. Generalized TopicConfigManager to DynamicConfigManager. It is now able to 
handle multiple types of entities.
2. Changed format of the notification znode as described in KIP-21
3. Replaced TopicConfigManager with DynamicConfigManager.
4. Added new testcases. Existing testcases all pass
5. Added ConfigCommand to handle all config changes. Eventually this will make 
calls to the broker once the new API's are built for now it speaks to ZK 
directly


Diffs
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
f06edf41c732a7b794e496d0048b0ce6f897e72b 
  core/src/main/scala/kafka/admin/ConfigCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
8e6f18633b25bf1beee3f813b28ef7aa7d779d7b 
  core/src/main/scala/kafka/cluster/Partition.scala 
730a232482fdf77be5704cdf5941cfab3828db88 
  core/src/main/scala/kafka/controller/KafkaController.scala 
69bba243a9a511cc5292b43da0cc48e421a428b0 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
3b15ab4eef22c6f50a7483e99a6af40fb55aca9f 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
64ecb499f24bc801d48f86e1612d927cc08e006d 
  core/src/main/scala/kafka/server/ConfigHandler.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
ea6d165d8e5c3146d2c65e8ad1a513308334bf6f 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
b675a7e45ea4f4179f8b15fe221fd988aff13aa0 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
2618dd39b925b979ad6e4c0abd5c6eaafb3db5d5 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
efb2f8e79b3faef78722774b951fea828cd50374 
  core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala 
c7136f20972614ac47aa57ab13e3c94ef775a4b7 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
7877f6ca1845c2edbf96d4a9783a07a552db8f07 

Diff: https://reviews.apache.org/r/34554/diff/


Testing (updated)
---

1. Added new testcases for new code.
2. Verified that both topic and client configs can be changed dynamically by 
starting a local cluster


Thanks,

Aditya Auradkar

[jira] [Commented] (KAFKA-2205) Generalize TopicConfigManager to handle multiple entity configs

[Aditya A Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554734#comment-14554734
 ] 

Aditya A Auradkar commented on KAFKA-2205:
--

Created reviewboard https://reviews.apache.org/r/34554/diff/
 against branch origin/trunk

> Generalize TopicConfigManager to handle multiple entity configs
> ---
>
> Key: KAFKA-2205
> URL: https://issues.apache.org/jira/browse/KAFKA-2205
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
> Attachments: KAFKA-2205.patch
>
>
> Acceptance Criteria:
> - TopicConfigManager should be generalized to handle Topic and Client configs 
> (and any type of config in the future). As described in KIP-21
> - Add a ConfigCommand tool to change topic and client configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2205) Generalize TopicConfigManager to handle multiple entity configs

[Aditya A Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya A Auradkar updated KAFKA-2205:
-
Attachment: KAFKA-2205.patch

> Generalize TopicConfigManager to handle multiple entity configs
> ---
>
> Key: KAFKA-2205
> URL: https://issues.apache.org/jira/browse/KAFKA-2205
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
> Attachments: KAFKA-2205.patch
>
>
> Acceptance Criteria:
> - TopicConfigManager should be generalized to handle Topic and Client configs 
> (and any type of config in the future). As described in KIP-21
> - Add a ConfigCommand tool to change topic and client configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-21 Dynamic Configuration

[Jun Rao]

What about TopicMetadataResponse in KIP-4? Do we remove the config part in
it?

Thanks,

Jun

On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hey Jun,
>
> I've added a section on error codes on the KIP-21 wiki.
>
> Here are the proposed changes to KIP-4. I'll update the wiki shortly.
> - Change AlterTopic to not allow setting configs. Config changes will flow
> through AlterConfig. CreateTopic will still allow setting configs as it is
> nice to be able to specify configs while creating the topic.
> - Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig"
> as proposed in KIP-21.
>
>
> Thanks,
> Aditya
>
> 
> From: Jun Rao [j...@confluent.io]
> Sent: Thursday, May 21, 2015 8:41 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Aditya,
>
> For completeness, could you list the set of error codes in the wiki? Also,
> could you summarize the changes that are needed for the requests listed in
> KIP-4 and update the wiki accordingly?
>
> Thanks,
>
> Jun
>
> On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Thanks Andrii. I'll make the changes.
> >
> > I've also updated KIP-21 to include the new config requests. Take a look
> > and vote.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Aditya
> > 
> > From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> > Sent: Tuesday, May 19, 2015 2:26 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hi,
> >
> > Sorry I wasn't able to participate. I don't have objections about
> removing
> > config changes from AlterTopic (as I understand both AddedConfig and
> > DeletedConfig) - you are welcome to update the KIP page.
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Updating the discussion with the latest comments.
> > >
> > > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig).
> I'll
> > > update KIP-21 with details on these.
> > > 2. Discussed during the KIP hangout. We are in agreement.
> > >
> > > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> > KIP
> > > can be implemented independently. Any concerns if we tackle it as two
> > > separate work items implementation wise?
> > >
> > > We also discussed changing the AlterTopic command in KIP-4 to not
> include
> > > config changes. Instead, all config changes will pass through the newly
> > > proposed AlterConfig. If no-one objects, I can make some changes to
> KIP-4
> > > to reflect this.
> > >
> > > Aditya
> > >
> > > 
> > > From: Jay Kreps [jay.kr...@gmail.com]
> > > Sent: Tuesday, May 19, 2015 10:51 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Hey Aditya,
> > >
> > > Two comments:
> > >
> > > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think it
> does
> > > make sense to allow setting config during topic creation. I agree with
> > your
> > > summary that having alter topic and alter config may be confusing, but
> > > there are also some non-config changes such as replication factor and
> > > partition count that alter topic can carry out. What is the final state
> > you
> > > are proposing?
> > >
> > > 2. This is implementation related so probably can be removed from the
> KIP
> > > entirely, but you seem to be proposing a separate config manager for
> each
> > > config override type. Should we just generalize TopicConfigManager to
> be
> > > ConfigOverrideManager and have it handle all the override types we will
> > > have? I think I may just be unclear on what you are proposing...
> > >
> > > -Jay
> > >
> > > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar <
> > > aaurad...@linkedin.com.invalid> wrote:
> > >
> > > > Yeah, that was just a typo. I've fixed it. Thanks for calling it out.
> > > >
> > > > In KIP-4, I believe we have 3 types of requests: CreateTopic,
> > AlterTopic
> > > > and DeleteTopic. The topic configs are a sub-type of the Create and
> > Alter
> > > > commands. I think it would be nice to simply have a AlterConfig
> command
> > > > that can alter any type of config rather than having a specific
> > > > ClientConfig.
> > > >
> > > > AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]]
> > > > ConfigType => string
> > > > AddedConfigEntry => ConfigKey ConfigValue
> > > > ConfigKey => string
> > > > ConfigValue => string
> > > > DeletedConfig => string
> > > >
> > > > The downside of this approach is that we will have 2 separate ways of
> > > > changing topic configs (AlterTopic and AlterConfig). While a general
> > > > AlterConfig only makes sense if we plan to have more than two types
> of
> > > > entity configs.. it's d

Review Request 34554: Patch for KAFKA-2205

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/
---

Review request for kafka.


Bugs: KAFKA-2205
https://issues.apache.org/jira/browse/KAFKA-2205


Repository: kafka


Description
---

Some fixes


KAFKA-2205


KAFKA-2205


Diffs
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
f06edf41c732a7b794e496d0048b0ce6f897e72b 
  core/src/main/scala/kafka/admin/ConfigCommand.scala PRE-CREATION 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
8e6f18633b25bf1beee3f813b28ef7aa7d779d7b 
  core/src/main/scala/kafka/cluster/Partition.scala 
730a232482fdf77be5704cdf5941cfab3828db88 
  core/src/main/scala/kafka/controller/KafkaController.scala 
69bba243a9a511cc5292b43da0cc48e421a428b0 
  core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala 
3b15ab4eef22c6f50a7483e99a6af40fb55aca9f 
  core/src/main/scala/kafka/controller/TopicDeletionManager.scala 
64ecb499f24bc801d48f86e1612d927cc08e006d 
  core/src/main/scala/kafka/server/ConfigHandler.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
ea6d165d8e5c3146d2c65e8ad1a513308334bf6f 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
b675a7e45ea4f4179f8b15fe221fd988aff13aa0 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
2618dd39b925b979ad6e4c0abd5c6eaafb3db5d5 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
efb2f8e79b3faef78722774b951fea828cd50374 
  core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala 
c7136f20972614ac47aa57ab13e3c94ef775a4b7 
  core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 
7877f6ca1845c2edbf96d4a9783a07a552db8f07 

Diff: https://reviews.apache.org/r/34554/diff/


Testing
---


Thanks,

Aditya Auradkar

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554724#comment-14554724
 ] 

Jason Gustafson commented on KAFKA-2168:


I feel a little wary about finer-grained synchronization given all the state in 
the consumer, the network client, and the selector. I actually think the 
two-lock approach is the least intrusive since it only touches the 
KafkaConsumer and preserves the current coarse synchronization design, but I 
agree that it's unusual. Here's an idea of what it might look like in the code:

{code:java}
lock.queue();
try {
  client.wakeup();
  lock.lock();

  // critical section
} finally {
  lock.unlock()
}
{code}

Definitely weird, but not that hard to understand. You'd still run into the 
same problem if multiple threads are trying to poll, but that seems like 
unintended usage anyway.

> New consumer poll() can block other calls like position(), commit(), and 
> close() indefinitely
> -
>
> Key: KAFKA-2168
> URL: https://issues.apache.org/jira/browse/KAFKA-2168
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The new consumer is currently using very coarse-grained synchronization. For 
> most methods this isn't a problem since they finish quickly once the lock is 
> acquired, but poll() might run for a long time (and commonly will since 
> polling with long timeouts is a normal use case). This means any operations 
> invoked from another thread may block until the poll() call completes.
> Some example use cases where this can be a problem:
> * A shutdown hook is registered to trigger shutdown and invokes close(). It 
> gets invoked from another thread and blocks indefinitely.
> * User wants to manage offset commit themselves in a background thread. If 
> the commit policy is not purely time based, it's not currently possibly to 
> make sure the call to commit() will be processed promptly.
> Two possible solutions to this:
> 1. Make sure a lock is not held during the actual select call. Since we have 
> multiple layers (KafkaConsumer -> NetworkClient -> Selector -> nio Selector) 
> this is probably hard to make work cleanly since locking is currently only 
> performed at the KafkaConsumer level and we'd want it unlocked around a 
> single line of code in Selector.
> 2. Wake up the selector before synchronizing for certain operations. This 
> would require some additional coordination to make sure the caller of 
> wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() 
> thread being woken up and then promptly reacquiring the lock with a 
> subsequent long poll() call).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Vote] KIP-11 Authorization design for kafka security

[Gwen Shapira]

The KIP and design were accepted, so the WIKI should say "accepted" or
something similar.
Specific patch status is reflected in the JIRA.

On Thu, May 21, 2015 at 8:37 PM, Parth Brahmbhatt <
pbrahmbh...@hortonworks.com> wrote:

> I am sorry to be ignorant about this but what is the new state? Adopted
> seems too early given we are still in code review process. Should I just
> make it ³Code review²?
>
> Thanks
> Parth
>
> On 5/21/15, 8:43 AM, "Jun Rao"  wrote:
>
> >Parth,
> >
> >Thanks for driving this. Could you update the status of the KIP in the
> >wiki?
> >
> >Thanks,
> >
> >Jun
> >
> >On Wed, May 20, 2015 at 2:37 PM, Parth Brahmbhatt <
> >pbrahmbh...@hortonworks.com> wrote:
> >
> >> This vote is now Closed with 4 binding +1s and 4 non binding +1s.
> >>
> >> Thanks
> >> Parth
> >>
> >> On 5/20/15, 12:04 PM, "Joel Koshy"  wrote:
> >>
> >> >+1
> >> >
> >> >On Fri, May 15, 2015 at 04:18:49PM +, Parth Brahmbhatt wrote:
> >> >> Hi,
> >> >>
> >> >> Opening the voting thread for KIP-11.
> >> >>
> >> >> Link to the KIP:
> >> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
> >> >>Interface
> >> >> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >> >>
> >> >> Thanks
> >> >> Parth
> >> >
> >>
> >>
>
>

[jira] [Updated] (KAFKA-1690) new java producer needs ssl support as a client

[Sriharsha Chintalapani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani updated KAFKA-1690:
--
Attachment: KAFKA-1690_2015-05-21_10:37:08.patch

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554714#comment-14554714
 ] 

Sriharsha Chintalapani commented on KAFKA-1690:
---

Updated reviewboard https://reviews.apache.org/r/33620/diff/
 against branch origin/trunk

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Vote] KIP-11 Authorization design for kafka security

[Parth Brahmbhatt]

I am sorry to be ignorant about this but what is the new state? Adopted
seems too early given we are still in code review process. Should I just
make it ³Code review²?

Thanks
Parth

On 5/21/15, 8:43 AM, "Jun Rao"  wrote:

>Parth,
>
>Thanks for driving this. Could you update the status of the KIP in the
>wiki?
>
>Thanks,
>
>Jun
>
>On Wed, May 20, 2015 at 2:37 PM, Parth Brahmbhatt <
>pbrahmbh...@hortonworks.com> wrote:
>
>> This vote is now Closed with 4 binding +1s and 4 non binding +1s.
>>
>> Thanks
>> Parth
>>
>> On 5/20/15, 12:04 PM, "Joel Koshy"  wrote:
>>
>> >+1
>> >
>> >On Fri, May 15, 2015 at 04:18:49PM +, Parth Brahmbhatt wrote:
>> >> Hi,
>> >>
>> >> Opening the voting thread for KIP-11.
>> >>
>> >> Link to the KIP:
>> >>
>> 
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
>> >>Interface
>> >> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> >>
>> >> Thanks
>> >> Parth
>> >
>>
>>

Re: Review Request 33620: Patch for KAFKA-1690

[Sriharsha Chintalapani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/
---

(Updated May 21, 2015, 5:37 p.m.)


Review request for kafka.


Bugs: KAFKA-1690
https://issues.apache.org/jira/browse/KAFKA-1690


Repository: kafka


Description (updated)
---

KAFKA-1690. new java producer needs ssl support as a client.


KAFKA-1690. new java producer needs ssl support as a client.


KAFKA-1690. new java producer needs ssl support as a client.


KAFKA-1690. new java producer needs ssl support as a client. SSLFactory tests.


KAFKA-1690. new java producer needs ssl support as a client. Added 
PrincipalBuilder.


KAFKA-1690. new java producer needs ssl support as a client. Addressing reviews.


KAFKA-1690. new java producer needs ssl support as a client. Addressing reviews.


KAFKA-1690. new java producer needs ssl support as a client. Addressing reviews.


KAFKA-1690. new java producer needs ssl support as a client. Fixed minor issues 
with the patch.


KAFKA-1690. new java producer needs ssl support as a client. Fixed minor issues 
with the patch.


Diffs (updated)
-

  build.gradle cd2aa838fd53e8124f308979b1d70efe0c5725a6 
  checkstyle/import-control.xml f2e6cec267e67ce8e261341e373718e14a8e8e03 
  clients/src/main/java/org/apache/kafka/clients/ClientUtils.java 
0d68bf1e1e90fe9d5d4397ddf817b9a9af8d9f7a 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
bdff518b732105823058e6182f445248b45dc388 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
d301be4709f7b112e1f3a39f3c04cfa65f00fa60 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
8e336a3aa96c73f52beaeb56b931baf4b026cf21 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
187d0004c8c46b6664ddaffecc6166d4b47351e5 
  clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java 
c4fa058692f50abb4f47bd344119d805c60123f5 
  clients/src/main/java/org/apache/kafka/common/config/SecurityConfigs.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/network/Authenticator.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/network/Channel.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/network/ChannelBuilder.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/network/DefaultAuthenticator.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/network/PlainTextChannelBuilder.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/network/PlainTextTransportLayer.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/network/SSLChannelBuilder.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/network/SSLFactory.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
b5f8d83e89f9026dc0853e5f92c00b2d7f043e22 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
57de0585e5e9a53eb9dcd99cac1ab3eb2086a302 
  clients/src/main/java/org/apache/kafka/common/network/TransportLayer.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/protocol/SecurityProtocol.java 
dab1a94dd29563688b6ecf4eeb0e180b06049d3f 
  
clients/src/main/java/org/apache/kafka/common/security/auth/DefaultPrincipalBuilder.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/security/auth/PrincipalBuilder.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
f73eedb030987f018d8446bb1dcd98d19fa97331 
  clients/src/test/java/org/apache/kafka/common/network/EchoServer.java 
PRE-CREATION 
  clients/src/test/java/org/apache/kafka/common/network/SSLFactoryTest.java 
PRE-CREATION 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
PRE-CREATION 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
d5b306b026e788b4e5479f3419805aa49ae889f3 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
2ebe3c21f611dc133a2dbb8c7dfb0845f8c21498 
  clients/src/test/java/org/apache/kafka/test/TestSSLUtils.java PRE-CREATION 

Diff: https://reviews.apache.org/r/33620/diff/


Testing
---


Thanks,

Sriharsha Chintalapani

RE: [VOTE] KIP-21 Dynamic Configuration

[Aditya Auradkar]

Hey Jun,

I've added a section on error codes on the KIP-21 wiki.

Here are the proposed changes to KIP-4. I'll update the wiki shortly.
- Change AlterTopic to not allow setting configs. Config changes will flow 
through AlterConfig. CreateTopic will still allow setting configs as it is nice 
to be able to specify configs while creating the topic.
- Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig" as 
proposed in KIP-21. 


Thanks,
Aditya


From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 21, 2015 8:41 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-21 Dynamic Configuration

Aditya,

For completeness, could you list the set of error codes in the wiki? Also,
could you summarize the changes that are needed for the requests listed in
KIP-4 and update the wiki accordingly?

Thanks,

Jun

On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Thanks Andrii. I'll make the changes.
>
> I've also updated KIP-21 to include the new config requests. Take a look
> and vote.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
>
> Aditya
> 
> From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> Sent: Tuesday, May 19, 2015 2:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Hi,
>
> Sorry I wasn't able to participate. I don't have objections about removing
> config changes from AlterTopic (as I understand both AddedConfig and
> DeletedConfig) - you are welcome to update the KIP page.
>
> Thanks,
> Andrii Biletskyi
>
> On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Updating the discussion with the latest comments.
> >
> > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig). I'll
> > update KIP-21 with details on these.
> > 2. Discussed during the KIP hangout. We are in agreement.
> >
> > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> KIP
> > can be implemented independently. Any concerns if we tackle it as two
> > separate work items implementation wise?
> >
> > We also discussed changing the AlterTopic command in KIP-4 to not include
> > config changes. Instead, all config changes will pass through the newly
> > proposed AlterConfig. If no-one objects, I can make some changes to KIP-4
> > to reflect this.
> >
> > Aditya
> >
> > 
> > From: Jay Kreps [jay.kr...@gmail.com]
> > Sent: Tuesday, May 19, 2015 10:51 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hey Aditya,
> >
> > Two comments:
> >
> > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think it does
> > make sense to allow setting config during topic creation. I agree with
> your
> > summary that having alter topic and alter config may be confusing, but
> > there are also some non-config changes such as replication factor and
> > partition count that alter topic can carry out. What is the final state
> you
> > are proposing?
> >
> > 2. This is implementation related so probably can be removed from the KIP
> > entirely, but you seem to be proposing a separate config manager for each
> > config override type. Should we just generalize TopicConfigManager to be
> > ConfigOverrideManager and have it handle all the override types we will
> > have? I think I may just be unclear on what you are proposing...
> >
> > -Jay
> >
> > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Yeah, that was just a typo. I've fixed it. Thanks for calling it out.
> > >
> > > In KIP-4, I believe we have 3 types of requests: CreateTopic,
> AlterTopic
> > > and DeleteTopic. The topic configs are a sub-type of the Create and
> Alter
> > > commands. I think it would be nice to simply have a AlterConfig command
> > > that can alter any type of config rather than having a specific
> > > ClientConfig.
> > >
> > > AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]]
> > > ConfigType => string
> > > AddedConfigEntry => ConfigKey ConfigValue
> > > ConfigKey => string
> > > ConfigValue => string
> > > DeletedConfig => string
> > >
> > > The downside of this approach is that we will have 2 separate ways of
> > > changing topic configs (AlterTopic and AlterConfig). While a general
> > > AlterConfig only makes sense if we plan to have more than two types of
> > > entity configs.. it's definitely more future proof. Thoughts?
> > >
> > > Aditya
> > >
> > > 
> > > From: Todd Palino [tpal...@gmail.com]
> > > Sent: Monday, May 18, 2015 12:39 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Agree with Jun here on the JSON format. I think your intention was
> likely
> > > to have actual JSON here and it was just a typo in the wiki?
> > >
> > > -

Re: Review Request 33065: Patch for KAFKA-1928

[Jun Rao]

> On May 21, 2015, 2:11 a.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/network/RequestChannel.scala, line 110
> > 
> >
> > This would result in the following log entry:
> > 
> > TRACE Completed 
> > request:{api_key=12,api_version=0,correlation_id=40,client_id=consumer-1} : 
> > {group_id=my-test,group_generation_id=1,consumer_id=e6c857b3-e362-4392-a272-94aa3e42d1fb}
> >  ...
> > 
> > which is not very intuitive. Could we add a toString function to the 
> > requests as we did in the old classes?
> 
> Gwen Shapira wrote:
> This is a bit out of scope for this patch. Can you open a new JIRA? I'll 
> be happy to add this.

Yes, I am actually not sure if we need to stick to the old string format of the 
request. The new string format looks different, but conveys the same 
information. The benefit is that we don't have to customize the string format 
in every request. It may be useful to represent api_key with a string (e.g., 
produce, fetch), but that can be done in a generic way and can be done in a 
separate jira.


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33065/#review84649
---


On May 20, 2015, 10:42 a.m., Gwen Shapira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33065/
> ---
> 
> (Updated May 20, 2015, 10:42 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: 1928 and KAFKA-1928
> https://issues.apache.org/jira/browse/1928
> https://issues.apache.org/jira/browse/KAFKA-1928
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> first pass on replacing Send
> 
> 
> implement maxSize and improved docs
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> Conflicts:
>   core/src/main/scala/kafka/network/RequestChannel.scala
> 
> moved selector out of abstract thread
> 
> 
> mid-way through putting selector in SocketServer
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> Also, SocketServer is now using Selector. Stil a bit messy - but all tests 
> pass.
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> renamed requestKey to connectionId to reflect new use and changed type from 
> Any to String
> 
> 
> Following Jun's comments - moved MultiSend to client. Cleaned up destinations 
> as well
> 
> 
> removed reify and remaining from send/recieve API, per Jun. moved 
> maybeCloseOldest() to Selector per Jay
> 
> 
> added idString to node API, changed written to int in Send API
> 
> 
> cleaning up MultiSend, added size() to Send interface
> 
> 
> fixed some issues with multisend
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> fixed metric thingies
> 
> 
> fixed response order bug
> 
> 
> error handling for illegal selector state and fix metrics bug
> 
> 
> optimized selection key lookup with identity hash
> 
> 
> fix accidental change
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> addressing Jun's comments
> 
> 
> removed connection-aging for clients
> 
> 
> fix issues with exception handling and other cleanup
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> Revert "removed connection-aging for clients"
> 
> This reverts commit 016669123a370b561b5ac78f8f1cf7bdd958e7d1.
> 
> improving exception handling and other minor fixes
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java 
> da76cc257b4cfe3c4bce7120a1f14c7f31ef8587 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 936487b16e7ac566f8bdcd39a7240ceb619fd30e 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 1311f85847b022efec8cb05c450bb18231db6979 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 435fbb5116e80302eba11ed1d3069cb577dbdcbd 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> bdff518b732105823058e6182f445248b45dc388 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> d301be4709f7b112e1f3a39f3c04cfa65f00fa60 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java
>  b2764df11afa7a99fce46d1ff48960d889032d14 
>   
> clients/src/main/java/org/apa

Kafka server validating incoming messages

[Grant Henke]

When working on my own implementation of the wire protocol, via a copy and
paste error, I accidentally sent an OffsetCommit message to the
ConsumerMetadata api. This was clearly my mistake but what surprised me is
that I got a valid ConsumerMetadata response back with no error.

Digging a little deeper this is because both messages expect a string in
the first position, so when the OffsetCommit (much larger) message is
parsed as a ConsumerMetadata message everything parses okay on the server
side. The only issue is that there are many bytes left over in the sent
message since OffsetCommit has many more fields following the initial
string.

Knowing that this is client/dev error, I still expected Kafka to fail my
message based on expected message size. If Kafka parsing messages would be
strict about "left over" bytes it could catch this. Should this check be
implemented in the Kafka server when parsing all messages? If so I can open
a Jira.

Thank you,
Grant

-- 
Grant Henke
Solutions Consultant | Cloudera
ghe...@cloudera.com | twitter.com/ghenke  |
linkedin.com/in/granthenke

Find all consumer groups without Zookeeper

[Grant Henke]

It doesn't look like there is a way to get all consumer groups via the wire
protocol/api. I can go to zookeeper and list all the children at
'/consumers' but would like to use zookeeper as minimally as possible.
Would this be a good feature to add the the ConsumerMetadataRequest? Am I
missing/misusing something here?

My initial thought was modeling it to parody the TopicMetadataRequest.
Where it takes a list of consumer groups, and if you leave it empty it will
return metadata for all. I would find this functionality useful for
monitoring/reporting tools and to be sure I am creating a new consumer
group instead of joining an existing one.

Note: I just made Jira KAFKA-2216
, which is somewhat
related to this discussion.

Thank you,
Grant

-- 
Grant Henke
Solutions Consultant | Cloudera
ghe...@cloudera.com | twitter.com/ghenke  |
linkedin.com/in/granthenke

Re: Review Request 33065: Patch for KAFKA-1928

[Gwen Shapira]

> On May 21, 2015, 2:11 a.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/network/RequestChannel.scala, line 110
> > 
> >
> > This would result in the following log entry:
> > 
> > TRACE Completed 
> > request:{api_key=12,api_version=0,correlation_id=40,client_id=consumer-1} : 
> > {group_id=my-test,group_generation_id=1,consumer_id=e6c857b3-e362-4392-a272-94aa3e42d1fb}
> >  ...
> > 
> > which is not very intuitive. Could we add a toString function to the 
> > requests as we did in the old classes?

This is a bit out of scope for this patch. Can you open a new JIRA? I'll be 
happy to add this.


- Gwen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33065/#review84649
---


On May 20, 2015, 10:42 a.m., Gwen Shapira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33065/
> ---
> 
> (Updated May 20, 2015, 10:42 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: 1928 and KAFKA-1928
> https://issues.apache.org/jira/browse/1928
> https://issues.apache.org/jira/browse/KAFKA-1928
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> first pass on replacing Send
> 
> 
> implement maxSize and improved docs
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> Conflicts:
>   core/src/main/scala/kafka/network/RequestChannel.scala
> 
> moved selector out of abstract thread
> 
> 
> mid-way through putting selector in SocketServer
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> Also, SocketServer is now using Selector. Stil a bit messy - but all tests 
> pass.
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> renamed requestKey to connectionId to reflect new use and changed type from 
> Any to String
> 
> 
> Following Jun's comments - moved MultiSend to client. Cleaned up destinations 
> as well
> 
> 
> removed reify and remaining from send/recieve API, per Jun. moved 
> maybeCloseOldest() to Selector per Jay
> 
> 
> added idString to node API, changed written to int in Send API
> 
> 
> cleaning up MultiSend, added size() to Send interface
> 
> 
> fixed some issues with multisend
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> fixed metric thingies
> 
> 
> fixed response order bug
> 
> 
> error handling for illegal selector state and fix metrics bug
> 
> 
> optimized selection key lookup with identity hash
> 
> 
> fix accidental change
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> addressing Jun's comments
> 
> 
> removed connection-aging for clients
> 
> 
> fix issues with exception handling and other cleanup
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> Revert "removed connection-aging for clients"
> 
> This reverts commit 016669123a370b561b5ac78f8f1cf7bdd958e7d1.
> 
> improving exception handling and other minor fixes
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
> KAFKA-1928-v2
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java 
> da76cc257b4cfe3c4bce7120a1f14c7f31ef8587 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 936487b16e7ac566f8bdcd39a7240ceb619fd30e 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 1311f85847b022efec8cb05c450bb18231db6979 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 435fbb5116e80302eba11ed1d3069cb577dbdcbd 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> bdff518b732105823058e6182f445248b45dc388 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> d301be4709f7b112e1f3a39f3c04cfa65f00fa60 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java
>  b2764df11afa7a99fce46d1ff48960d889032d14 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
>  ef9dd5238fbc771496029866ece1d85db6d7b7a5 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 8e336a3aa96c73f52beaeb56b931baf4b026cf21 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 187d0004c8c46b6664ddaffecc6166d4b47351e5 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 

[jira] [Created] (KAFKA-2216) ConsumerMetadataRequest does not validate consumer group exists

[Grant Henke (JIRA)]

Grant Henke created KAFKA-2216:
--

 Summary: ConsumerMetadataRequest does not validate consumer group 
exists
 Key: KAFKA-2216
 URL: https://issues.apache.org/jira/browse/KAFKA-2216
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Grant Henke
Assignee: Grant Henke


Currently the offset partition is found by hashing & modding the name:
{quote}
   // Utils.abs(group.hashCode) % config.offsetsTopicNumPartitions
   val partition = offsetManager.partitionFor(consumerMetadataRequest.group)
{quote}
And then the broker for the partition is returned as the coordinator broker. 

I expected an error to be returned for a group that does not exist. If this is 
not the expectation, then updating the docs may be a result of this Jira.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

[Jun Rao]

Honghai,

Could you update the wiki on the preallocated size? Instead of
config.segmentSize
- 2 * config.maxMessageSize, we just want to use config.segmentSize.

Thanks,

Jun

On Mon, May 11, 2015 at 8:16 PM, Honghai Chen 
wrote:

> All issues fixed, test cases added, performance result on windows
> attached.  The patch can help improve the consume performance around
> 25%~50%.
>
> Thanks, Honghai Chen
>
> -Original Message-
> From: Jun Rao [mailto:j...@confluent.io]
> Sent: Wednesday, May 6, 2015 5:39 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
> performance under windows and some old Linux file system
>
> Thanks. Could you updated the wiki? Also, commented on the jira.
>
> Jun
>
> On Tue, May 5, 2015 at 12:48 AM, Honghai Chen 
> wrote:
>
> > Use config.segmentSize should be ok.   Previously add that one for make
> > sure the file not exceed config.segmentSize, actually the function
> > maybeRoll already make sure that.
> > When try add test case for recover, blocked by the rename related
> > issue, just open one jira at
> > https://issues.apache.org/jira/browse/KAFKA-2170 , any recommendation
> for fix that issue?
> >
> > Thanks, Honghai Chen
> >
> > -Original Message-
> > From: Jun Rao [mailto:j...@confluent.io]
> > Sent: Tuesday, May 5, 2015 12:51 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve
> > consume performance under windows and some old Linux file system
> >
> > This seems similar to what's in
> > https://issues.apache.org/jira/browse/KAFKA-1065.
> >
> > Also, could you explain why the preallocated size is set to
> > config.segmentSize
> > - 2 * config.maxMessageSize, instead of just config.segmentSize?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, May 4, 2015 at 8:12 PM, Honghai Chen
> > 
> > wrote:
> >
> > >   Hi guys,
> > > I'm trying add test cases, but below case crashed at line "
> > > segReopen.recover(64*1024)--> index.trimToValidSize()  ", any idea
> > > for
> > it?
> > > Appreciate your help.
> > > The case assume kafka suddenly crash, and need recover the
> > > last segment.
> > >
> > > kafka.log.LogSegmentTest > testCreateWithInitFileSizeCrash FAILED
> > > java.io.IOException: The requested operation cannot be performed
> > > on a file w ith a user-mapped section open
> > > at java.io.RandomAccessFile.setLength(Native Method)
> > > at
> > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:292)
> > > at
> > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
> > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:288)
> > > at kafka.log.OffsetIndex.resize(OffsetIndex.scala:283)
> > > at
> > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(Offset
> > > I
> > > ndex.scala:272)
> > > at
> > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.s
> > > c
> > > ala:272)
> > > at
> > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.s
> > > c
> > > ala:272)
> > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:288)
> > > at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:271)
> > > at kafka.log.LogSegment.recover(LogSegment.scala:199)
> > > at
> > > kafka.log.LogSegmentTest.testCreateWithInitFileSizeCrash(LogSegmentT
> > > e
> > > st.scala:306)
> > >
> > >   def recover(maxMessageSize: Int): Int = {
> > > index.truncate()
> > > index.resize(index.maxIndexSize)
> > > var validBytes = 0
> > > var lastIndexEntry = 0
> > > val iter = log.iterator(maxMessageSize)
> > > try {
> > >   while(iter.hasNext) {
> > > val entry = iter.next
> > > entry.message.ensureValid()
> > > if(validBytes - lastIndexEntry > indexIntervalBytes) {
> > >   // we need to decompress the message, if required, to get
> > > the offset of the first uncompressed message
> > >   val startOffset =
> > > entry.message.compressionCodec match {
> > >   case NoCompressionCodec =>
> > > entry.offset
> > >   case _ =>
> > >
> > > ByteBufferMessageSet.deepIterator(entry.message).next().offset
> > >   }
> > >   index.append(startOffset, validBytes)
> > >   lastIndexEntry = validBytes
> > > }
> > > validBytes += MessageSet.entrySize(entry.message)
> > >   }
> > > } catch {
> > >   case e: InvalidMessageException =>
> > > logger.warn("Found invalid messages in log segment %s at
> > > byte offset %d: %s.".format(log.file.getAbsolutePath, validBytes,
> > e.getMessage))
> > > }
> > > val truncated = log.sizeInBytes - validBytes
> > > log.truncateTo(validBytes)
> > > index.trimToValidSize()
> > > truncated
> > >   }
> > >
> > > /* create a segment with   pre allocate and Crash*/
> > >   @Test
> > >   def testCreateWit

Re: [Vote] KIP-11 Authorization design for kafka security

[Jun Rao]

Parth,

Thanks for driving this. Could you update the status of the KIP in the wiki?

Thanks,

Jun

On Wed, May 20, 2015 at 2:37 PM, Parth Brahmbhatt <
pbrahmbh...@hortonworks.com> wrote:

> This vote is now Closed with 4 binding +1s and 4 non binding +1s.
>
> Thanks
> Parth
>
> On 5/20/15, 12:04 PM, "Joel Koshy"  wrote:
>
> >+1
> >
> >On Fri, May 15, 2015 at 04:18:49PM +, Parth Brahmbhatt wrote:
> >> Hi,
> >>
> >> Opening the voting thread for KIP-11.
> >>
> >> Link to the KIP:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
> >>Interface
> >> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >>
> >> Thanks
> >> Parth
> >
>
>

Re: [VOTE] KIP-21 Dynamic Configuration

[Jun Rao]

Aditya,

For completeness, could you list the set of error codes in the wiki? Also,
could you summarize the changes that are needed for the requests listed in
KIP-4 and update the wiki accordingly?

Thanks,

Jun

On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Thanks Andrii. I'll make the changes.
>
> I've also updated KIP-21 to include the new config requests. Take a look
> and vote.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
>
> Aditya
> 
> From: Andrii Biletskyi [andrii.bilets...@stealth.ly]
> Sent: Tuesday, May 19, 2015 2:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-21 Dynamic Configuration
>
> Hi,
>
> Sorry I wasn't able to participate. I don't have objections about removing
> config changes from AlterTopic (as I understand both AddedConfig and
> DeletedConfig) - you are welcome to update the KIP page.
>
> Thanks,
> Andrii Biletskyi
>
> On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Updating the discussion with the latest comments.
> >
> > 1. We discussed adding 2 new API's (AlterConfig and DescribeConfig). I'll
> > update KIP-21 with details on these.
> > 2. Discussed during the KIP hangout. We are in agreement.
> >
> > (1) has a dependency on KIP-4 being completed. Rest of the work in the
> KIP
> > can be implemented independently. Any concerns if we tackle it as two
> > separate work items implementation wise?
> >
> > We also discussed changing the AlterTopic command in KIP-4 to not include
> > config changes. Instead, all config changes will pass through the newly
> > proposed AlterConfig. If no-one objects, I can make some changes to KIP-4
> > to reflect this.
> >
> > Aditya
> >
> > 
> > From: Jay Kreps [jay.kr...@gmail.com]
> > Sent: Tuesday, May 19, 2015 10:51 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> >
> > Hey Aditya,
> >
> > Two comments:
> >
> > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think it does
> > make sense to allow setting config during topic creation. I agree with
> your
> > summary that having alter topic and alter config may be confusing, but
> > there are also some non-config changes such as replication factor and
> > partition count that alter topic can carry out. What is the final state
> you
> > are proposing?
> >
> > 2. This is implementation related so probably can be removed from the KIP
> > entirely, but you seem to be proposing a separate config manager for each
> > config override type. Should we just generalize TopicConfigManager to be
> > ConfigOverrideManager and have it handle all the override types we will
> > have? I think I may just be unclear on what you are proposing...
> >
> > -Jay
> >
> > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Yeah, that was just a typo. I've fixed it. Thanks for calling it out.
> > >
> > > In KIP-4, I believe we have 3 types of requests: CreateTopic,
> AlterTopic
> > > and DeleteTopic. The topic configs are a sub-type of the Create and
> Alter
> > > commands. I think it would be nice to simply have a AlterConfig command
> > > that can alter any type of config rather than having a specific
> > > ClientConfig.
> > >
> > > AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]]
> > > ConfigType => string
> > > AddedConfigEntry => ConfigKey ConfigValue
> > > ConfigKey => string
> > > ConfigValue => string
> > > DeletedConfig => string
> > >
> > > The downside of this approach is that we will have 2 separate ways of
> > > changing topic configs (AlterTopic and AlterConfig). While a general
> > > AlterConfig only makes sense if we plan to have more than two types of
> > > entity configs.. it's definitely more future proof. Thoughts?
> > >
> > > Aditya
> > >
> > > 
> > > From: Todd Palino [tpal...@gmail.com]
> > > Sent: Monday, May 18, 2015 12:39 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration
> > >
> > > Agree with Jun here on the JSON format. I think your intention was
> likely
> > > to have actual JSON here and it was just a typo in the wiki?
> > >
> > > -Todd
> > >
> > > On Mon, May 18, 2015 at 12:07 PM, Jun Rao  wrote:
> > >
> > > > Aditya,
> > > >
> > > > Another thing to consider. In KIP-4, we are adding a new RPC request
> to
> > > > change and retrieve topic configs. Do we want to add a similar RPC
> > > request
> > > > to change configs per client id? If so, do we want to introduce a
> > > separate
> > > > new request or have a combined new request for both topic and client
> id
> > > > level config changes?
> > > >
> > > > A minor point in the wiki, for the json format in ZK, we should
> change
> > > > {X1=Y1,
> > > > X2=Y2..} to a json map, right?
> > > >
> > > > Thanks,
> > > >
> > > > 

[jira] [Commented] (KAFKA-1778) Create new re-elect controller admin function

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554515#comment-14554515
 ] 

Jun Rao commented on KAFKA-1778:


[~anigam], for the two use cases that you mentioned, it seems that a simpler 
approach is what Joe said. Just configure a couple of brokers to be eligible 
for becoming the controller. Then, only those brokers will try to become the 
controller. I am not sure what the use case is to have an admin command to 
force the controller to move.

> Create new re-elect controller admin function
> -
>
> Key: KAFKA-1778
> URL: https://issues.apache.org/jira/browse/KAFKA-1778
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Abhishek Nigam
> Fix For: 0.8.3
>
>
> kafka --controller --elect



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2215) Improve Randomness for ConsoleConsumer

[Fabian Lange (JIRA)]

Fabian Lange created KAFKA-2215:
---

 Summary: Improve Randomness for ConsoleConsumer
 Key: KAFKA-2215
 URL: https://issues.apache.org/jira/browse/KAFKA-2215
 Project: Kafka
  Issue Type: Bug
Reporter: Fabian Lange


Right now the console consumer does a new Random().nextInt(100_000)
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L123

I would propose to use UUID.randomUUID().toString() instead.

I know this is quite edgy, but Random has shown its quirks from time to time.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2207) The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest fails consistently with the following exception:

[Deepthi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepthi updated KAFKA-2207:
---
Attachment: KAFKA-2207.patch

> The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest 
> fails consistently with the following exception:
> -
>
> Key: KAFKA-2207
> URL: https://issues.apache.org/jira/browse/KAFKA-2207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Deepthi
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-2207.patch
>
>
> kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic FAILED
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 3000 ms.
> at 
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:437)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:352)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248)
> at 
> kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic(ProducerFailureHandlingTest.scala:309)
> Caused by:
> org.apache.kafka.common.errors.TimeoutException: Failed to update 
> metadata after 3000 ms.
> The following attached patch has resolved the issue 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2207) The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest fails consistently with the following exception:

[Deepthi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepthi updated KAFKA-2207:
---
Attachment: (was: KAFKA-2207.patch)

> The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest 
> fails consistently with the following exception:
> -
>
> Key: KAFKA-2207
> URL: https://issues.apache.org/jira/browse/KAFKA-2207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Deepthi
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-2207.patch
>
>
> kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic FAILED
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 3000 ms.
> at 
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:437)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:352)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248)
> at 
> kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic(ProducerFailureHandlingTest.scala:309)
> Caused by:
> org.apache.kafka.common.errors.TimeoutException: Failed to update 
> metadata after 3000 ms.
> The following attached patch has resolved the issue 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2207) The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest fails consistently with the following exception:

[Deepthi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepthi updated KAFKA-2207:
---
Attachment: KAFKA-2207.patch

> The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest 
> fails consistently with the following exception:
> -
>
> Key: KAFKA-2207
> URL: https://issues.apache.org/jira/browse/KAFKA-2207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Deepthi
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-2207.patch
>
>
> kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic FAILED
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 3000 ms.
> at 
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:437)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:352)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248)
> at 
> kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic(ProducerFailureHandlingTest.scala:309)
> Caused by:
> org.apache.kafka.common.errors.TimeoutException: Failed to update 
> metadata after 3000 ms.
> The following attached patch has resolved the issue 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2207) The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest fails consistently with the following exception:

[Deepthi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepthi updated KAFKA-2207:
---
Attachment: (was: KAFKA-2207.patch)

> The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest 
> fails consistently with the following exception:
> -
>
> Key: KAFKA-2207
> URL: https://issues.apache.org/jira/browse/KAFKA-2207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Deepthi
> Fix For: 0.8.2.1
>
>
> kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic FAILED
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 3000 ms.
> at 
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:437)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:352)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248)
> at 
> kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic(ProducerFailureHandlingTest.scala:309)
> Caused by:
> org.apache.kafka.common.errors.TimeoutException: Failed to update 
> metadata after 3000 ms.
> The following attached patch has resolved the issue 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2214) kafka-reassign-partitions.sh --verify should return non-zero exit codes when reassignment is not completed yet

[Michael Noll (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Noll updated KAFKA-2214:

Description: 
h4. Background

The admin script {{kafka-reassign-partitions.sh}} should integrate better with 
automation tools such as Ansible, which rely on scripts adhering to Unix best 
practices such as appropriate exit codes on success/failure.

h4. Current behavior (incorrect)

When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
prints {{ERROR}} messages but returns an exit code of zero, which indicates 
success.  This behavior makes it a bit cumbersome to integrate the script into 
automation tools such as Ansible.

{code}
$ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
--reassignment-json-file partitions-to-move.json --verify

Status of partition reassignment:
ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
reassignment (316,324) for partition [mytopic,2]
Reassignment of partition [mytopic,0] completed successfully
Reassignment of partition [myothertopic,1] completed successfully
Reassignment of partition [myothertopic,3] completed successfully
...

$ echo $?
0

# But preferably the exit code in the presence of ERRORs should be, say, 1.
{code}

h3. How to improve

I'd suggest that, using the above as the running example, if there are any 
{{ERROR}} entries in the output (i.e. if there are any assignments remaining 
that don't match the desired assignments), then the 
{{kafka-reassign-partitions.sh}}  should return a non-zero exit code.

  was:
h4. Background

The admin script {{kafka-reassign-partitions.sh}} should integrate better with 
automation tools such as Ansible, which rely on scripts adhering to Unix best 
practices such as appropriate exit codes on success/failure.

h4. Current behavior (incorrect)

When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
prints {{ERROR}} messages but returns an exit code of zero, which indicates 
success.  This behavior makes it a bit cumbersome to integrate the script into 
automation tools such as Ansible.

{code}
$ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
--reassignment-json-file partitions-to-move.json --verify

Status of partition reassignment:
ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
reassignment (316,324) for partition [mytopic,2]
Reassignment of partition [mytopic,0] completed successfully
Reassignment of partition [myothertopic,1] completed successfully
Reassignment of partition [myothertopic,3] completed successfully
...

$ echo $?
0

# But preferably the exit code would be, say, 1.
{code}

h3. How to improve

I'd suggest that, using the above as the running example, if there are any 
{{ERROR}} entries in the output (i.e. if there are any assignments remaining 
that don't match the desired assignments), then the 
{{kafka-reassign-partitions.sh}}  should return a non-zero exit code.


> kafka-reassign-partitions.sh --verify should return non-zero exit codes when 
> reassignment is not completed yet
> --
>
> Key: KAFKA-2214
> URL: https://issues.apache.org/jira/browse/KAFKA-2214
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.8.1.1
>Reporter: Michael Noll
>Priority: Minor
>
> h4. Background
> The admin script {{kafka-reassign-partitions.sh}} should integrate better 
> with automation tools such as Ansible, which rely on scripts adhering to Unix 
> best practices such as appropriate exit codes on success/failure.
> h4. Current behavior (incorrect)
> When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
> prints {{ERROR}} messages but returns an exit code of zero, which indicates 
> success.  This behavior makes it a bit cumbersome to integrate the script 
> into automation tools such as Ansible.
> {code}
> $ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
> --reassignment-json-file partitions-to-move.json --verify
> Status of partition reassignment:
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> Reassignment of partition [mytopic,0] completed successfully
> Reassignment of partition [myothertopic,1] completed successfully
> Reassignment of partition [myothertopic,3] completed successfully
> ...
> $ echo $?
> 0
> # But preferably the exit code in the presence of ERRORs should be, say, 1.
> {code}
> h3. How to improve
> I'd suggest that, using the above as the running example, if there are any 
> {{ERROR}} entries in the output (i.e. if there are any assignments remaining 
> that don't match the desired assignments), then the 
> {{kafka-reassign-partitions.sh}}  should return a n

[jira] [Created] (KAFKA-2214) kafka-reassign-partitions.sh --verify should return non-zero exit codes when reassignment is not completed yet

[Michael Noll (JIRA)]

Michael Noll created KAFKA-2214:
---

 Summary: kafka-reassign-partitions.sh --verify should return 
non-zero exit codes when reassignment is not completed yet
 Key: KAFKA-2214
 URL: https://issues.apache.org/jira/browse/KAFKA-2214
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Affects Versions: 0.8.1.1
Reporter: Michael Noll
Priority: Minor


h4. Background

The admin script {{kafka-reassign-partitions.sh}} should integrate better with 
automation tools such as Ansible, which rely on scripts adhering to Unix best 
practices such as appropriate exit codes on success/failure.

h4. Current behavior (incorrect)

When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
prints {{ERROR}} messages but returns an exit code of zero, which indicates 
success.  This behavior makes it a bit cumbersome to integrate the script into 
automation tools such as Ansible.

{code}
$ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
--reassignment-json-file partitions-to-move.json --verify

Status of partition reassignment:
ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
reassignment (316,324) for partition [mytopic,2]
Reassignment of partition [mytopic,0] completed successfully
Reassignment of partition [myothertopic,1] completed successfully
Reassignment of partition [myothertopic,3] completed successfully
...

$ echo $?
0

# But preferably the exit code would be, say, 1.
{code}

h3. How to improve

I'd suggest that, using the above as the running example, if there are any 
{{ERROR}} entries in the output (i.e. if there are any assignments remaining 
that don't match the desired assignments), then the 
{{kafka-reassign-partitions.sh}}  should return a non-zero exit code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2201) Open file handle leak

[Albert Visagie (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553882#comment-14553882
 ] 

Albert Visagie commented on KAFKA-2201:
---

Excerpt from lsof:
COMMAND PID   USER   FD  TYPE DEVICE  SIZE/OFF   
NODE NAME
java  29258   root  DEL   REG   8,49  
1654802 /tmp/kafka-logs/x.phid-3/92175377.index.deleted
java  29258   root  mem   REG   8,49 55816
1687762 /tmp/kafka-logs/x.zupdate.phid-0/41645890.index
java  29258   root  DEL   REG   8,49  
1851417 /tmp/kafka-logs/x.phid-2/92164826.index.deleted
java  29258   root  mem   REG   8,49 55760 
802895 /tmp/kafka-logs/x.zupdate.phid-3/41748899.index
java  29258   root  DEL   REG   8,49  
1867784 /tmp/kafka-logs/x.phid-1/91807304.index.deleted
java  29258   root  DEL   REG   8,49  
1679464 /tmp/kafka-logs/x.phid-0/91714298.index.deleted
java  29258   root  DEL   REG   8,49  
1654800 /tmp/kafka-logs/x.phid-3/91778942.index.deleted
java  29258   root  DEL   REG   8,49  
1851413 /tmp/kafka-logs/x.phid-2/91765907.index.deleted
java  29258   root  mem   REG   8,49 59440
1720341 /tmp/kafka-logs/y.zupdate.-3/18996626.index
java  29258   root  mem   REG   8,49 59432
3514436 /tmp/kafka-logs/y.zupdate.-1/18778185.index
java  29258   root  DEL   REG   8,49  
1835168 /tmp/kafka-logs/x.phid-1/91409225.index.deleted

The .deleted ones grow steadily.


> Open file handle leak
> -
>
> Key: KAFKA-2201
> URL: https://issues.apache.org/jira/browse/KAFKA-2201
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.1
> Environment: Debian Linux 7, 64 bit
> Oracle JDK 1.7.0u40, 64-bit
>Reporter: Albert Visagie
>
> The kafka broker crashes with the following stack trace from the server.log 
> roughly every 18 hours:
> [2015-05-19 07:39:22,924] FATAL [KafkaApi-0] Halting due to unrecoverable I/O 
> error while handling produce request:  (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: I/O exception in append to log 'nnn-1'
> at kafka.log.Log.append(Log.scala:266)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:379)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:365)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inReadLock(Utils.scala:541)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365)
> at 
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:291)
> at 
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:282)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:282)
> at 
> kafka.server.KafkaApis.handleProducerOrOffsetCommitRequest(KafkaApis.scala:204)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
> at 
> kafka.log.OffsetIndex$$anonfun$trimToValidSi

[jira] [Comment Edited] (KAFKA-2201) Open file handle leak

[Albert Visagie (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553829#comment-14553829
 ] 

Albert Visagie edited comment on KAFKA-2201 at 5/21/15 8:08 AM:


Some new information:
1. We adjusted the segment size to about 30MB instead of roughly 1MB as before. 
The segments roll over more slowly, as one would expect.
2. Our stats graph (https://www.dropbox.com/s/0ugmonkqujw0457/files1.png) show 
the previous climb in open files right up to the crash. (Zoomed to more recent: 
https://www.dropbox.com/s/wx6m04gf5dvlonw/files2.png)
3. The lower slope of the line after the high peak shows that it still climbs, 
but much more slowly.
4. The more recent sudden drop in open filehandles corresponds exactly in time 
with the following output from kafkaServer-gc.log (good idea logging that by 
default right out the box, btw).

2015-05-21T07:38:47.294+: 65001.794: [GC2015-05-21T07:38:47.294+: 
65001.794: [ParNew: 5827K->5191K(314560K), 0.0083640 secs] 
648593K->648101K(1013632K), 0.0084390 secs] [Times: user=0.09 sys=0.00, 
real=0.01 secs]
2015-05-21T07:38:53.377+: 65007.877: [GC2015-05-21T07:38:53.377+: 
65007.877: [ParNew: 284807K->6304K(314560K), 0.0356520 secs] 
927717K->649362K(1013632K), 0.0357490 secs] [Times: user=0.29 sys=0.00, 
real=0.04 secs]
2015-05-21T07:38:59.836+: 65014.336: [GC2015-05-21T07:38:59.836+: 
65014.336: [ParNew: 285920K->5286K(314560K), 0.0391160 secs] 
928978K->648489K(1013632K), 0.0392210 secs] [Times: user=0.30 sys=0.00, 
real=0.04 secs]
2015-05-21T07:38:59.876+: 65014.376: [GC [1 CMS-initial-mark: 
643203K(699072K)] 650842K(1013632K), 0.0078780 secs] [Times: user=0.01 
sys=0.00, real=0.01 secs]
2015-05-21T07:38:59.884+: 65014.384: [CMS-concurrent-mark-start]
2015-05-21T07:38:59.903+: 65014.403: [CMS-concurrent-mark: 0.017/0.019 
secs] [Times: user=0.13 sys=0.01, real=0.02 secs]
2015-05-21T07:38:59.903+: 65014.403: [CMS-concurrent-preclean-start]
2015-05-21T07:38:59.908+: 65014.408: [CMS-concurrent-preclean: 0.004/0.005 
secs] [Times: user=0.02 sys=0.01, real=0.00 secs]
2015-05-21T07:38:59.908+: 65014.408: 
[CMS-concurrent-abortable-preclean-start]
2015-05-21T07:39:03.094+: 65017.595: [CMS-concurrent-abortable-preclean: 
3.079/3.186 secs] [Times: user=4.73 sys=0.36, real=3.19 secs]
2015-05-21T07:39:03.095+: 65017.595: [GC[YG occupancy: 147607 K (314560 
K)]2015-05-21T07:39:03.095+: 65017.595: [Rescan (parallel) , 0.0171330 
secs]2015-05-21T07:39:03.112+: 65017.612: [weak refs processing, 0.0015230 
secs]2015-05-21T07:39:03.113+: 65017.614: [clas
s unloading, 0.0037630 secs]2015-05-21T07:39:03.117+: 65017.617: [scrub 
symbol table, 0.0015450 secs]2015-05-21T07:39:03.119+: 65017.619: [scrub 
string table, 0.0002790 secs] [1 CMS-remark: 643203K(699072K)] 
790811K(1013632K), 0.0261450 secs] [Times: user=0.21 sys=0.00, re
al=0.03 secs]
2015-05-21T07:39:03.121+: 65017.621: [CMS-concurrent-sweep-start]
2015-05-21T07:39:03.121+: 65017.622: [GC2015-05-21T07:39:03.121+: 
65017.622: [ParNew: 147607K->3379K(314560K), 0.0178100 secs] 
790811K->646718K(1013632K), 0.0178920 secs] [Times: user=0.14 sys=0.00, 
real=0.02 secs]
2015-05-21T07:39:04.378+: 65018.879: [CMS-concurrent-sweep: 1.238/1.257 
secs] [Times: user=2.12 sys=0.20, real=1.26 secs]
2015-05-21T07:39:04.378+: 65018.879: [CMS-concurrent-reset-start]
2015-05-21T07:39:04.380+: 65018.881: [CMS-concurrent-reset: 0.002/0.002 
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2015-05-21T07:39:09.187+: 65023.688: [GC2015-05-21T07:39:09.187+: 
65023.688: [ParNew: 282995K->5097K(314560K), 0.0353740 secs] 
289202K->11447K(1013632K), 0.0354670 secs] [Times: user=0.26 sys=0.01, 
real=0.04 secs]
2015-05-21T07:39:15.904+: 65030.404: [GC2015-05-21T07:39:15.904+: 
65030.404: [ParNew: 284713K->5448K(314560K), 0.0376610 secs] 
291063K->11942K(1013632K), 0.0377660 secs] [Times: user=0.29 sys=0.01, 
real=0.04 secs]
2015-05-21T07:39:22.546+: 65037.047: [GC2015-05-21T07:39:22.547+: 
65037.047: [ParNew: 285064K->5615K(314560K), 0.0354600 secs] 
291558K->12109K(1013632K), 0.0355580 secs] [Times: user=0.30 sys=0.00, 
real=0.04 secs]
2015-05-21T07:39:28.956+: 65043.456: [GC2015-05-21T07:39:28.956+: 
65043.456: [ParNew: 285231K->5984K(314560K), 0.0420590 secs] 
291725K->12642K(1013632K), 0.0421550 secs] [Times: user=0.31 sys=0.00, 
real=0.05 secs]
2015-05-21T07:39:35.536+: 65050.037: [GC2015-05-21T07:39:35.536+: 
65050.037: [ParNew: 285600K->5180K(314560K), 0.0457240 secs] 
292258K->11992K(1013632K), 0.0458610 secs] [Times: user=0.32 sys=0.00, 
real=0.05 secs]

It only ever does the parallel collection on the young generation. The drop 
corresponds to the full GC.

This appears to support the hypothesis that the files are only closed on full 
GC, and thus are not closed right after they ar

[jira] [Commented] (KAFKA-2201) Open file handle leak

[Albert Visagie (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553829#comment-14553829
 ] 

Albert Visagie commented on KAFKA-2201:
---

Some new information:
1. We adjusted the segment size to about 30MB instead of roughly 1MB as before. 
The segments roll over more slowly, as one would expect.
2. Our stats graphs attached show the previous climb in open files right up to 
the crash.
3. The lower slope of the line after the high peak shows that it still climbs, 
but much more slowly.
4. The more recent sudden drop in open filehandles corresponds exactly in time 
with the following output from kafkaServer-gc.log (good idea logging that by 
default right out the box, btw).

2015-05-21T07:38:47.294+: 65001.794: [GC2015-05-21T07:38:47.294+: 
65001.794: [ParNew: 5827K->5191K(314560K), 0.0083640 secs] 
648593K->648101K(1013632K), 0.0084390 secs] [Times: user=0.09 sys=0.00, 
real=0.01 secs]
2015-05-21T07:38:53.377+: 65007.877: [GC2015-05-21T07:38:53.377+: 
65007.877: [ParNew: 284807K->6304K(314560K), 0.0356520 secs] 
927717K->649362K(1013632K), 0.0357490 secs] [Times: user=0.29 sys=0.00, 
real=0.04 secs]
2015-05-21T07:38:59.836+: 65014.336: [GC2015-05-21T07:38:59.836+: 
65014.336: [ParNew: 285920K->5286K(314560K), 0.0391160 secs] 
928978K->648489K(1013632K), 0.0392210 secs] [Times: user=0.30 sys=0.00, 
real=0.04 secs]
2015-05-21T07:38:59.876+: 65014.376: [GC [1 CMS-initial-mark: 
643203K(699072K)] 650842K(1013632K), 0.0078780 secs] [Times: user=0.01 
sys=0.00, real=0.01 secs]
2015-05-21T07:38:59.884+: 65014.384: [CMS-concurrent-mark-start]
2015-05-21T07:38:59.903+: 65014.403: [CMS-concurrent-mark: 0.017/0.019 
secs] [Times: user=0.13 sys=0.01, real=0.02 secs]
2015-05-21T07:38:59.903+: 65014.403: [CMS-concurrent-preclean-start]
2015-05-21T07:38:59.908+: 65014.408: [CMS-concurrent-preclean: 0.004/0.005 
secs] [Times: user=0.02 sys=0.01, real=0.00 secs]
2015-05-21T07:38:59.908+: 65014.408: 
[CMS-concurrent-abortable-preclean-start]
2015-05-21T07:39:03.094+: 65017.595: [CMS-concurrent-abortable-preclean: 
3.079/3.186 secs] [Times: user=4.73 sys=0.36, real=3.19 secs]
2015-05-21T07:39:03.095+: 65017.595: [GC[YG occupancy: 147607 K (314560 
K)]2015-05-21T07:39:03.095+: 65017.595: [Rescan (parallel) , 0.0171330 
secs]2015-05-21T07:39:03.112+: 65017.612: [weak refs processing, 0.0015230 
secs]2015-05-21T07:39:03.113+: 65017.614: [clas
s unloading, 0.0037630 secs]2015-05-21T07:39:03.117+: 65017.617: [scrub 
symbol table, 0.0015450 secs]2015-05-21T07:39:03.119+: 65017.619: [scrub 
string table, 0.0002790 secs] [1 CMS-remark: 643203K(699072K)] 
790811K(1013632K), 0.0261450 secs] [Times: user=0.21 sys=0.00, re
al=0.03 secs]
2015-05-21T07:39:03.121+: 65017.621: [CMS-concurrent-sweep-start]
2015-05-21T07:39:03.121+: 65017.622: [GC2015-05-21T07:39:03.121+: 
65017.622: [ParNew: 147607K->3379K(314560K), 0.0178100 secs] 
790811K->646718K(1013632K), 0.0178920 secs] [Times: user=0.14 sys=0.00, 
real=0.02 secs]
2015-05-21T07:39:04.378+: 65018.879: [CMS-concurrent-sweep: 1.238/1.257 
secs] [Times: user=2.12 sys=0.20, real=1.26 secs]
2015-05-21T07:39:04.378+: 65018.879: [CMS-concurrent-reset-start]
2015-05-21T07:39:04.380+: 65018.881: [CMS-concurrent-reset: 0.002/0.002 
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2015-05-21T07:39:09.187+: 65023.688: [GC2015-05-21T07:39:09.187+: 
65023.688: [ParNew: 282995K->5097K(314560K), 0.0353740 secs] 
289202K->11447K(1013632K), 0.0354670 secs] [Times: user=0.26 sys=0.01, 
real=0.04 secs]
2015-05-21T07:39:15.904+: 65030.404: [GC2015-05-21T07:39:15.904+: 
65030.404: [ParNew: 284713K->5448K(314560K), 0.0376610 secs] 
291063K->11942K(1013632K), 0.0377660 secs] [Times: user=0.29 sys=0.01, 
real=0.04 secs]
2015-05-21T07:39:22.546+: 65037.047: [GC2015-05-21T07:39:22.547+: 
65037.047: [ParNew: 285064K->5615K(314560K), 0.0354600 secs] 
291558K->12109K(1013632K), 0.0355580 secs] [Times: user=0.30 sys=0.00, 
real=0.04 secs]
2015-05-21T07:39:28.956+: 65043.456: [GC2015-05-21T07:39:28.956+: 
65043.456: [ParNew: 285231K->5984K(314560K), 0.0420590 secs] 
291725K->12642K(1013632K), 0.0421550 secs] [Times: user=0.31 sys=0.00, 
real=0.05 secs]
2015-05-21T07:39:35.536+: 65050.037: [GC2015-05-21T07:39:35.536+: 
65050.037: [ParNew: 285600K->5180K(314560K), 0.0457240 secs] 
292258K->11992K(1013632K), 0.0458610 secs] [Times: user=0.32 sys=0.00, 
real=0.05 secs]

It only ever does the parallel collection on the young generation. The drop 
corresponds to the full GC.

This appears to support the hypothesis that the files are only closed on full 
GC, and thus are not closed right after they are deleted.

I will upgrade the jdk from 1.7u40 to 1.7u79 next.


> Open file handle leak
> -
>
> Key: KAFKA-2201
> URL: h

[jira] [Commented] (KAFKA-2201) Open file handle leak

[Albert Visagie (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553818#comment-14553818
 ] 

Albert Visagie commented on KAFKA-2201:
---

We log stats using lsof, and distinguish between files, sockets and other file 
descriptors. The count represents files. We only have one broker and at most 5 
clients. The number of sockets corresponds with that.

> Open file handle leak
> -
>
> Key: KAFKA-2201
> URL: https://issues.apache.org/jira/browse/KAFKA-2201
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.1
> Environment: Debian Linux 7, 64 bit
> Oracle JDK 1.7.0u40, 64-bit
>Reporter: Albert Visagie
>
> The kafka broker crashes with the following stack trace from the server.log 
> roughly every 18 hours:
> [2015-05-19 07:39:22,924] FATAL [KafkaApi-0] Halting due to unrecoverable I/O 
> error while handling produce request:  (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: I/O exception in append to log 'nnn-1'
> at kafka.log.Log.append(Log.scala:266)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:379)
> at 
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:365)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inReadLock(Utils.scala:541)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365)
> at 
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:291)
> at 
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:282)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:282)
> at 
> kafka.server.KafkaApis.handleProducerOrOffsetCommitRequest(KafkaApis.scala:204)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
> at 
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(OffsetIndex.scala:265)
> at 
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:265)
> at 
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:265)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:264)
> at kafka.log.Log.roll(Log.scala:563)
> at kafka.log.Log.maybeRoll(Log.scala:539)
> at kafka.log.Log.append(Log.scala:306)
> ... 21 more
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
> ... 33 more
> The Kafka broker's open filehandles as seen by
> lsof | grep pid | wc -l 
> grows steadily as it runs. Under our load it lasts about 18 hours before 
> crashing with the stack trace above.
> We were experimenting with settings under Log Retention Policy in 
> server.properties:
> log.retention.hours=168
> log.retention.bytes=107374182
> log.segment.bytes=1073741
> log.retention.check.interval.ms=3000
> The result is that the broker rolls over segments quite rapidly. We don't 
> have to run it that way of course.
> We are running only one broker at the moment.
> lsof shows many open files without size and absent from ls in the log 
> directory with the suffix ".deleted"
> This is kafka 0.8.2.1 with scala 2.10.4 as downloaded from the website last 
> week.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Using GitHub Pull Requests for contributions and code review

[Ismael Juma]

On Fri, May 1, 2015 at 8:38 AM, Ewen Cheslack-Postava 
wrote:

> One thing I noticed is that when you try to generate a PR it defaults to
> the 0.8.2 branch. Can we fix that up to be trunk by default?
>

I filed an INFRA ticket for this:

https://issues.apache.org/jira/browse/INFRA-9680

Best,
Ismael