Re: Producer becomes slow over time

[Prabhjot Bharaj]

Hi Erik,

I've not tested it on a producer that is not a part of the kafka cluster.


*Producer configuration on all machines:-*

root@x.x.x.x:/a/kafka/config# cat producer.properties  | egrep -v '^#|^$'

metadata.broker.list=localhost:9092

producer.type=sync

compression.codec=none

serializer.class=kafka.serializer.DefaultEncoder


*Server configuration on all machines:-*

root@y.y.y.y:/a/kafka/config# cat server.properties  | egrep -v '^#|^$'

broker.id=0

port=9092

num.network.threads=6

num.io.threads=8

socket.send.buffer.bytes=10485760

socket.receive.buffer.bytes=10485760

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=30

log.cleaner.enable=false

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=6000

num.replica.fetchers=4

*Command used from both the machines (slow and fast producer):*

kafka-producer-perf-test.sh --broker-list
x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092,a.a.a.a:9092,b.b.b.b:9092 --messages
1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats
--threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms
1 --compression-codec 2 --reporting-interval 1000

Regards,

Prabhjot

On Thu, Sep 24, 2015 at 6:36 PM, Helleren, Erik 
wrote:

> What happens when the new producer that is getting 70 MB/s is started on a
> machine that is not part of the kafka cluster?
>
> Can you include your topic description/configuration, producer
> configuration, and broker configuration?
>
> On 9/24/15, 1:44 AM, "Prabhjot Bharaj"  wrote:
>
> >Hi,
> >
> >I would like to dig deep into this issue. I've changed log4j.properties
> >for
> >logging in ALL mode in all places. I am getting lost in the logs.
> >
> >Any pointers would be welcome
> >
> >Please let me know if you would need any information regarding this
> >
> >Thanks,
> >Prabhjot
> >
> >On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj 
> >wrote:
> >
> >> Hello Folks,
> >>
> >> I've noticed that 2 producer machines, that I had configured, have
> >>become
> >> very slow over time
> >> They are giving 17-19 MB/s
> >>
> >> But, a producer that I setup today is giving 70MB/s as the write
> >>throughput
> >>
> >> If I see the contents of bin, libs, config directories, nothing is
> >> different in the files on any of the producer machines.
> >>
> >> Producer is running on the kafka machines itself
> >>
> >> Request your expertise
> >>
> >> Regards,
> >> Prabhjot
> >>
> >>
> >>
> >
> >
> >--
> >-
> >"There are only 10 types of people in the world: Those who understand
> >binary, and those who don't"
>
>


-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

[jira] [Updated] (KAFKA-2494) Document ReplicaId in OffsetRequest in the protocol guide

[Magnus Reftel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Magnus Reftel updated KAFKA-2494:
-
Summary: Document ReplicaId in OffsetRequest in the protocol guide  (was: 
Documetn ReplicaId in OffsetRequest in the protocol guide)

> Document ReplicaId in OffsetRequest in the protocol guide
> -
>
> Key: KAFKA-2494
> URL: https://issues.apache.org/jira/browse/KAFKA-2494
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Reporter: Magnus Reftel
>
> The documentation for OffsetRequest in the protocol guide lists ReplicaId as 
> one of the fields, but does not say what the field is for. It appears that 
> it's similar to FetchRequest, where it's set to -1 by clients. It would be 
> nice if that was documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2597) Add Eclipse directories to .gitignore

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch updated KAFKA-2597:
-
Fix Version/s: 0.9.0.0
   Status: Patch Available  (was: Open)

This contribution is ready for review.

> Add Eclipse directories to .gitignore
> -
>
> Key: KAFKA-2597
> URL: https://issues.apache.org/jira/browse/KAFKA-2597
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Trivial
> Fix For: 0.9.0.0
>
>
> Add to {{.gitignore}} the Eclipse IDE directories {{.metadata}} and 
> {{.recommenders}}. These store state of the IDE's workspace, and should not 
> be checked in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2593) KeyValueStores should not require use of the context's default serializers and deserializers

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch updated KAFKA-2593:
-
Status: Patch Available  (was: Open)

This contribution is ready for review.

> KeyValueStores should not require use of the context's default serializers 
> and deserializers
> 
>
> Key: KAFKA-2593
> URL: https://issues.apache.org/jira/browse/KAFKA-2593
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Reporter: Randall Hauch
>Assignee: Randall Hauch
> Fix For: 0.9.0.0
>
>
> Currently the {{InMemoryKeyValueStore}} is only able to use the key and value 
> serializers and deserializers (aka, "serdes") from the {{ProcessingContext}}. 
> This means that a {{Processor}} implementation that wants to use the 
> {{InMemoryKeyValueStore}} can only do this if the key and value types match 
> those set up as the default serdes in the topology's configuration.
> Additionally, the {{RocksDBKeyValueStore}} is only capable of {{byte[]}} keys 
> and values.
> Both of these key-value stores should allow the component using them to 
> specify the serdes for both the keys and values. As a convenience, the 
> current behavior should still be supported, as should a way to infer the 
> serdes for the "built-in" serializers and deserializers (e.g., strings, 
> integers, longs, and byte arrays).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2594) Add a key-value store that is a fixed-capacity in-memory LRU cache

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch updated KAFKA-2594:
-
Status: Patch Available  (was: Open)

This contribution is ready for review.

> Add a key-value store that is a fixed-capacity in-memory LRU cache 
> ---
>
> Key: KAFKA-2594
> URL: https://issues.apache.org/jira/browse/KAFKA-2594
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Reporter: Randall Hauch
>Assignee: Randall Hauch
> Fix For: 0.9.0.0
>
>
> The current {{KeyValueStore}} implementations are not limited in size, and 
> thus are less useful for some use cases. This subtask will add a simple 
> key-value store that maintains in memory at most a maximum number of entries 
> that were recently read or written. When the cache size reaches the capacity 
> and a new entry is to be added, the least recently used entry will be 
> automatically purged from the cache. This key-value store will extend 
> {{MeteredKeyValueStore}} for monitoring and recording of changes to a backing 
> topic, enabling recovery of the cache contents from the replicated state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Grant Henke]

If we create a protocol version negotiation api for clients, can we use it
to replace or improve the ease of upgrades that break inter-broker
messaging?

Currently upgrades that break the wire protocol take 2 rolling restarts.
The first restart we set inter.broker.protocol.version telling all brokers
to communicate on the old version, and then we restart again removing the
inter.broker.protocol.version. With this api the brokers could agree on a
version to communicate with, and when bounced re-negotiate to the new
version.


On Mon, Sep 28, 2015 at 10:26 PM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> Nice write-up.
>
> Just had a question, instead of returning an empty response back to the
> client, would it be better for the broker to return a response that gives
> some more info to the client regarding the min version they need to upgrade
> to in order to communicate with the broker.
>
>
> Thanks,
>
> Mayuresh
>
> On Mon, Sep 28, 2015 at 6:36 PM, Jiangjie Qin 
> wrote:
>
> > Thanks for the writeup. I also think having a specific protocol for
> > client-broker version negotiation is better.
> >
> > I'm wondering is it better to let the broker to decide the version to
> use?
> > It might have some value If brokers have preference for a particular
> > version.
> > Using a global version is a good approach. For the client-broker
> > negotiation, I am thinking about something like:
> >
> > ProtocolSyncRequest => ClientType [ProtocolVersion]
> > ClientType => int8
> > ProtocolVersion => int16
> >
> > ProtocolSyncResponse => PreferredVersion
> > PreferredVersion => int16
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Sep 28, 2015 at 11:59 AM, Jun Rao  wrote:
> >
> > > I agree with Ewen that having the keys explicitly specified in the
> > response
> > > is better.
> > >
> > > In addition to the supported protocol version, there are other
> > interesting
> > > metadata at the broker level that could be of interest to things like
> > admin
> > > tools (e.g., used disk space, remaining disk space, etc). I am
> wondering
> > if
> > > we should separate those into different requests. For inquiring the
> > > protocol version, we can have a separate BrokerProtocolRequest. The
> > > response will just include the broker version and perhaps a list of
> > > supported requests and versions?
> > >
> > > As for sending an empty response for unrecognized requests, do you how
> is
> > > that handled in other similar systems?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Sep 28, 2015 at 10:47 AM, Jason Gustafson 
> > > wrote:
> > >
> > > > Having the version API can make clients more robust, so I'm in favor.
> > One
> > > > note on the addition of the "rack" field. Since this is a
> > broker-specific
> > > > setting, the client would have to query BrokerMetadata for every new
> > > broker
> > > > it connects to (unless we also expose rack in TopicMetadata). This is
> > > also
> > > > kind of unfortunate for admin utilities leveraging this API. It might
> > be
> > > > more convenient to allow this API to return broker metadata for the
> > full
> > > > cluster, assuming all of it could be made available in Zookeeper.
> > > >
> > > > As for using the empty response to indicate an incompatible API, it
> > seems
> > > > like that could work. I think some of the clients might catch
> response
> > > > parsing exceptions and retry anyway, but that's no worse than
> retrying
> > > > because of a disconnect in the same case.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Sep 25, 2015 at 9:34 PM, Ewen Cheslack-Postava <
> > > e...@confluent.io>
> > > > wrote:
> > > >
> > > > > The basic functionality is definitely useful here. I'm generally in
> > > favor
> > > > > of exposing some info about broker versions to clients.
> > > > >
> > > > > I'd prefer to keep the key/values explicit. Making them extensible
> > > > > string/string pairs is good for avoiding unnecessary version
> changes
> > in
> > > > the
> > > > > protocol, but I think we should explicitly define the valid
> key/value
> > > > > formats in each version of the protocol. New keys can safely be
> > > ignored,
> > > > > but actually specifying the format of the values is important if we
> > > ever
> > > > > need to evolve those formats.
> > > > >
> > > > > I like some of the examples you've provided for returned key/value
> > > pairs
> > > > > and I think we should provide some of them even when the values
> > should
> > > be
> > > > > obvious from the broker version.
> > > > >
> > > > > * broker.version - are we definitely standardizing on this
> versioning
> > > > > format? 4 digits, with each level indicating the intuitive levels
> of
> > > > > compatibility? Is there any chance we'll have a 0.10.0.0? This
> might
> > > seem
> > > > > like a trivial consideration, but after fighting versioning in
> > > different
> > > > > packaging systems, I'm a bit more sensitive to the annoying effects
> > > that
> > > > > not considering 

[jira] [Commented] (KAFKA-2526) Console Producer / Consumer's serde config is not working

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935428#comment-14935428
 ] 

Mayuresh Gharat commented on KAFKA-2526:


Hi [~guozhang], can you list the steps to reproduce this issue. I think we had 
faced this issue sometime back at Linkedin. 

> Console Producer / Consumer's serde config is not working
> -
>
> Key: KAFKA-2526
> URL: https://issues.apache.org/jira/browse/KAFKA-2526
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Fix For: 0.10.0.0
>
>
> Although in the console producer one can specify the key value serializer, 
> they are actually not used since 1) it always serialize the input string as 
> String.getBytes (hence always pre-assume the string serializer) and 2) it is 
> actually only passed into the old producer. The same issues exist in console 
> consumer.
> In addition the configs in the console producer is messy: we have 1) some 
> config values exposed as cmd parameters, and 2) some config values in 
> --producer-property and 3) some in --property.
> It will be great to clean the configs up in both console producer and 
> consumer, and put them into a single --property parameter which could 
> possibly take a file to reading in property values as well, and only leave 
> --new-producer as the other command line parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2526) Console Producer / Consumer's serde config is not working

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935428#comment-14935428
 ] 

Mayuresh Gharat edited comment on KAFKA-2526 at 9/29/15 4:53 PM:
-

Hi [~guozhang], what steps did you followed to reproduce this? 
I think we had faced this issue sometime back at Linkedin. 


was (Author: mgharat):
Hi [~guozhang], can you list the steps to reproduce this issue. I think we had 
faced this issue sometime back at Linkedin. 

> Console Producer / Consumer's serde config is not working
> -
>
> Key: KAFKA-2526
> URL: https://issues.apache.org/jira/browse/KAFKA-2526
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Fix For: 0.10.0.0
>
>
> Although in the console producer one can specify the key value serializer, 
> they are actually not used since 1) it always serialize the input string as 
> String.getBytes (hence always pre-assume the string serializer) and 2) it is 
> actually only passed into the old producer. The same issues exist in console 
> consumer.
> In addition the configs in the console producer is messy: we have 1) some 
> config values exposed as cmd parameters, and 2) some config values in 
> --producer-property and 3) some in --property.
> It will be great to clean the configs up in both console producer and 
> consumer, and put them into a single --property parameter which could 
> possibly take a file to reading in property values as well, and only leave 
> --new-producer as the other command line parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2417) Ducktape tests for SSL/TLS

[Geoff Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935462#comment-14935462
 ] 

Geoff Anderson commented on KAFKA-2417:
---

[~rsivaram] Thanks for offering to help, this would be tremendously useful. I 
spoke yesterday with [~junrao] about what might be a good minimum test set 
here, and we concluded that we could adapt and reuse the existing ducktape 
replication test(s), but with ssl enabled on the clients-broker communication. 
These replication tests introduce various broker failures while producing and 
consuming in the background, and then validate consumed messages against 
acknowledged messages.

Updating would involve a couple steps:
- Some helper python logic for setting up trust store etc
- Update kafka.py, verifiable_producer.py, and console_consumer.py to add hooks 
for enabling ssl
- We can reuse replication_test.py by either a) parametrizing the test itself 
with the ducktape @parameterize or @matrix decorator, and/or b) using the 
common base test class called ProduceConsumeValidateTest introduced in 
https://github.com/apache/kafka/pull/229

[~ijuma] [~junrao] What do you think?





> Ducktape tests for SSL/TLS
> --
>
> Key: KAFKA-2417
> URL: https://issues.apache.org/jira/browse/KAFKA-2417
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Geoff Anderson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> The tests should be complementary to the unit/integration tests written as 
> part of KAFKA-1685.
> Things to consider:
> * Upgrade/downgrade to turning on/off SSL
> * Failure testing
> * Expired/revoked certificates
> * Renegotiation
> Some changes to ducktape may be required for upgrade scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Quota's equals() is buggy.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/182


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-2120:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 and committed to trunk.

> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, 
> KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, 
> KAFKA-2120_2015-09-11_14:54:15.patch, KAFKA-2120_2015-09-15_18:57:20.patch, 
> KAFKA-2120_2015-09-18_19:27:48.patch, KAFKA-2120_2015-09-28_16:13:02.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2598) Add Test with authorizer for producer and consumer

[Parth Brahmbhatt (JIRA)]

Parth Brahmbhatt created KAFKA-2598:
---

 Summary: Add Test with authorizer for producer and consumer
 Key: KAFKA-2598
 URL: https://issues.apache.org/jira/browse/KAFKA-2598
 Project: Kafka
  Issue Type: Bug
  Components: security, unit tests
Affects Versions: 0.8.2.2
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt
 Fix For: 0.8.2.2


Now that we have all the authorizer code merged into trunk we should add a test 
that enables authorizer and tests that only authorized users can 
produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Mayuresh Gharat]

Right. But there should be a max old version that the broker should support
to avoid these incompatibility issues.
For example, if the broker is at version X, it should be able to support
the versions (clients and interbroker) till X-2. In case we have brokers
and clients older than that it can send a response warning them to upgrade
till X-2 minimum.
The backward compatibility limit can be discussed further. This will help
for rolling upgrades.

Thanks,

Mayuresh

On Tue, Sep 29, 2015 at 8:25 AM, Grant Henke  wrote:

> If we create a protocol version negotiation api for clients, can we use it
> to replace or improve the ease of upgrades that break inter-broker
> messaging?
>
> Currently upgrades that break the wire protocol take 2 rolling restarts.
> The first restart we set inter.broker.protocol.version telling all brokers
> to communicate on the old version, and then we restart again removing the
> inter.broker.protocol.version. With this api the brokers could agree on a
> version to communicate with, and when bounced re-negotiate to the new
> version.
>
>
> On Mon, Sep 28, 2015 at 10:26 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com> wrote:
>
> > Nice write-up.
> >
> > Just had a question, instead of returning an empty response back to the
> > client, would it be better for the broker to return a response that gives
> > some more info to the client regarding the min version they need to
> upgrade
> > to in order to communicate with the broker.
> >
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Mon, Sep 28, 2015 at 6:36 PM, Jiangjie Qin  >
> > wrote:
> >
> > > Thanks for the writeup. I also think having a specific protocol for
> > > client-broker version negotiation is better.
> > >
> > > I'm wondering is it better to let the broker to decide the version to
> > use?
> > > It might have some value If brokers have preference for a particular
> > > version.
> > > Using a global version is a good approach. For the client-broker
> > > negotiation, I am thinking about something like:
> > >
> > > ProtocolSyncRequest => ClientType [ProtocolVersion]
> > > ClientType => int8
> > > ProtocolVersion => int16
> > >
> > > ProtocolSyncResponse => PreferredVersion
> > > PreferredVersion => int16
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Sep 28, 2015 at 11:59 AM, Jun Rao  wrote:
> > >
> > > > I agree with Ewen that having the keys explicitly specified in the
> > > response
> > > > is better.
> > > >
> > > > In addition to the supported protocol version, there are other
> > > interesting
> > > > metadata at the broker level that could be of interest to things like
> > > admin
> > > > tools (e.g., used disk space, remaining disk space, etc). I am
> > wondering
> > > if
> > > > we should separate those into different requests. For inquiring the
> > > > protocol version, we can have a separate BrokerProtocolRequest. The
> > > > response will just include the broker version and perhaps a list of
> > > > supported requests and versions?
> > > >
> > > > As for sending an empty response for unrecognized requests, do you
> how
> > is
> > > > that handled in other similar systems?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Sep 28, 2015 at 10:47 AM, Jason Gustafson <
> ja...@confluent.io>
> > > > wrote:
> > > >
> > > > > Having the version API can make clients more robust, so I'm in
> favor.
> > > One
> > > > > note on the addition of the "rack" field. Since this is a
> > > broker-specific
> > > > > setting, the client would have to query BrokerMetadata for every
> new
> > > > broker
> > > > > it connects to (unless we also expose rack in TopicMetadata). This
> is
> > > > also
> > > > > kind of unfortunate for admin utilities leveraging this API. It
> might
> > > be
> > > > > more convenient to allow this API to return broker metadata for the
> > > full
> > > > > cluster, assuming all of it could be made available in Zookeeper.
> > > > >
> > > > > As for using the empty response to indicate an incompatible API, it
> > > seems
> > > > > like that could work. I think some of the clients might catch
> > response
> > > > > parsing exceptions and retry anyway, but that's no worse than
> > retrying
> > > > > because of a disconnect in the same case.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Sep 25, 2015 at 9:34 PM, Ewen Cheslack-Postava <
> > > > e...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > The basic functionality is definitely useful here. I'm generally
> in
> > > > favor
> > > > > > of exposing some info about broker versions to clients.
> > > > > >
> > > > > > I'd prefer to keep the key/values explicit. Making them
> extensible
> > > > > > string/string pairs is good for avoiding unnecessary version
> > changes
> > > in
> > > > > the
> > > > > > protocol, but I think we should explicitly define the valid
> > key/value
> > > > > > formats in each version of the protocol. New keys can safely be
> > > > ignored,
> > > > > > but actually specif

[jira] [Updated] (KAFKA-2598) Add Test with authorizer for producer and consumer

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2598:
---
Fix Version/s: (was: 0.8.2.2)
   0.9.0.0
   Issue Type: Task  (was: Bug)

> Add Test with authorizer for producer and consumer
> --
>
> Key: KAFKA-2598
> URL: https://issues.apache.org/jira/browse/KAFKA-2598
> Project: Kafka
>  Issue Type: Task
>  Components: security, unit tests
>Affects Versions: 0.8.2.2
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> Now that we have all the authorizer code merged into trunk we should add a 
> test that enables authorizer and tests that only authorized users can 
> produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Kafka-trunk #648

[Apache Jenkins Server]

See 

Changes:

[jjkoshy] KAFKA-2120; Add a request timeout to NetworkClient (KIP-19); reviewed 
by Jason Gustafson, Ismael Juma, Joel Koshy, Jun Rao, and Edward Ribeiro

[wangguoz] MINOR: fix Quota's equal() function

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu3 (Ubuntu ubuntu legacy-ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 73720113b97ccfcd796a844f87438546ed2e09e4 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 73720113b97ccfcd796a844f87438546ed2e09e4
 > git rev-list 4e7db39556ba916c17f7dfeb32ff8ebefe468a33 # timeout=10
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
[Kafka-trunk] $ /bin/bash -xe /tmp/hudson5025315192640143351.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.5
:downloadWrapper UP-TO-DATE

BUILD SUCCESSFUL

Total time: 25.2 secs
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
[Kafka-trunk] $ /bin/bash -xe /tmp/hudson1016358297680867827.sh
+ ./gradlew -PscalaVersion=2.10.1 test
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.1
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:rat
Rat report: build/rat/rat-report.html
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test UP-TO-DATE
:clients:compileJavaNote: 
/x1/jenkins/jenkins-slave/workspace/Kafka-trunk/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

:clients:processResources UP-TO-DATE
:clients:classes
:clients:checkstyleMain
:clients:compileTestJavaNote: 
/x1/jenkins/jenkins-slave/workspace/Kafka-trunk/clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:clients:processTestResources UP-TO-DATE
:clients:testClasses
:clients:checkstyleTest[ant:checkstyle] 
/x1/jenkins/jenkins-slave/workspace/Kafka-trunk/clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java:38:1:
 Duplicate import to line 18 - org.junit.Assert.fail.
[ant:checkstyle] 
/x1/jenkins/jenkins-slave/workspace/Kafka-trunk/clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java:39:1:
 Duplicate import to line 15 - org.junit.Assert.assertEquals.
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':clients:checkstyleTest'.
> Checkstyle rule violations were found. See the report at: 
> file:///x1/jenkins/jenkins-slave/workspace/Kafka-trunk/clients/build/reports/checkstyle/test.xml

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 41.165 secs
Build step 'Execute shell' marked build as failure
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1

[jira] [Updated] (KAFKA-2598) Add Test with authorizer for producer and consumer

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2598:
---
Issue Type: Sub-task  (was: Task)
Parent: KAFKA-1682

> Add Test with authorizer for producer and consumer
> --
>
> Key: KAFKA-2598
> URL: https://issues.apache.org/jira/browse/KAFKA-2598
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security, unit tests
>Affects Versions: 0.8.2.2
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> Now that we have all the authorizer code merged into trunk we should add a 
> test that enables authorizer and tests that only authorized users can 
> produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2598) Add Test with authorizer for producer and consumer

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935587#comment-14935587
 ] 

Ismael Juma commented on KAFKA-2598:


Thanks for filing this Parth. Do you have the cycles to work on this one soon?

> Add Test with authorizer for producer and consumer
> --
>
> Key: KAFKA-2598
> URL: https://issues.apache.org/jira/browse/KAFKA-2598
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security, unit tests
>Affects Versions: 0.8.2.2
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> Now that we have all the authorizer code merged into trunk we should add a 
> test that enables authorizer and tests that only authorized users can 
> produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2598) Add Test with authorizer for producer and consumer

[Parth Brahmbhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935589#comment-14935589
 ] 

Parth Brahmbhatt commented on KAFKA-2598:
-

I should be able to pick this one up next week.

> Add Test with authorizer for producer and consumer
> --
>
> Key: KAFKA-2598
> URL: https://issues.apache.org/jira/browse/KAFKA-2598
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security, unit tests
>Affects Versions: 0.8.2.2
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> Now that we have all the authorizer code merged into trunk we should add a 
> test that enables authorizer and tests that only authorized users can 
> produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2058) ProducerTest.testSendWithDeadBroker transient failure

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935583#comment-14935583
 ] 

Guozhang Wang commented on KAFKA-2058:
--

I did not see this issue on my local machine either, but on Jekins it kept 
showing up, for example:

https://builds.apache.org/job/kafka-trunk-git-pr/591/console


> ProducerTest.testSendWithDeadBroker transient failure
> -
>
> Key: KAFKA-2058
> URL: https://issues.apache.org/jira/browse/KAFKA-2058
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Bill Bejeck
>  Labels: newbie
>
> {code}
> kafka.producer.ProducerTest > testSendWithDeadBroker FAILED
> java.lang.AssertionError: Message set should have 1 message
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:44)
> at 
> kafka.producer.ProducerTest.testSendWithDeadBroker(ProducerTest.scala:260)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2598) Add Test with authorizer for producer and consumer

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935604#comment-14935604
 ] 

Ismael Juma commented on KAFKA-2598:


Awesome, thanks.

> Add Test with authorizer for producer and consumer
> --
>
> Key: KAFKA-2598
> URL: https://issues.apache.org/jira/browse/KAFKA-2598
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security, unit tests
>Affects Versions: 0.8.2.2
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> Now that we have all the authorizer code merged into trunk we should add a 
> test that enables authorizer and tests that only authorized users can 
> produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2581) Run all existing ducktape tests with SSL-enabled clients and brokers

[Geoff Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935676#comment-14935676
 ] 

Geoff Anderson commented on KAFKA-2581:
---

[~rsivaram] My apologies for being behind on this discussion.

Your proposal sounds good to me. Most of the work here will be the one-time 
overhead of modifying the relevant services to use SSL.
As for the specific subset of tests to update, these are the categories you all 
have mentioned:
1) A simple sanity test to check that end-to-end production/consumption works 
with SSL enabled is a good place to start, and will be helpful to have when 
validating/debugging your updates to the service classes.
2) parametrizing a few of the replication tests to enable SSL would then not 
require much extra incremental effort
3) benchmark tests - these would at least allow for continued comparison to the 
corresponding non-ssl benchmark, as well as for future regression checks

>From discussions with [~junrao] [~ismael], it seems they'd prefer to see some 
>tests in all three categories. I might start with a PR on (1) and aim for (1) 
>and (2) at a minimum.

> Run all existing ducktape tests with SSL-enabled clients and brokers
> 
>
> Key: KAFKA-2581
> URL: https://issues.apache.org/jira/browse/KAFKA-2581
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> New ducktape tests for testing SSL are being added under KAFKA-2417. This 
> task will enable existing ducktape tests to be run with SSL-enabled brokers 
> and clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: HOTFIX: Checkstyle import fix

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/261

HOTFIX: Checkstyle import fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka checkstyle-import-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/261.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #261


commit bd0d77ebf70bf31e20bf2a1a8b671582d0065d63
Author: Ismael Juma 
Date:   2015-09-29T20:06:49Z

HOTFIX: Checkstyle import fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: HOTFIX: Checkstyle import fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/261


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2599) Metadata#getClusterForCurrentTopics can throw NPE even with null checking

[Edward Ribeiro (JIRA)]

Edward Ribeiro created KAFKA-2599:
-

 Summary: Metadata#getClusterForCurrentTopics can throw NPE even 
with null checking
 Key: KAFKA-2599
 URL: https://issues.apache.org/jira/browse/KAFKA-2599
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.8.2.1
Reporter: Edward Ribeiro
Assignee: Edward Ribeiro
Priority: Minor
 Fix For: 0.8.1.2, 0.9.0.0


While working on another issue I have just seen the following:

{code}
private Cluster getClusterForCurrentTopics(Cluster cluster) {
Collection partitionInfos = new ArrayList<>();
if (cluster != null) {
for (String topic : this.topics) {
partitionInfos.addAll(cluster.partitionsForTopic(topic));
}
}
return new Cluster(cluster.nodes(), partitionInfos);
}
{code}

Well, there's a null check for cluster, but if cluster is null it will throw 
NPE. So, I put together a quick fix and changed {{MetadataTest}} to reproduce 
this error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2599 Metadata#getClusterForCurrentTopics...

[eribeiro]

GitHub user eribeiro opened a pull request:

https://github.com/apache/kafka/pull/262

KAFKA-2599 Metadata#getClusterForCurrentTopics can throw NPE even wit…

…h null checking

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eribeiro/kafka KAFKA-2599

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #262


commit 960107edf131a710e1af808fe93a6ad8566098a5
Author: Edward Ribeiro 
Date:   2015-09-29T20:42:31Z

KAFKA-2599 Metadata#getClusterForCurrentTopics can throw NPE even with null 
checking




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2599) Metadata#getClusterForCurrentTopics can throw NPE even with null checking

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935835#comment-14935835
 ] 

ASF GitHub Bot commented on KAFKA-2599:
---

GitHub user eribeiro opened a pull request:

https://github.com/apache/kafka/pull/262

KAFKA-2599 Metadata#getClusterForCurrentTopics can throw NPE even wit…

…h null checking

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eribeiro/kafka KAFKA-2599

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #262


commit 960107edf131a710e1af808fe93a6ad8566098a5
Author: Edward Ribeiro 
Date:   2015-09-29T20:42:31Z

KAFKA-2599 Metadata#getClusterForCurrentTopics can throw NPE even with null 
checking




> Metadata#getClusterForCurrentTopics can throw NPE even with null checking
> -
>
> Key: KAFKA-2599
> URL: https://issues.apache.org/jira/browse/KAFKA-2599
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: Edward Ribeiro
>Assignee: Edward Ribeiro
>Priority: Minor
> Fix For: 0.8.1.2, 0.9.0.0
>
>
> While working on another issue I have just seen the following:
> {code}
> private Cluster getClusterForCurrentTopics(Cluster cluster) {
> Collection partitionInfos = new ArrayList<>();
> if (cluster != null) {
> for (String topic : this.topics) {
> partitionInfos.addAll(cluster.partitionsForTopic(topic));
> }
> }
> return new Cluster(cluster.nodes(), partitionInfos);
> }
> {code}
> Well, there's a null check for cluster, but if cluster is null it will throw 
> NPE. So, I put together a quick fix and changed {{MetadataTest}} to reproduce 
> this error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2599) Metadata#getClusterForCurrentTopics can throw NPE even with null checking

[Edward Ribeiro (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Ribeiro updated KAFKA-2599:
--
Description: 
While working on another issue I have just seen the following:

{code}
private Cluster getClusterForCurrentTopics(Cluster cluster) {
Collection partitionInfos = new ArrayList<>();
if (cluster != null) {
for (String topic : this.topics) {
partitionInfos.addAll(cluster.partitionsForTopic(topic));
}
}
return new Cluster(cluster.nodes(), partitionInfos);
}
{code}

Well, there's a null check for cluster, but if cluster is null it will throw 
NPE at the return line by calling {{cluster.nodes()}}! So, I put together a 
quick fix and changed {{MetadataTest}} to reproduce this error.

  was:
While working on another issue I have just seen the following:

{code}
private Cluster getClusterForCurrentTopics(Cluster cluster) {
Collection partitionInfos = new ArrayList<>();
if (cluster != null) {
for (String topic : this.topics) {
partitionInfos.addAll(cluster.partitionsForTopic(topic));
}
}
return new Cluster(cluster.nodes(), partitionInfos);
}
{code}

Well, there's a null check for cluster, but if cluster is null it will throw 
NPE. So, I put together a quick fix and changed {{MetadataTest}} to reproduce 
this error.


> Metadata#getClusterForCurrentTopics can throw NPE even with null checking
> -
>
> Key: KAFKA-2599
> URL: https://issues.apache.org/jira/browse/KAFKA-2599
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: Edward Ribeiro
>Assignee: Edward Ribeiro
>Priority: Minor
> Fix For: 0.8.1.2, 0.9.0.0
>
>
> While working on another issue I have just seen the following:
> {code}
> private Cluster getClusterForCurrentTopics(Cluster cluster) {
> Collection partitionInfos = new ArrayList<>();
> if (cluster != null) {
> for (String topic : this.topics) {
> partitionInfos.addAll(cluster.partitionsForTopic(topic));
> }
> }
> return new Cluster(cluster.nodes(), partitionInfos);
> }
> {code}
> Well, there's a null check for cluster, but if cluster is null it will throw 
> NPE at the return line by calling {{cluster.nodes()}}! So, I put together a 
> quick fix and changed {{MetadataTest}} to reproduce this error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Kafka-trunk #649

[Apache Jenkins Server]

See 

[GitHub] kafka pull request: Client Metadata internal state should be synch...

[eribeiro]

GitHub user eribeiro opened a pull request:

https://github.com/apache/kafka/pull/263

Client Metadata internal state should be synchronized



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eribeiro/kafka KAFKA-2578

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/263.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #263


commit ff5719dbcce8aef03494d2982131a8c8034a5680
Author: Edward Ribeiro 
Date:   2015-09-29T21:23:19Z

Client Metadata internal state should be synchronized




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA 2578 Client Metadata internal state shou...

[eribeiro]

GitHub user eribeiro reopened a pull request:

https://github.com/apache/kafka/pull/263

KAFKA 2578 Client Metadata internal state should be synchronized



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eribeiro/kafka KAFKA-2578

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/263.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #263


commit 67b792339090006589799604a47d183e402b465c
Author: Edward Ribeiro 
Date:   2015-09-29T21:23:19Z

KAFKA-2578 Client Metadata internal state should be synchronized




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA 2578 Client Metadata internal state shou...

[eribeiro]

Github user eribeiro closed the pull request at:

https://github.com/apache/kafka/pull/263


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-2578) Client Metadata internal state should be synchronized

[Edward Ribeiro (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934290#comment-14934290
 ] 

Edward Ribeiro edited comment on KAFKA-2578 at 9/29/15 9:35 PM:


Thanks!

[~jasong35], Hi, Jason! When I pushed to the github repo I forgot to prefix it 
with the "KAFKA-XXX" number, and even after doing a {{commit --amend}} the 
asfbot didn't pick it up to show here on the JIRA. Nevertheless, the PR is at:

https://github.com/apache/kafka/pull/263


was (Author: eribeiro):
Thanks, mate! :)

> Client Metadata internal state should be synchronized
> -
>
> Key: KAFKA-2578
> URL: https://issues.apache.org/jira/browse/KAFKA-2578
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Edward Ribeiro
>Priority: Trivial
>
> Some recent patches introduced a couple new fields in o.a.k.clients.Metadata: 
> 'listeners' and 'needMetadataForAllTopics'. Accessor methods for these fields 
> should be synchronized like the rest of the internal Metadata state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2578) Client Metadata internal state should be synchronized

[Edward Ribeiro (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934290#comment-14934290
 ] 

Edward Ribeiro edited comment on KAFKA-2578 at 9/29/15 9:36 PM:


Thanks!

[~jasong35], Hi, Jason! When I pushed to the github repo I forgot to prefix it 
with the "KAFKA-XXX" number, and even after doing a {{commit --amend}} the 
asfbot didn't pick it up to show here on the JIRA. :( I tried to close and open 
and the PR, but no success either.

Nevertheless, the PR is at:

https://github.com/apache/kafka/pull/263


was (Author: eribeiro):
Thanks!

[~jasong35], Hi, Jason! When I pushed to the github repo I forgot to prefix it 
with the "KAFKA-XXX" number, and even after doing a {{commit --amend}} the 
asfbot didn't pick it up to show here on the JIRA. Nevertheless, the PR is at:

https://github.com/apache/kafka/pull/263

> Client Metadata internal state should be synchronized
> -
>
> Key: KAFKA-2578
> URL: https://issues.apache.org/jira/browse/KAFKA-2578
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Edward Ribeiro
>Priority: Trivial
>
> Some recent patches introduced a couple new fields in o.a.k.clients.Metadata: 
> 'listeners' and 'needMetadataForAllTopics'. Accessor methods for these fields 
> should be synchronized like the rest of the internal Metadata state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2599) Metadata#getClusterForCurrentTopics can throw NPE even with null checking

[Edward Ribeiro (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Ribeiro updated KAFKA-2599:
--
Status: Patch Available  (was: Open)

> Metadata#getClusterForCurrentTopics can throw NPE even with null checking
> -
>
> Key: KAFKA-2599
> URL: https://issues.apache.org/jira/browse/KAFKA-2599
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: Edward Ribeiro
>Assignee: Edward Ribeiro
>Priority: Minor
> Fix For: 0.8.1.2, 0.9.0.0
>
>
> While working on another issue I have just seen the following:
> {code}
> private Cluster getClusterForCurrentTopics(Cluster cluster) {
> Collection partitionInfos = new ArrayList<>();
> if (cluster != null) {
> for (String topic : this.topics) {
> partitionInfos.addAll(cluster.partitionsForTopic(topic));
> }
> }
> return new Cluster(cluster.nodes(), partitionInfos);
> }
> {code}
> Well, there's a null check for cluster, but if cluster is null it will throw 
> NPE at the return line by calling {{cluster.nodes()}}! So, I put together a 
> quick fix and changed {{MetadataTest}} to reproduce this error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2578) Client Metadata internal state should be synchronized

[Edward Ribeiro (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Ribeiro updated KAFKA-2578:
--
Status: Patch Available  (was: Open)

> Client Metadata internal state should be synchronized
> -
>
> Key: KAFKA-2578
> URL: https://issues.apache.org/jira/browse/KAFKA-2578
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Edward Ribeiro
>Priority: Trivial
>
> Some recent patches introduced a couple new fields in o.a.k.clients.Metadata: 
> 'listeners' and 'needMetadataForAllTopics'. Accessor methods for these fields 
> should be synchronized like the rest of the internal Metadata state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2587:Increasing timeout for the test ver...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/260


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2587) Transient test failure: `SimpleAclAuthorizerTest`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936019#comment-14936019
 ] 

ASF GitHub Bot commented on KAFKA-2587:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/260


> Transient test failure: `SimpleAclAuthorizerTest`
> -
>
> Key: KAFKA-2587
> URL: https://issues.apache.org/jira/browse/KAFKA-2587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Parth Brahmbhatt
>
> I've seen `SimpleAclAuthorizerTest ` fail a couple of times since its recent 
> introduction. Here's one such build:
> https://builds.apache.org/job/kafka-trunk-git-pr/576/console
> [~parth.brahmbhatt], can you please take a look and see if it's an easy fix?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2587) Transient test failure: `SimpleAclAuthorizerTest`

[Sriharsha Chintalapani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani resolved KAFKA-2587.
---
   Resolution: Fixed
Fix Version/s: 0.9.0.0

> Transient test failure: `SimpleAclAuthorizerTest`
> -
>
> Key: KAFKA-2587
> URL: https://issues.apache.org/jira/browse/KAFKA-2587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> I've seen `SimpleAclAuthorizerTest ` fail a couple of times since its recent 
> introduction. Here's one such build:
> https://builds.apache.org/job/kafka-trunk-git-pr/576/console
> [~parth.brahmbhatt], can you please take a look and see if it's an easy fix?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Todd Palino]

We should also consider what else should be negotiated between the broker
and the client as this comes together. The version is definitely first, but
there are other things, such as the max message size, that should not need
to be replicated on both the broker and the client. Granted, max message
size has per-topic overrides as well, but that should also be considered
(possibly as an addition to the topic metadata response).

Ideally you never want a requirement that is enforced by the broker to be a
surprise to the client, whether that's a supported version or a
configuration parameter. The client should not have to know it in advance
(except for the most basic of connection parameters), and even if it does
have it as a configuration option, it should be able to know before it even
starts running that what it has configured is in conflict with the server.

-Todd


On Tue, Sep 29, 2015 at 11:08 AM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> Right. But there should be a max old version that the broker should support
> to avoid these incompatibility issues.
> For example, if the broker is at version X, it should be able to support
> the versions (clients and interbroker) till X-2. In case we have brokers
> and clients older than that it can send a response warning them to upgrade
> till X-2 minimum.
> The backward compatibility limit can be discussed further. This will help
> for rolling upgrades.
>
> Thanks,
>
> Mayuresh
>
> On Tue, Sep 29, 2015 at 8:25 AM, Grant Henke  wrote:
>
> > If we create a protocol version negotiation api for clients, can we use
> it
> > to replace or improve the ease of upgrades that break inter-broker
> > messaging?
> >
> > Currently upgrades that break the wire protocol take 2 rolling restarts.
> > The first restart we set inter.broker.protocol.version telling all
> brokers
> > to communicate on the old version, and then we restart again removing the
> > inter.broker.protocol.version. With this api the brokers could agree on a
> > version to communicate with, and when bounced re-negotiate to the new
> > version.
> >
> >
> > On Mon, Sep 28, 2015 at 10:26 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com> wrote:
> >
> > > Nice write-up.
> > >
> > > Just had a question, instead of returning an empty response back to the
> > > client, would it be better for the broker to return a response that
> gives
> > > some more info to the client regarding the min version they need to
> > upgrade
> > > to in order to communicate with the broker.
> > >
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Mon, Sep 28, 2015 at 6:36 PM, Jiangjie Qin
>  > >
> > > wrote:
> > >
> > > > Thanks for the writeup. I also think having a specific protocol for
> > > > client-broker version negotiation is better.
> > > >
> > > > I'm wondering is it better to let the broker to decide the version to
> > > use?
> > > > It might have some value If brokers have preference for a particular
> > > > version.
> > > > Using a global version is a good approach. For the client-broker
> > > > negotiation, I am thinking about something like:
> > > >
> > > > ProtocolSyncRequest => ClientType [ProtocolVersion]
> > > > ClientType => int8
> > > > ProtocolVersion => int16
> > > >
> > > > ProtocolSyncResponse => PreferredVersion
> > > > PreferredVersion => int16
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Mon, Sep 28, 2015 at 11:59 AM, Jun Rao  wrote:
> > > >
> > > > > I agree with Ewen that having the keys explicitly specified in the
> > > > response
> > > > > is better.
> > > > >
> > > > > In addition to the supported protocol version, there are other
> > > > interesting
> > > > > metadata at the broker level that could be of interest to things
> like
> > > > admin
> > > > > tools (e.g., used disk space, remaining disk space, etc). I am
> > > wondering
> > > > if
> > > > > we should separate those into different requests. For inquiring the
> > > > > protocol version, we can have a separate BrokerProtocolRequest. The
> > > > > response will just include the broker version and perhaps a list of
> > > > > supported requests and versions?
> > > > >
> > > > > As for sending an empty response for unrecognized requests, do you
> > how
> > > is
> > > > > that handled in other similar systems?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Sep 28, 2015 at 10:47 AM, Jason Gustafson <
> > ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Having the version API can make clients more robust, so I'm in
> > favor.
> > > > One
> > > > > > note on the addition of the "rack" field. Since this is a
> > > > broker-specific
> > > > > > setting, the client would have to query BrokerMetadata for every
> > new
> > > > > broker
> > > > > > it connects to (unless we also expose rack in TopicMetadata).
> This
> > is
> > > > > also
> > > > > > kind of unfortunate for admin utilities leveraging this API. It
> > might
> > > > be
> > > > > > more con

[jira] [Commented] (KAFKA-2587) Transient test failure: `SimpleAclAuthorizerTest`

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936102#comment-14936102
 ] 

Jason Gustafson commented on KAFKA-2587:


[~parth.brahmbhatt] Can you take a look at this test failure? 
https://builds.apache.org/job/kafka-trunk-git-pr/615/console

Here's the failure snippet:
{code}
kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs FAILED
java.lang.AssertionError: changes not propagated in timeout period. 
expected Set() but got Set(User:bob has Allow permission for operations: Write 
from hosts: host2, User:alice has Allow permission for operations: Write from 
hosts: host1, User:bob has Allow permission for operations: Read from hosts: 
host2)
at org.junit.Assert.fail(Assert.java:88)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:654)
at 
kafka.security.auth.SimpleAclAuthorizerTest.changeAclAndVerify(SimpleAclAuthorizerTest.scala:229)
at 
kafka.security.auth.SimpleAclAuthorizerTest.testAclManagementAPIs(SimpleAclAuthorizerTest.scala:191)
{code}

> Transient test failure: `SimpleAclAuthorizerTest`
> -
>
> Key: KAFKA-2587
> URL: https://issues.apache.org/jira/browse/KAFKA-2587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> I've seen `SimpleAclAuthorizerTest ` fail a couple of times since its recent 
> introduction. Here's one such build:
> https://builds.apache.org/job/kafka-trunk-git-pr/576/console
> [~parth.brahmbhatt], can you please take a look and see if it's an easy fix?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Kafka-trunk #650

[Apache Jenkins Server]

See 

Changes:

[harsha] KAFKA-2587; Increasing timeout for the test verification.

--
[...truncated 5729 lines...]
kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.server.LeaderElectionTest > testLeaderElectionAndEpoch PASSED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.server.ServerShutdownTest > testConsecutiveShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled PASSED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialExists PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathExists PASSED

kafka.zk.ZKPathTest > testCreatePersistentPath PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExistsThrowsException PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentPathThrowsException PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExists PASSED

kafka.api.ProducerBounceTest > testBrokerFailure PASSED

kafka.api.ConsumerTest > testShrinkingTopicSubscriptions PASSED

kafka.api.ConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.ConsumerTest > testSeek PASSED

kafka.api.ConsumerTest > testUnsubscribeTopic PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.api.ConsumerTest > testPatternUnsubscription PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.api.ConsumerTest > testPartitionsFor PASSED

kafka.api.ConsumerTest > testSimpleConsumption PASSED

kafka.api.ConsumerTest > testPartitionPauseAndResume PASSED

kafka.api.ConsumerTest > testPartitionReassignmentCallback PASSED

kafka.api.ConsumerTest > testAutoOffsetReset PASSED

kafka.api.ConsumerTest > testCommitSpecifiedOffsets PASSED

kafka.api.ConsumerTest > testCommitMetadata PASSED

kafka.api.ConsumerTest > testPatternSubscription PASSED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testEquality PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testBindGroupToTopicsAlreadyListenedOn PASSED

kafka.coordinator.CoordinatorMetadataTest > testGetGroup PASSED

kafka.coordinator.CoordinatorMetadataTest > testBindNonexistentGroupToTopics 
PASSED

kafka.coordinator.CoordinatorMetadataTest > testBindGroupToTopicsNotListenedOn 
PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testUnbindNonexistentGroupFromTopics PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testAddGroupReturnsPreexistingGroupIfItAlreadyExists PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testUnbindGroupFromTopicsListenedOnByNoOtherGroup PASSED

kafka.coordinator.CoordinatorMetadataTest > testRemoveNonexistentGroup PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testUnbindGroupFromTopicsNotListenedOn PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testUnbindGroupFromTopicsListenedOnByOtherGroups PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testRemoveGroupWithNoOtherGroupsBoundToItsTopics PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testRemoveGroupWithOtherGroupsBoundToItsTopics PASSED

kafka.coordinator.CoordinatorMetadataTest > testGetNonexistentGroup

Re: [DISCUSS] KIP-31 - Message format change proposal

[Jiangjie Qin]

Hi Joel and other folks.

I updated the KIP page with the two phase roll out, which avoids the
conversion for majority of users.

To do that we need to add a message.format.version configuration to broker.
Other than that there is no interface change from the previous proposal.
Please let me know if you have concern about the updated proposal.

Thanks,

Jiangjie (Becket) Qin

On Fri, Sep 25, 2015 at 11:26 AM, Joel Koshy  wrote:

> Hey Becket,
>
> I do think we need the interim deployment phase, set
> message.format.version and down-convert for producer request v2.
> Down-conversion for v2 is no worse than what the broker is doing now.
> I don't think we want a prolonged phase where we down-convert for
> every v1 fetch - in fact I'm less concerned about losing zero-copy for
> those fetch requests than the overhead of decompress/recompress for
> those fetches as that would increase your CPU usage by 4x, 5x or
> whatever the average consumer fan-out is. The
> decompression/recompression will put further memory pressure as well.
>
> It is true that clients send the latest request version that it is
> compiled with and that does not need to change. The broker can
> continue to send back with zero-copy for fetch request version 2 as
> well (even if during the interim phase during which it down-converts
> producer request v2). The consumer iterator (for old consumer) or the
> Fetcher (for new consumer) needs to be able to handle messages that
> are in original as well as new (relative offset) format.
>
> Thanks,
>
> Joel
>
>
> On Thu, Sep 24, 2015 at 7:56 PM, Jiangjie Qin 
> wrote:
> > Hi Joel,
> >
> > That is a valid concern. And that is actually why we had the
> > message.format.version before.
> >
> > My original thinking was:
> > 1. upgrade the broker to support both V1 and V2 for consumer/producer
> > request.
> > 2. configure broker to store V1 on the disk. (message.format.version = 1)
> > 3. upgrade the consumer to support both V1 and V2 for consumer request.
> > 4. Meanwhile some producer might also be upgraded to use producer request
> > V2.
> > 5. At this point, for producer request V2, broker will do down
> conversion.
> > Regardless consumers are upgraded or not, broker will always use
> zero-copy
> > transfer. Because supposedly both old and upgraded consumer should be
> able
> > to understand that.
> > 6. After most of the consumers are upgraded, We set
> message.format.version
> > = 1 and only do down conversion for old consumers.
> >
> > This way we don't need to reject producer request V2. And we always to
> > version conversion for the minority of the consumers. However I have a
> few
> > concerns over this approach, not sure if they actually matters.
> >
> > A. (5) is not true for now. Today the clients only uses the highest
> > version, i.e. a producer/consumer wouldn't parse a lower version of
> > response even the code exist there. I think supposedly, consumer should
> > stick to one version and broker should do the conversion.
> > B. Let's say (A) is not a concern, we make all the clients support all
> the
> > versions it knows. At step(6), there will be a transitional period that
> > user will see both messages with new and old version. For KIP-31 only it
> > might be OK because we are not adding anything into the message. But if
> the
> > message has different fields (e.g. KIP-32), that means people will get
> > those fields from some messages but not from some other messages. Would
> > that be a problem?
> >
> > If (A) and (B) are not a problem. Is the above procedure able to address
> > your concern?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Sep 24, 2015 at 6:32 PM, Joel Koshy  wrote:
> >
> >> The upgrade plan works, but the potentially long interim phase of
> >> skipping zero-copy for down-conversion could be problematic especially
> >> for large deployments with large consumer fan-out. It is not only
> >> going to be memory overhead but CPU as well - since you need to
> >> decompress, write absolute offsets, then recompress for every v1
> >> fetch. i.e., it may be safer (but obviously more tedious) to have a
> >> multi-step upgrade process. For e.g.,:
> >>
> >> 1 - Upgrade brokers, but disable the feature. i.e., either reject
> >> producer requests v2 or down-convert to old message format (with
> >> absolute offsets)
> >> 2 - Upgrade clients, but they should only use v1 requests
> >> 3 - Switch (all or most) consumers to use v2 fetch format (which will
> >> use zero-copy).
> >> 4 - Turn on the feature on the brokers to allow producer requests v2
> >> 5 - Switch producers to use v2 produce format
> >>
> >> (You may want a v1 fetch rate metric and decide to proceed to step 4
> >> only when that comes down to a trickle)
> >>
> >> I'm not sure if the prolonged upgrade process is viable in every
> >> scenario. I think it should work at LinkedIn for e.g., but may not for
> >> other environments.
> >>
> >> Joel
> >>
> >>
> >> On Tue, Sep 22, 2015 at 12:55 AM, Jiangjie Qin

[jira] [Commented] (KAFKA-2570) New consumer should commit before every rebalance when auto-commit is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936168#comment-14936168
 ] 

ASF GitHub Bot commented on KAFKA-2570:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/257


> New consumer should commit before every rebalance when auto-commit is enabled
> -
>
> Key: KAFKA-2570
> URL: https://issues.apache.org/jira/browse/KAFKA-2570
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.9.0.0
>
>
> If not, then the consumer may see duplicates even on normal rebalances, since 
> we will always reset to the previous commit after rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2570) New consumer should commit before every rebalance when auto-commit is enabled

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2570.
--
Resolution: Fixed

Issue resolved by pull request 257
[https://github.com/apache/kafka/pull/257]

> New consumer should commit before every rebalance when auto-commit is enabled
> -
>
> Key: KAFKA-2570
> URL: https://issues.apache.org/jira/browse/KAFKA-2570
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.9.0.0
>
>
> If not, then the consumer may see duplicates even on normal rebalances, since 
> we will always reset to the previous commit after rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Set `sendTime` in `doSend` instead of `...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/264

MINOR: Set `sendTime` in `doSend` instead of `InFlightRequests.add` and 
rename method names for consistency

@hachikuji @MayureshGharat @jjkoshy Thoughts?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka tweak-send-ms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/264.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #264


commit de29695522ed5c4e408b5b385aa990d099d0c87e
Author: Ismael Juma 
Date:   2015-09-30T00:22:50Z

Set `sendTime` in `doSend` instead of `InFlightRequests.add` and rename 
method names for consistency




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2570: commit offsets on rebalance/close ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/257


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : Kafka-trunk #651

[Apache Jenkins Server]

See 

[GitHub] kafka pull request: KAFKA-2591: fix StreamingMetrics [WIP]

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/265

KAFKA-2591: fix StreamingMetrics [WIP]

Remove state storage upon unclean shutdown and fix streaming metrics used 
for local state.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2591

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/265.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #265


commit 1da221ecbdd9a4cd23662d21e0d805bc1a9bcd41
Author: Guozhang Wang 
Date:   2015-09-30T02:34:45Z

KAFKA-2591.v1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2591) Remove Persistent Data before Restoringafter a Fault

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936273#comment-14936273
 ] 

ASF GitHub Bot commented on KAFKA-2591:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/265

KAFKA-2591: fix StreamingMetrics [WIP]

Remove state storage upon unclean shutdown and fix streaming metrics used 
for local state.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2591

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/265.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #265


commit 1da221ecbdd9a4cd23662d21e0d805bc1a9bcd41
Author: Guozhang Wang 
Date:   2015-09-30T02:34:45Z

KAFKA-2591.v1




> Remove Persistent Data before Restoringafter a Fault
> 
>
> Key: KAFKA-2591
> URL: https://issues.apache.org/jira/browse/KAFKA-2591
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Yasuhiro Matsuda
> Fix For: 0.9.0.0
>
>
> When the checkpoint is missing upon starting up, it should be indicating that 
> the previous run is stopped uncleanly (or it is the first time running this 
> process job), hence any persisted data is not reliable any more and we need 
> to restore from the beginning.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)