[jira] [Created] (KAFKA-1996) Scaladoc error: unknown tag parameter

[Yaguo Zhou (JIRA)]

Yaguo Zhou created KAFKA-1996:
-

 Summary: Scaladoc error: unknown tag parameter
 Key: KAFKA-1996
 URL: https://issues.apache.org/jira/browse/KAFKA-1996
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Yaguo Zhou
Priority: Minor


There are some scala doc error: unknown tag parameter



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1996) Scaladoc error: unknown tag parameter

[Yaguo Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaguo Zhou updated KAFKA-1996:
--
Attachment: scala-doc-unknown-tag-parameter.patch

> Scaladoc error: unknown tag parameter
> -
>
> Key: KAFKA-1996
> URL: https://issues.apache.org/jira/browse/KAFKA-1996
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Yaguo Zhou
>Priority: Minor
>  Labels: doc
> Attachments: scala-doc-unknown-tag-parameter.patch
>
>
> There are some scala doc error: unknown tag parameter



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343109#comment-14343109
 ] 

Tong Li commented on KAFKA-1988:


[~guozhang] Yes, the o.a.k.c.c.u.Utils.abs used in few places.  in patch set 
for issue 1926, I will consolidate both Utils modules from clients and core 
into one. So that we do not have name conflict all over the place. The patch 
set for issue 1926 will be quite big. I would like to get this thing fixed for 
coming up release first, then we can address issue 1926. Thanks.

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

[Joe Stein]

Hey, I just sent out a google hangout invite to all pmc, committers and
everyone I found working on a KIP. If I missed anyone in the invite please
let me know and can update it, np.

We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA
help to make a google account so we can manage better?

To discuss
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
in progress and related JIRA that are interdependent and common work.

~ Joe Stein

On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps  wrote:

> Let's stay on Google hangouts that will also record and make the sessions
> available on youtube.
>
> -Jay
>
> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman 
> wrote:
>
> > Jay / Joe
> >
> > We're happy to send out a Webex for this purpose. We could record the
> > sessions if there is interest and publish them out.
> >
> > Thanks
> >
> > Jeff
> >
> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps  wrote:
> >
> > > Let's try to get the technical hang-ups sorted out, though. I really
> > think
> > > there is some benefit to live discussion vs writing. I am hopeful that
> if
> > > we post instructions and give ourselves a few attempts we can get it
> > > working.
> > >
> > > Tuesday at that time would work for me...any objections?
> > >
> > > -Jay
> > >
> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein 
> wrote:
> > >
> > > > Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT
> 
> > > >
> > > > I don't mind google hangout but there is always some issue or
> whatever
> > so
> > > > we know the apache irc channel works. We can start there and see how
> it
> > > > goes? We can pull transcripts too and associate to tickets if need be
> > > makes
> > > > it helpful for things.
> > > >
> > > > ~ Joestein
> > > >
> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps 
> > wrote:
> > > >
> > > > > We'd talked about doing a Google Hangout to chat about this. What
> > about
> > > > > generalizing that a little further...I actually think it would be
> > good
> > > > for
> > > > > everyone spending a reasonable chunk of their week on Kafka stuff
> to
> > > > maybe
> > > > > sync up once a week. I think we could use time to talk through
> design
> > > > > stuff, make sure we are on top of code reviews, talk through any
> > tricky
> > > > > issues, etc.
> > > > >
> > > > > We can make it publicly available so that any one can follow along
> > who
> > > > > likes.
> > > > >
> > > > > Any interest in doing this? If so I'll try to set it up starting
> next
> > > > week.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi <
> > > > > andrii.bilets...@stealth.ly> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I've updated KIP page, fixed / aligned document structure. Also I
> > > added
> > > > > > some
> > > > > > very initial proposal for AdminClient so we have something to
> start
> > > > from
> > > > > > while
> > > > > > discussing the KIP.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > > >
> > > > > > Thanks,
> > > > > > Andrii Biletskyi
> > > > > >
> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi <
> > > > > > andrii.bilets...@stealth.ly> wrote:
> > > > > >
> > > > > > > Jay,
> > > > > > >
> > > > > > > Re error messages: you are right, in most cases client will
> have
> > > > enough
> > > > > > > context to show descriptive error message. My concern is that
> we
> > > will
> > > > > > have
> > > > > > > to
> > > > > > > add lots of new error codes for each possible error. Of course,
> > we
> > > > > could
> > > > > > > reuse
> > > > > > > some of existing like UknownTopicOrPartitionCode, but we will
> > also
> > > > need
> > > > > > to
> > > > > > > add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both
> > for
> > > > > topic
> > > > > > > name and config, and probably user would like to know what
> > exactly
> > > > > > > is wrong in his config), InvalidReplicaAssignment,
> InternalError
> > > > (e.g.
> > > > > > > zookeeper failure) etc.
> > > > > > > And this is only for TopicCommand, we will also need to add
> > similar
> > > > > stuff
> > > > > > > for
> > > > > > > ReassignPartitions, PreferredReplica. So we'll end up with a
> > large
> > > > list
> > > > > > of
> > > > > > > error codes, used only in Admin protocol.
> > > > > > > Having said that, I agree my proposal is not consistent with
> > other
> > > > > cases.
> > > > > > > Maybe we can find better solution or something in-between.
> > > > > > >
> > > > > > > Re Hangout chat: I think it is a great idea. This way we can
> move
> > > on
> > > > > > > faster.
> > > > > > > Let's agree somehow on date/time so people can join. Will work
> > for
> > > me
> > > > > > this
> > > > > > > and
> > > > > > > next week almost anytime if agreed in advance.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Andrii

[jira] [Commented] (KAFKA-1877) Expose version via JMX for 'new' producer

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343275#comment-14343275
 ] 

Manikumar Reddy commented on KAFKA-1877:


Yes, version info can be exposed as JMX info.   If some one want to 
programmatically retrieve the version  info,  how to retrieve?

> Expose version via JMX for 'new' producer 
> --
>
> Key: KAFKA-1877
> URL: https://issues.apache.org/jira/browse/KAFKA-1877
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.8.2.0
>Reporter: Vladimir Tretyakov
>Assignee: Manikumar Reddy
> Fix For: 0.8.3
>
>
> Add version of Kafka to jmx (monitoring tool can use this info).
> Something like that
> {code}
> kafka.common:type=AppInfo,name=Version
>   Value java.lang.Object = 0.8.2-beta
> {code}
> we already have this in "core" Kafka module (see kafka.common.AppInfo object).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1884) New Producer blocks forever for Invalid topic names

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1884:
---
Assignee: Manikumar Reddy
  Status: Patch Available  (was: Open)

> New Producer blocks forever for Invalid topic names
> ---
>
> Key: KAFKA-1884
> URL: https://issues.apache.org/jira/browse/KAFKA-1884
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2.0
>Reporter: Manikumar Reddy
>Assignee: Manikumar Reddy
> Fix For: 0.8.3
>
> Attachments: KAFKA-1884.patch
>
>
> New producer blocks forever for invalid topics names
> producer logs:
> DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50845,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,416] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50845.
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50846,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,417] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50846.
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,418] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50847,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,418] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50847.
> Broker logs:
> [2015-01-20 12:46:14,074] ERROR [KafkaApi-0] error when handling request 
> Name: TopicMetadataRequest; Version: 0; CorrelationId: 51020; ClientId: 
> my-producer; Topics: TOPIC= (kafka.server.KafkaApis)
> kafka.common.InvalidTopicException: topic name TOPIC= is illegal, contains a 
> character other than ASCII alphanumerics, '.', '_' and '-'
>   at kafka.common.Topic$.validate(Topic.scala:42)
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:186)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:177)
>   at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:367)
>   at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:350)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at 
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
>   at scala.collection.SetLike$class.map(SetLike.scala:93)
>   at scala.collection.AbstractSet.map(Set.scala:47)
>   at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:350)
>   at 
> kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:389)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:57)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
>   at java.lang.Thread.run(Thread.java:722)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1884) New Producer blocks forever for Invalid topic names

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343294#comment-14343294
 ] 

Manikumar Reddy commented on KAFKA-1884:


Created reviewboard https://reviews.apache.org/r/31627/diff/
 against branch origin/trunk

> New Producer blocks forever for Invalid topic names
> ---
>
> Key: KAFKA-1884
> URL: https://issues.apache.org/jira/browse/KAFKA-1884
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2.0
>Reporter: Manikumar Reddy
> Fix For: 0.8.3
>
> Attachments: KAFKA-1884.patch
>
>
> New producer blocks forever for invalid topics names
> producer logs:
> DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50845,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,416] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50845.
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50846,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,417] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50846.
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,418] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50847,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,418] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50847.
> Broker logs:
> [2015-01-20 12:46:14,074] ERROR [KafkaApi-0] error when handling request 
> Name: TopicMetadataRequest; Version: 0; CorrelationId: 51020; ClientId: 
> my-producer; Topics: TOPIC= (kafka.server.KafkaApis)
> kafka.common.InvalidTopicException: topic name TOPIC= is illegal, contains a 
> character other than ASCII alphanumerics, '.', '_' and '-'
>   at kafka.common.Topic$.validate(Topic.scala:42)
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:186)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:177)
>   at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:367)
>   at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:350)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at 
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
>   at scala.collection.SetLike$class.map(SetLike.scala:93)
>   at scala.collection.AbstractSet.map(Set.scala:47)
>   at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:350)
>   at 
> kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:389)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:57)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
>   at java.lang.Thread.run(Thread.java:722)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31627: Patch for KAFKA-1884

[Manikumar Reddy O]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31627/
---

Review request for kafka.


Bugs: KAFKA-1884
https://issues.apache.org/jira/browse/KAFKA-1884


Repository: kafka


Description
---

Added logs to print metadata response errors


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
a7fa4a9dfbcfbc4d9e9259630253cbcded158064 

Diff: https://reviews.apache.org/r/31627/diff/


Testing
---


Thanks,

Manikumar Reddy O

[jira] [Updated] (KAFKA-1884) New Producer blocks forever for Invalid topic names

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-1884:
---
Attachment: KAFKA-1884.patch

> New Producer blocks forever for Invalid topic names
> ---
>
> Key: KAFKA-1884
> URL: https://issues.apache.org/jira/browse/KAFKA-1884
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2.0
>Reporter: Manikumar Reddy
> Fix For: 0.8.3
>
> Attachments: KAFKA-1884.patch
>
>
> New producer blocks forever for invalid topics names
> producer logs:
> DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50845,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,416] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50845.
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50846,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,417] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50846.
> DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying 
> to send metadata request to node -1
> DEBUG [2015-01-20 12:46:13,418] NetworkClient: maybeUpdateMetadata(): Sending 
> metadata request ClientRequest(expectResponse=true, payload=null, 
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=50847,client_id=my-producer},
>  body={topics=[TOPIC=]})) to node -1
> TRACE [2015-01-20 12:46:13,418] NetworkClient: handleMetadataResponse(): 
> Ignoring empty metadata response with correlation id 50847.
> Broker logs:
> [2015-01-20 12:46:14,074] ERROR [KafkaApi-0] error when handling request 
> Name: TopicMetadataRequest; Version: 0; CorrelationId: 51020; ClientId: 
> my-producer; Topics: TOPIC= (kafka.server.KafkaApis)
> kafka.common.InvalidTopicException: topic name TOPIC= is illegal, contains a 
> character other than ASCII alphanumerics, '.', '_' and '-'
>   at kafka.common.Topic$.validate(Topic.scala:42)
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:186)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:177)
>   at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:367)
>   at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:350)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at 
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
>   at scala.collection.SetLike$class.map(SetLike.scala:93)
>   at scala.collection.AbstractSet.map(Set.scala:47)
>   at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:350)
>   at 
> kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:389)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:57)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
>   at java.lang.Thread.run(Thread.java:722)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

[Gwen Shapira]

Thanks for sending this out Joe. Looking forward to chatting with everyone :)

On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein  wrote:
> Hey, I just sent out a google hangout invite to all pmc, committers and
> everyone I found working on a KIP. If I missed anyone in the invite please
> let me know and can update it, np.
>
> We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA
> help to make a google account so we can manage better?
>
> To discuss
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> in progress and related JIRA that are interdependent and common work.
>
> ~ Joe Stein
>
> On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps  wrote:
>
>> Let's stay on Google hangouts that will also record and make the sessions
>> available on youtube.
>>
>> -Jay
>>
>> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman 
>> wrote:
>>
>> > Jay / Joe
>> >
>> > We're happy to send out a Webex for this purpose. We could record the
>> > sessions if there is interest and publish them out.
>> >
>> > Thanks
>> >
>> > Jeff
>> >
>> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps  wrote:
>> >
>> > > Let's try to get the technical hang-ups sorted out, though. I really
>> > think
>> > > there is some benefit to live discussion vs writing. I am hopeful that
>> if
>> > > we post instructions and give ourselves a few attempts we can get it
>> > > working.
>> > >
>> > > Tuesday at that time would work for me...any objections?
>> > >
>> > > -Jay
>> > >
>> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein 
>> wrote:
>> > >
>> > > > Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT
>> 
>> > > >
>> > > > I don't mind google hangout but there is always some issue or
>> whatever
>> > so
>> > > > we know the apache irc channel works. We can start there and see how
>> it
>> > > > goes? We can pull transcripts too and associate to tickets if need be
>> > > makes
>> > > > it helpful for things.
>> > > >
>> > > > ~ Joestein
>> > > >
>> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps 
>> > wrote:
>> > > >
>> > > > > We'd talked about doing a Google Hangout to chat about this. What
>> > about
>> > > > > generalizing that a little further...I actually think it would be
>> > good
>> > > > for
>> > > > > everyone spending a reasonable chunk of their week on Kafka stuff
>> to
>> > > > maybe
>> > > > > sync up once a week. I think we could use time to talk through
>> design
>> > > > > stuff, make sure we are on top of code reviews, talk through any
>> > tricky
>> > > > > issues, etc.
>> > > > >
>> > > > > We can make it publicly available so that any one can follow along
>> > who
>> > > > > likes.
>> > > > >
>> > > > > Any interest in doing this? If so I'll try to set it up starting
>> next
>> > > > week.
>> > > > >
>> > > > > -Jay
>> > > > >
>> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi <
>> > > > > andrii.bilets...@stealth.ly> wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > I've updated KIP page, fixed / aligned document structure. Also I
>> > > added
>> > > > > > some
>> > > > > > very initial proposal for AdminClient so we have something to
>> start
>> > > > from
>> > > > > > while
>> > > > > > discussing the KIP.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Andrii Biletskyi
>> > > > > >
>> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi <
>> > > > > > andrii.bilets...@stealth.ly> wrote:
>> > > > > >
>> > > > > > > Jay,
>> > > > > > >
>> > > > > > > Re error messages: you are right, in most cases client will
>> have
>> > > > enough
>> > > > > > > context to show descriptive error message. My concern is that
>> we
>> > > will
>> > > > > > have
>> > > > > > > to
>> > > > > > > add lots of new error codes for each possible error. Of course,
>> > we
>> > > > > could
>> > > > > > > reuse
>> > > > > > > some of existing like UknownTopicOrPartitionCode, but we will
>> > also
>> > > > need
>> > > > > > to
>> > > > > > > add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both
>> > for
>> > > > > topic
>> > > > > > > name and config, and probably user would like to know what
>> > exactly
>> > > > > > > is wrong in his config), InvalidReplicaAssignment,
>> InternalError
>> > > > (e.g.
>> > > > > > > zookeeper failure) etc.
>> > > > > > > And this is only for TopicCommand, we will also need to add
>> > similar
>> > > > > stuff
>> > > > > > > for
>> > > > > > > ReassignPartitions, PreferredReplica. So we'll end up with a
>> > large
>> > > > list
>> > > > > > of
>> > > > > > > error codes, used only in Admin protocol.
>> > > > > > > Having said that, I agree my proposal is not consistent with
>> > other
>> > > > > cases.
>> > > > > > > Maybe we can find better solution or something in-between.
>> > > > > > >
>> > > > > > > Re Hangout chat: I think it is a great idea. Thi

[jira] [Created] (KAFKA-1997) Refactor Mirror Maker

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-1997:
---

 Summary: Refactor Mirror Maker
 Key: KAFKA-1997
 URL: https://issues.apache.org/jira/browse/KAFKA-1997
 Project: Kafka
  Issue Type: Improvement
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin


Refactor mirror maker based on KIP-3



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 28481: Patch for KAFKA-1792

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28481/#review74760
---



core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala


I'm not sure if this makes sense. Basically, the entire point of 
--rebalance is to figure out the best balanced replica placement with the 
minimum number of replicas moved. If you ask the user to list the topics or the 
brokers, this may not lead to the most balanced replica placement in the 
cluster. If we did this, then the only thing the user would want to do is limit 
the number of replicas moved in one go, in order to manually throttle the data 
movement in the cluster. It is ok to do that in a separate JIRA. 

Same with the replace broker use case. Replacing a broker is much easier to 
use if it is a separate option (--replace-broker --from-broker 1 --to-broker 
2). Though if you want to cover that in a separate JIRA, that's fine.


- Neha Narkhede


On Feb. 26, 2015, 2:58 p.m., Dmitry Pekar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28481/
> ---
> 
> (Updated Feb. 26, 2015, 2:58 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1792
> https://issues.apache.org/jira/browse/KAFKA-1792
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1792: CR
> 
> 
> KAFKA-1792: CR2
> 
> 
> KAFKA-1792: merge of KAFKA-1753
> 
> 
> KAFKA-1792: generate renamed to rebalance
> 
> 
> KAFKA-1792: --rebalance renamed back to --generate, removed 
> --decomission-broker command
> 
> 
> KAFKA-1792: added back --decommission-broker command
> 
> 
> KAFKA-1792: --generate renamed back to --rebalance
> 
> 
> KAFKA-1792: added old --generate command for compatibility
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> b700110f2d7f1ede235af55d8e37e1b5592c6c7d 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 979992b68af3723cd229845faff81c641123bb88 
>   core/src/test/scala/unit/kafka/admin/AdminTest.scala 
> e28979827110dfbbb92fe5b152e7f1cc973de400 
>   topics.json ff011ed381e781b9a177036001d44dca3eac586f 
> 
> Diff: https://reviews.apache.org/r/28481/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dmitry Pekar
> 
>

Apache Samza Meetup - March 4 @6PM hosted at LinkedIn's campus in Mountain View CA

[Ed Yakabosky]

Hi all -

I would like to announce the first Bay Area Apache Samza 
Meetup hosted at 
LinkedIn in Mountain View, CA on March 4, 2015 @6PM.  We plan to host the event 
every 2-months to encourage knowledge sharing & collaboration in Samza’s 
usage and open 
source community.

The agenda for the meetup is::

  *   6:00 – 6:15PM: Doors open, sign NDAs, networking, food & drinks
  *   6:15 - 6:45PM: Naveen Somasundaram (LinkedIn) – Getting Started with 
Apache Samza
  *   6:45 - 7:30PM: Shekar Tippur (Intuit) – Powering Contextual Alerts for 
Intuit’s Operations Center with Apache Samza
  *   7:30 - 8:00PM: Chris Riccomini (LinkedIn) – Where we’re headed next: 
Apache Samza Roadmap

We plan to provide food & drinks so please RSVP 
here to help us 
with estimation.  Please let me know if you have any questions or ideas for 
future meet ups.

We plan to announce a live stream the day of the event for remote attendance.

Excited to see you there!
Ed Yakabosky

[BCC:
Kafka Open Source
Samza Open Source
LinkedIn’s DDS and DAI teams
Linkedin’s Samza customers
Tech-Talk]

[jira] [Commented] (KAFKA-1997) Refactor Mirror Maker

[Jonathan Creasy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343488#comment-14343488
 ] 

Jonathan Creasy commented on KAFKA-1997:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement

> Refactor Mirror Maker
> -
>
> Key: KAFKA-1997
> URL: https://issues.apache.org/jira/browse/KAFKA-1997
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>
> Refactor mirror maker based on KIP-3



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29467: Patch for KAFKA-1660

[Parth Brahmbhatt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/
---

(Updated March 2, 2015, 6:41 p.m.)


Review request for kafka.


Bugs: KAFKA-1660
https://issues.apache.org/jira/browse/KAFKA-1660


Repository: kafka


Description (updated)
---

Merge remote-tracking branch 'origin/trunk' into KAFKA-1660

Conflicts:

clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java

Merge remote-tracking branch 'origin/trunk' into KAFKA-1660


Changing log levels as suggested.


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
7397e565fd865214529ffccadd4222d835ac8110 
  clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
6913090af03a455452b0b5c3df78f266126b3854 
  clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
ed9c63a6679e3aaf83d19fde19268553a4c107c2 

Diff: https://reviews.apache.org/r/29467/diff/


Testing
---

existing unit tests passed.


Thanks,

Parth Brahmbhatt

[jira] [Updated] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[Parth Brahmbhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Brahmbhatt updated KAFKA-1660:

Attachment: KAFKA-1660_2015-03-02_10:41:49.patch

> Ability to call close() with a timeout on the Java Kafka Producer. 
> ---
>
> Key: KAFKA-1660
> URL: https://issues.apache.org/jira/browse/KAFKA-1660
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2.0
>Reporter: Andrew Stein
>Assignee: Parth Brahmbhatt
> Fix For: 0.8.3
>
> Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch, 
> KAFKA-1660_2015-03-02_10:41:49.patch
>
>
> I would like the ability to call {{close}} with a timeout on the Java 
> Client's KafkaProducer.
> h6. Workaround
> Currently, it is possible to ensure that {{close}} will return quickly by 
> first doing a {{future.get(timeout)}} on the last future produced on each 
> partition, but this means that the user has to define the partitions up front 
> at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[Parth Brahmbhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343552#comment-14343552
 ] 

Parth Brahmbhatt commented on KAFKA-1660:
-

Updated reviewboard https://reviews.apache.org/r/29467/diff/
 against branch origin/trunk

> Ability to call close() with a timeout on the Java Kafka Producer. 
> ---
>
> Key: KAFKA-1660
> URL: https://issues.apache.org/jira/browse/KAFKA-1660
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2.0
>Reporter: Andrew Stein
>Assignee: Parth Brahmbhatt
> Fix For: 0.8.3
>
> Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch, 
> KAFKA-1660_2015-03-02_10:41:49.patch
>
>
> I would like the ability to call {{close}} with a timeout on the Java 
> Client's KafkaProducer.
> h6. Workaround
> Currently, it is possible to ensure that {{close}} will return quickly by 
> first doing a {{future.get(timeout)}} on the last future produced on each 
> partition, but this means that the user has to define the partitions up front 
> at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29467: Patch for KAFKA-1660

[Parth Brahmbhatt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74777
---



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


Changed log level as suggested.



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


included.



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


changed log level to suggested value.


- Parth Brahmbhatt


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

Review Request 31633: Patch for KAFKA-902

[Geoffrey Anderson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31633/
---

Review request for kafka.


Bugs: KAFKA-902
https://issues.apache.org/jira/browse/KAFKA-902


Repository: kafka


Description
---

Add simple unit test for ClientUtils.randomizeBackoff


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/ClientUtils.java 
d0da5d7a08a0c3e67e0fe14bb0b0e7c73380f416 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
67ceb754a52c07143c69b053fe128b9e24060b99 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
7397e565fd865214529ffccadd4222d835ac8110 
  clients/src/test/java/org/apache/kafka/clients/ClientUtilsTest.java 
13ce519f03d13db041e1f2dbcd6b59414d2775b8 

Diff: https://reviews.apache.org/r/31633/diff/


Testing
---


Thanks,

Geoffrey Anderson

[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests

[Geoffrey Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343607#comment-14343607
 ] 

Geoffrey Anderson commented on KAFKA-902:
-

Created reviewboard https://reviews.apache.org/r/31633/diff/
 against branch origin/trunk

> Randomize backoff on the clients for metadata requests
> --
>
> Key: KAFKA-902
> URL: https://issues.apache.org/jira/browse/KAFKA-902
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Geoffrey Anderson
>Priority: Critical
>  Labels: newbie
> Attachments: KAFKA-902.patch
>
>
> If a Kafka broker dies and there are a large number of clients talking to the 
> Kafka cluster, each of the clients can end up shooting metadata requests at 
> around the same time. It is better to randomize the backoff on the clients so 
> the metadata requests are more evenly spread out



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests

[Geoffrey Anderson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Anderson updated KAFKA-902:

Status: Patch Available  (was: Open)

> Randomize backoff on the clients for metadata requests
> --
>
> Key: KAFKA-902
> URL: https://issues.apache.org/jira/browse/KAFKA-902
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Geoffrey Anderson
>Priority: Critical
>  Labels: newbie
> Attachments: KAFKA-902.patch
>
>
> If a Kafka broker dies and there are a large number of clients talking to the 
> Kafka cluster, each of the clients can end up shooting metadata requests at 
> around the same time. It is better to randomize the backoff on the clients so 
> the metadata requests are more evenly spread out



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests

[Geoffrey Anderson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Anderson updated KAFKA-902:

Attachment: KAFKA-902.patch

> Randomize backoff on the clients for metadata requests
> --
>
> Key: KAFKA-902
> URL: https://issues.apache.org/jira/browse/KAFKA-902
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Geoffrey Anderson
>Priority: Critical
>  Labels: newbie
> Attachments: KAFKA-902.patch
>
>
> If a Kafka broker dies and there are a large number of clients talking to the 
> Kafka cluster, each of the clients can end up shooting metadata requests at 
> around the same time. It is better to randomize the backoff on the clients so 
> the metadata requests are more evenly spread out



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31591: Patch for KAFKA-1992

[Jiangjie Qin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31591/#review74827
---

Ship it!


LGTM, just a minor comment.


core/src/main/scala/kafka/cluster/Partition.scala


This part seems now serving only logging purpose. If that is the case, can 
we make it even clearer. For example, print all the acked replicas instead of 
just a number.


- Jiangjie Qin


On March 1, 2015, 7:58 a.m., Gwen Shapira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31591/
> ---
> 
> (Updated March 1, 2015, 7:58 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1992
> https://issues.apache.org/jira/browse/KAFKA-1992
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> remove unnecessary requiredAcks parameter and clean up few comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> c4bf48a801007ebe7497077d2018d6dffe1677d4 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 4d763bf05efb24a394662721292fc54d32467969 
> 
> Diff: https://reviews.apache.org/r/31591/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Gwen Shapira
> 
>

Re: How to get a JIRA assigned

[Guozhang Wang]

Filed INFRA-9219 for this.

On Mon, Feb 23, 2015 at 5:29 AM, Joe Stein  wrote:

> Jay,
>
> I thought it was the same issue like with confluence and comments and why
> we have to grant rights for that. Bots coming and reassigning everything to
> them or something in JIRA.
>
> We could ask/open a ticket with INFRA, if nothing else maybe help come up
> with a different way to solve it.
>
> ~ Joestein
>
> On Sun, Feb 22, 2015 at 7:31 PM, Jay Kreps  wrote:
>
> > Anyone know if there a way to turn this off? Is it possible to configure
> > JIRA to let anyone assign them? Unlike the other lockdown stuff which
> > prevents spam this doesn't seem like it could be a spam vector and it
> would
> > be awesome to make it easier for people.
> >
> > -Jay
> >
> > On Sun, Feb 22, 2015 at 3:43 PM, Guozhang Wang 
> wrote:
> >
> > > Hi Jonathan,
> > >
> > > You need to be added to the "contributor" list before can be assigned
> to
> > > jiras, and only committers can do that for you.
> > >
> > > I have just add you to the list so you should be able to assign
> yourself
> > > now.
> > >
> > > Guozhang
> > >
> > > On Sun, Feb 22, 2015 at 3:08 PM, Jonathan Rafalski <
> > > jonathan.rafal...@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > >   I was wondering if there are any rights to be able to assign JIRA
> > > > tickets to myself?  I found what I think is a bug while working on
> 1679
> > > so
> > > > I opened a ticket and was going to assign a review board for both
> with
> > my
> > > > solution but now some else has attempted a patch.  Just want to be
> able
> > > to
> > > > assign a ticket to me so time isn't wasted.
> > > >
> > > > If it is something that I need to be granted after submitting a few
> > > > patches that are accepted can someone at least assign 1679 and 1972
> to
> > me
> > > > so nobody else attempts to work while I am?
> > > >
> > > > Thanks!
> > > >
> > > > Jonathan.
> > > >
> > > > Sent from my iPhone
> > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>



-- 
-- Guozhang

Re: Review Request 29467: Patch for KAFKA-1660

[Jiangjie Qin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74833
---

Ship it!


LGTM.

- Jiangjie Qin


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

Re: Review Request 29467: Patch for KAFKA-1660

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74835
---


Could we add some unit tests for this new API as I mentioned in my previous 
comment?

- Guozhang Wang


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

Re: Unit tests in java7 vs java8

[Guozhang Wang]

Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest
modified a bit):

 JDK 8 

Total time: 18 mins 3.649 secs

real18m4.091s
user0m7.105s
sys0m0.426s

 JDK 7 

Total time: 18 mins 55.546 secs

real18m55.997s
user0m4.157s
sys0m0.341s



Guozhang



On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang 
wrote:

> I am working on the test handing / NPE / failure issues of ConsumerTest
> only.
>
> I currently run Java 8 and the unit test takes about 10 minutes, I can do
> "time ./gradlew test" with both versions and see if there is a clear
> difference.
>
> Guozhang
> 
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Wednesday, February 25, 2015 4:53 PM
> To: dev@kafka.apache.org; Guozhang Wang
> Subject: Re: Unit tests in java7 vs java8
>
> Yeah, hey Guozhang, is that fix part of the larger consumer patch you just
> posted or is that a separate issue?
>
> -Jay
>
> On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira  > wrote:
> The Consumer tests are currently hanging :(
>
> I think Guozhang is working on a solution. I'm commenting them out until
> the problem is resolved...
>
>
>
> On Wed, Feb 25, 2015 at 4:00 PM, Tong Li  liton...@us.ibm.com>> wrote:
>
> > Gwen,
> > I have not tried Java 8. Still on Java 7, but I always run into the
> > test hung problems (no errors on the screen and the system is completely
> > idle), it may be a different problem. I can recreate that problem every
> > time when I run "gradle --daemon testAll", I recall that couple of weeks
> > ago there was one patch saying fixed the problem, but I am still seeing
> the
> > problem with latest code. What I noticed is that seems tests always stop
> at
> > one of the ConsumerTest test cases. What puzzled me the most is that it
> was
> > not always a particular test case. Being very new in this community, I
> > think that error must be something related to my env. Here is my
> > environment:
> >
> >  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
> > big enough max lock memory,
> >
> > not complaining, just some observations in case you wonder what other
> > developers may face.
> >
> > Thanks.
> >
> > Tong Li
> > OpenStack & Kafka Community Development
> > Building 501/B205
> > liton...@us.ibm.com
> >
> > [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
> > PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
> > Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I
> > just noticed that they take almost twice
> >
> > From: Gwen Shapira mailto:gshap...@cloudera.com>>
> > To: "dev@kafka.apache.org" <
> dev@kafka.apache.org>
> > Date: 02/25/2015 03:47 PM
> > Subject: Unit tests in java7 vs java8
> > --
> >
> >
> >
> > Hi,
> >
> > Anyone running tests on Java 8? I just noticed that they take almost
> twice
> > as long to run compared to Java 7 (at least on my box, and with Scala
> > 2.10.4).
> >
> > Anyone else noticed this? Maybe even did some digging on the causes?
> >
> > Gwen
> >
> >
>
>


-- 
-- Guozhang

[jira] [Created] (KAFKA-1998) Partitions Missing From MetadataResponse

[Evan Huus (JIRA)]

Evan Huus created KAFKA-1998:


 Summary: Partitions Missing From MetadataResponse
 Key: KAFKA-1998
 URL: https://issues.apache.org/jira/browse/KAFKA-1998
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.2.0
Reporter: Evan Huus


It is known behaviour that when a partition is entirely offline (it has no 
leader because all of its replicas are down) then that partition will not be 
included in the metadata returned by other brokers. For example, if topic "foo" 
has 3 partitions, but all replicas of partition 3 are offline, then requesting 
metadata for "foo" will only return information about partitions 1 and 2.

This means that there is no way to reliably determine the number of partitions 
for a topic via kafka's metadata API; if I receive information on partitions 1 
and 2, I don't know if partition 3 is offline or if it is simply that there are 
only two partitions total. (You can presumably still ask zookeeper directly, 
but that is a work-around).

This ambiguity, in turn, can lead to a consistency problem with the default 
partitioner, since that effectively implements `hash(key) mod #partitions`. If 
a partition goes offline and is removed from the metadata response, then the 
number of partitions the producer knows about will change (on its next metadata 
refresh) and the mapping from keys to partitions will also change. Instead of 
distributing messages among (for example) 3 partitions, and failing to produce 
to the offline partition, it will distribute *all* messages among the two 
online partitions. This results in messages being sent to the wrong partition.

Since kafka already returns partitions with error messages in many cases (e.g. 
`LeaderNotAvailable`) I think it makes much more sense and fixes the above 
partition problem if it would simply return offline partitions as well with the 
appropriate error (whether that is `LeaderNotAvailable` or it would be better 
to add an additional error is up to you).

CC [~guozhang]

(This issue was originally described/discussed on the kafka-users mailing list, 
in the thread involving 
https://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3CCAA4pprAZvp2XhdNmy0%2BqVZ1UVdVxmUfz3DDArhGbwP-iiH%2BGyg%40mail.gmail.com%3E)

If there are any questions I am happy to clarify, I realize the scenario is 
somewhat complex.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-1998) Partitions Missing From MetadataResponse

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat reassigned KAFKA-1998:
--

Assignee: Mayuresh Gharat

> Partitions Missing From MetadataResponse
> 
>
> Key: KAFKA-1998
> URL: https://issues.apache.org/jira/browse/KAFKA-1998
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Evan Huus
>Assignee: Mayuresh Gharat
>
> It is known behaviour that when a partition is entirely offline (it has no 
> leader because all of its replicas are down) then that partition will not be 
> included in the metadata returned by other brokers. For example, if topic 
> "foo" has 3 partitions, but all replicas of partition 3 are offline, then 
> requesting metadata for "foo" will only return information about partitions 1 
> and 2.
> This means that there is no way to reliably determine the number of 
> partitions for a topic via kafka's metadata API; if I receive information on 
> partitions 1 and 2, I don't know if partition 3 is offline or if it is simply 
> that there are only two partitions total. (You can presumably still ask 
> zookeeper directly, but that is a work-around).
> This ambiguity, in turn, can lead to a consistency problem with the default 
> partitioner, since that effectively implements `hash(key) mod #partitions`. 
> If a partition goes offline and is removed from the metadata response, then 
> the number of partitions the producer knows about will change (on its next 
> metadata refresh) and the mapping from keys to partitions will also change. 
> Instead of distributing messages among (for example) 3 partitions, and 
> failing to produce to the offline partition, it will distribute *all* 
> messages among the two online partitions. This results in messages being sent 
> to the wrong partition.
> Since kafka already returns partitions with error messages in many cases 
> (e.g. `LeaderNotAvailable`) I think it makes much more sense and fixes the 
> above partition problem if it would simply return offline partitions as well 
> with the appropriate error (whether that is `LeaderNotAvailable` or it would 
> be better to add an additional error is up to you).
> CC [~guozhang]
> (This issue was originally described/discussed on the kafka-users mailing 
> list, in the thread involving 
> https://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3CCAA4pprAZvp2XhdNmy0%2BqVZ1UVdVxmUfz3DDArhGbwP-iiH%2BGyg%40mail.gmail.com%3E)
> If there are any questions I am happy to clarify, I realize the scenario is 
> somewhat complex.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29467: Patch for KAFKA-1660

[Jiangjie Qin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74843
---


Actually I spoke too fast... As the flush() has been checked in, we need to 
take care of the caller thread that are doing a flush when invoking close().
This is a little bit tricky. If we close the producer forcibily when caller 
thread were doing a flush, we have to notify the caller thread that the flush 
failed. The simplest way might be letting flush return a boolean value. So we 
do the following:
1. In RecordAccumulator add a new forceClose(), it sets an forceClosed flag 
first, then clear up the imcomplete batchset and wake up all the caller threads.
2. In RecordAccumulator.awaitFlushCompletion(), it checks the forceClosed flag 
to determine whether flush succeeded or not and return the result to 
KafkaProducer.flush().
3. KafkaProducer.flush() return this result to caller threads.


clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java


We probably need to release the caller threads that are waiting on flush() 
at this point.


- Jiangjie Qin


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

Re: How to get a JIRA assigned

[Neha Narkhede]

Thanks Guozhang!

On Mon, Mar 2, 2015 at 1:59 PM, Guozhang Wang  wrote:

> Filed INFRA-9219 for this.
>
> On Mon, Feb 23, 2015 at 5:29 AM, Joe Stein  wrote:
>
> > Jay,
> >
> > I thought it was the same issue like with confluence and comments and why
> > we have to grant rights for that. Bots coming and reassigning everything
> to
> > them or something in JIRA.
> >
> > We could ask/open a ticket with INFRA, if nothing else maybe help come up
> > with a different way to solve it.
> >
> > ~ Joestein
> >
> > On Sun, Feb 22, 2015 at 7:31 PM, Jay Kreps  wrote:
> >
> > > Anyone know if there a way to turn this off? Is it possible to
> configure
> > > JIRA to let anyone assign them? Unlike the other lockdown stuff which
> > > prevents spam this doesn't seem like it could be a spam vector and it
> > would
> > > be awesome to make it easier for people.
> > >
> > > -Jay
> > >
> > > On Sun, Feb 22, 2015 at 3:43 PM, Guozhang Wang 
> > wrote:
> > >
> > > > Hi Jonathan,
> > > >
> > > > You need to be added to the "contributor" list before can be assigned
> > to
> > > > jiras, and only committers can do that for you.
> > > >
> > > > I have just add you to the list so you should be able to assign
> > yourself
> > > > now.
> > > >
> > > > Guozhang
> > > >
> > > > On Sun, Feb 22, 2015 at 3:08 PM, Jonathan Rafalski <
> > > > jonathan.rafal...@gmail.com> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > >   I was wondering if there are any rights to be able to assign JIRA
> > > > > tickets to myself?  I found what I think is a bug while working on
> > 1679
> > > > so
> > > > > I opened a ticket and was going to assign a review board for both
> > with
> > > my
> > > > > solution but now some else has attempted a patch.  Just want to be
> > able
> > > > to
> > > > > assign a ticket to me so time isn't wasted.
> > > > >
> > > > > If it is something that I need to be granted after submitting a few
> > > > > patches that are accepted can someone at least assign 1679 and 1972
> > to
> > > me
> > > > > so nobody else attempts to work while I am?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Jonathan.
> > > > >
> > > > > Sent from my iPhone
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>



-- 
Thanks,
Neha

Re: Unit tests in java7 vs java8

[Gwen Shapira]

I guess its just my machine then.

Thanks!

On Mon, Mar 2, 2015 at 2:47 PM, Guozhang Wang  wrote:
> Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest
> modified a bit):
>
>  JDK 8 
>
> Total time: 18 mins 3.649 secs
>
> real18m4.091s
> user0m7.105s
> sys0m0.426s
>
>  JDK 7 
>
> Total time: 18 mins 55.546 secs
>
> real18m55.997s
> user0m4.157s
> sys0m0.341s
>
> 
>
> Guozhang
>
>
>
> On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang 
> wrote:
>
>> I am working on the test handing / NPE / failure issues of ConsumerTest
>> only.
>>
>> I currently run Java 8 and the unit test takes about 10 minutes, I can do
>> "time ./gradlew test" with both versions and see if there is a clear
>> difference.
>>
>> Guozhang
>> 
>> From: Jay Kreps [jay.kr...@gmail.com]
>> Sent: Wednesday, February 25, 2015 4:53 PM
>> To: dev@kafka.apache.org; Guozhang Wang
>> Subject: Re: Unit tests in java7 vs java8
>>
>> Yeah, hey Guozhang, is that fix part of the larger consumer patch you just
>> posted or is that a separate issue?
>>
>> -Jay
>>
>> On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira > > wrote:
>> The Consumer tests are currently hanging :(
>>
>> I think Guozhang is working on a solution. I'm commenting them out until
>> the problem is resolved...
>>
>>
>>
>> On Wed, Feb 25, 2015 at 4:00 PM, Tong Li > liton...@us.ibm.com>> wrote:
>>
>> > Gwen,
>> > I have not tried Java 8. Still on Java 7, but I always run into the
>> > test hung problems (no errors on the screen and the system is completely
>> > idle), it may be a different problem. I can recreate that problem every
>> > time when I run "gradle --daemon testAll", I recall that couple of weeks
>> > ago there was one patch saying fixed the problem, but I am still seeing
>> the
>> > problem with latest code. What I noticed is that seems tests always stop
>> at
>> > one of the ConsumerTest test cases. What puzzled me the most is that it
>> was
>> > not always a particular test case. Being very new in this community, I
>> > think that error must be something related to my env. Here is my
>> > environment:
>> >
>> >  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
>> > big enough max lock memory,
>> >
>> > not complaining, just some observations in case you wonder what other
>> > developers may face.
>> >
>> > Thanks.
>> >
>> > Tong Li
>> > OpenStack & Kafka Community Development
>> > Building 501/B205
>> > liton...@us.ibm.com
>> >
>> > [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
>> > PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
>> > Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I
>> > just noticed that they take almost twice
>> >
>> > From: Gwen Shapira mailto:gshap...@cloudera.com>>
>> > To: "dev@kafka.apache.org" <
>> dev@kafka.apache.org>
>> > Date: 02/25/2015 03:47 PM
>> > Subject: Unit tests in java7 vs java8
>> > --
>> >
>> >
>> >
>> > Hi,
>> >
>> > Anyone running tests on Java 8? I just noticed that they take almost
>> twice
>> > as long to run compared to Java 7 (at least on my box, and with Scala
>> > 2.10.4).
>> >
>> > Anyone else noticed this? Maybe even did some digging on the causes?
>> >
>> > Gwen
>> >
>> >
>>
>>
>
>
> --
> -- Guozhang

Fwd: patch set 1988

[Tong Li]

Folks, 
> Do not want to nag you, but wonder if any of you has couple of minutes to 
> review patch set for 1988 again so that I do not have to rebase this so many 
> times. Guozhang already +1ed(thanks Guozhang!) Here are the links for your 
> convenience. 
> 
> The issue
> https://issues.apache.org/jira/browse/KAFKA-1988
> 
> The patch set
> https://reviews.apache.org/r/31566/diff/
> 
> 
> Tong Li
> OpenStack & Kafka Community Development
> Building 501/B205
> liton...@us.ibm.com

Re: Unit tests in java7 vs java8

[Jay Kreps]

Wow, 18 mins?

I was seeing 8 mins a couple weeks ago and 12 mins now. Anyone know what's
up? Not sure if the 12<=>18 is just because I have SSDs or what. It is
really easy to make a small change that adds a few hundred ms of startup or
shutdown time and that have that multiply by 500 server start and stops in
the test execution.

-Jay

On Mon, Mar 2, 2015 at 2:47 PM, Guozhang Wang  wrote:

> Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest
> modified a bit):
>
>  JDK 8 
>
> Total time: 18 mins 3.649 secs
>
> real18m4.091s
> user0m7.105s
> sys0m0.426s
>
>  JDK 7 
>
> Total time: 18 mins 55.546 secs
>
> real18m55.997s
> user0m4.157s
> sys0m0.341s
>
> 
>
> Guozhang
>
>
>
> On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang  >
> wrote:
>
> > I am working on the test handing / NPE / failure issues of ConsumerTest
> > only.
> >
> > I currently run Java 8 and the unit test takes about 10 minutes, I can do
> > "time ./gradlew test" with both versions and see if there is a clear
> > difference.
> >
> > Guozhang
> > 
> > From: Jay Kreps [jay.kr...@gmail.com]
> > Sent: Wednesday, February 25, 2015 4:53 PM
> > To: dev@kafka.apache.org; Guozhang Wang
> > Subject: Re: Unit tests in java7 vs java8
> >
> > Yeah, hey Guozhang, is that fix part of the larger consumer patch you
> just
> > posted or is that a separate issue?
> >
> > -Jay
> >
> > On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira  > > wrote:
> > The Consumer tests are currently hanging :(
> >
> > I think Guozhang is working on a solution. I'm commenting them out until
> > the problem is resolved...
> >
> >
> >
> > On Wed, Feb 25, 2015 at 4:00 PM, Tong Li  > liton...@us.ibm.com>> wrote:
> >
> > > Gwen,
> > > I have not tried Java 8. Still on Java 7, but I always run into the
> > > test hung problems (no errors on the screen and the system is
> completely
> > > idle), it may be a different problem. I can recreate that problem every
> > > time when I run "gradle --daemon testAll", I recall that couple of
> weeks
> > > ago there was one patch saying fixed the problem, but I am still seeing
> > the
> > > problem with latest code. What I noticed is that seems tests always
> stop
> > at
> > > one of the ConsumerTest test cases. What puzzled me the most is that it
> > was
> > > not always a particular test case. Being very new in this community, I
> > > think that error must be something related to my env. Here is my
> > > environment:
> > >
> > >  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
> > > big enough max lock memory,
> > >
> > > not complaining, just some observations in case you wonder what other
> > > developers may face.
> > >
> > > Thanks.
> > >
> > > Tong Li
> > > OpenStack & Kafka Community Development
> > > Building 501/B205
> > > liton...@us.ibm.com
> > >
> > > [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
> > > PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
> > > Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java
> 8? I
> > > just noticed that they take almost twice
> > >
> > > From: Gwen Shapira mailto:gshap...@cloudera.com
> >>
> > > To: "dev@kafka.apache.org" <
> > dev@kafka.apache.org>
> > > Date: 02/25/2015 03:47 PM
> > > Subject: Unit tests in java7 vs java8
> > > --
> > >
> > >
> > >
> > > Hi,
> > >
> > > Anyone running tests on Java 8? I just noticed that they take almost
> > twice
> > > as long to run compared to Java 7 (at least on my box, and with Scala
> > > 2.10.4).
> > >
> > > Anyone else noticed this? Maybe even did some digging on the causes?
> > >
> > > Gwen
> > >
> > >
> >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] 0.8.2.1 Candidate 2

[Jun Rao]

+1 from me. Verified quickstart and unit tests.

Thanks,

Jun

On Thu, Feb 26, 2015 at 2:59 PM, Jun Rao  wrote:

> This is the second candidate for release of Apache Kafka 0.8.2.1. This
> fixes 4 critical issue in 0.8.2.0.
>
> Release Notes for the 0.8.2.1 release
>
> https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Mar 2, 3pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/javadoc/
>
> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bd1bfb63ec73c10d08432ac893a23f28281ea021
> (git commit ee1267b127f3081db491fa1bf9a287084c324e36)
>
> /***
>
> Thanks,
>
> Jun
>
>

[jira] [Comment Edited] (KAFKA-1910) Refactor KafkaConsumer

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336985#comment-14336985
 ] 

Guozhang Wang edited comment on KAFKA-1910 at 3/3/15 12:45 AM:
---

The uploaded patch contains multiple fixes to the related JIRAs as well as 
refactoring the new consumer itself. I will summarize them here instead of in 
the RB:

1. Fix ConsumerTest.testXXXwithBrokerFailure: in RestartDeadBroker we need to 
call startup() on the old brokers instead of creating new ones as the last 
approach will case the metadata to be mess up and cause the test to hang 
(KAFKA-1948). Also make sure the "test" topic is created with correct 
replication factor to avoid hanging when the only replica broker was shutdown. 
Also make the bouncing of the brokers in the background thread so that it will 
eventually be restarted.

2. Fix ConsumerTest's __consumer_offsets topic: when we call partitionFor() the 
__consumer_offsets topic may be created with replication as 
min(offsetTopicRaplicationFactor, aliveBrokers.size), see KAFKA-1864 for 
details (KAFKA-1975). 

3. Add the IllegalGeneration logic in the coordinator as it is important for 
consumers rebalancing after rediscovering the coordinator, in the current stub 
it always return OK and hence consumers migrating to the new coordinator will 
not trigger rebalance (KAFKA-1964).

4. Create the Coodinator and the FetchManager modules as KafkaConsumer 
internals. Coordinator is responsible for assign partitions (join groups), 
commit offsets and fetch offsets from coordinator, and FetchManager is 
responsible for handling fetch request / responses.

4.1 After the refactoring it is easier to detect and fix a bug where response 
callbacks being triggered multiple times, causing the coordinator NPE 
(KAFKA-1969).

4.2 Avoid always trying to fetch offsets from coordinator whenever the consumer 
decides to update fetch positions, introduce a few new variables / APIs in 
SubscriptionState accordingly.

4.3 Move serializer / de-serializer configs / constructors to AbstractConfig.

4.4 Add missing error handling in commit offset / heartbeat responses. In 
general I think we should make notes about possible error codes in each of the 
response type to help coding error handling logic, has filed KAFKA-1985 for 
that.


was (Author: guozhang):
The uploaded patch contains multiple fixes to the related JIRAs as well as 
refactoring the new consumer itself. I will summarize them here instead of in 
the RB:

1. Fix ConsumerTest.testXXXwithBrokerFailure: in RestartDeadBroker we need to 
call startup() on the old brokers instead of creating new ones as the last 
approach will case the metadata to be mess up and cause the test to hang 
(KAFKA-1948). Also make sure the "test" topic is created with correct 
replication factor to avoid hanging when the only replica broker was shutdown.

2. Fix ConsumerTest's __consumer_offsets topic: when we call partitionFor() the 
__consumer_offsets topic may be created with replication as 
min(offsetTopicRaplicationFactor, aliveBrokers.size), see KAFKA-1864 for 
details (KAFKA-1975). 

3. Add the IllegalGeneration logic in the coordinator as it is important for 
consumers rebalancing after rediscovering the coordinator, in the current stub 
it always return OK and hence consumers migrating to the new coordinator will 
not trigger rebalance (KAFKA-1964).

4. Create the Coodinator and the FetchManager modules as KafkaConsumer 
internals. Coordinator is responsible for assign partitions (join groups), 
commit offsets and fetch offsets from coordinator, and FetchManager is 
responsible for handling fetch request / responses.

4.1 After the refactoring it is easier to detect and fix a bug where response 
callbacks being triggered multiple times, causing the coordinator NPE 
(KAFKA-1969).

4.2 Avoid always trying to fetch offsets from coordinator whenever the consumer 
decides to update fetch positions, introduce a few new variables / APIs in 
SubscriptionState accordingly.

4.3 Move serializer / de-serializer configs / constructors to AbstractConfig.

4.4 Add missing error handling in commit offset / heartbeat responses. In 
general I think we should make notes about possible error codes in each of the 
response type to help coding error handling logic, has filed KAFKA-1985 for 
that.

> Refactor KafkaConsumer
> --
>
> Key: KAFKA-1910
> URL: https://issues.apache.org/jira/browse/KAFKA-1910
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> KafkaConsumer now contains all the logic on the consumer side, making it a 
> very huge class file, better re-factoring it to have multiple layers on top 
> of KafkaClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#633

[jira] [Updated] (KAFKA-1910) Refactor KafkaConsumer

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1910:
-
Attachment: KAFKA-1910.patch

> Refactor KafkaConsumer
> --
>
> Key: KAFKA-1910
> URL: https://issues.apache.org/jira/browse/KAFKA-1910
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Attachments: KAFKA-1910.patch
>
>
> KafkaConsumer now contains all the logic on the consumer side, making it a 
> very huge class file, better re-factoring it to have multiple layers on top 
> of KafkaClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1910) Refactor KafkaConsumer

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1910:
-
Status: Patch Available  (was: Open)

> Refactor KafkaConsumer
> --
>
> Key: KAFKA-1910
> URL: https://issues.apache.org/jira/browse/KAFKA-1910
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Attachments: KAFKA-1910.patch
>
>
> KafkaConsumer now contains all the logic on the consumer side, making it a 
> very huge class file, better re-factoring it to have multiple layers on top 
> of KafkaClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31650: Drag Coordinator and FetchManager out of KafkaConsumer, fix a bunch of consumer test issues

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31650/
---

Review request for kafka.


Bugs: KAFKA-1910
https://issues.apache.org/jira/browse/KAFKA-1910


Repository: kafka


Description
---

See comments in KAFKA-1910


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
06fcfe62cc1fe76f58540221698ef076fe150e96 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
8a3e55aaff7d8c26e56a8407166a4176c1da2644 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
a7fa4a9dfbcfbc4d9e9259630253cbcded158064 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
5fb21001abd77cac839bd724afa04e377a3e82aa 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
67ceb754a52c07143c69b053fe128b9e24060b99 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchManager.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Heartbeat.java
 ee0751e4949120d114202c2299d49612a89b9d97 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java
 d41d3068c11d4b5c640467dc0ae1b7c20a8d128c 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
7397e565fd865214529ffccadd4222d835ac8110 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
122375c473bf73caf05299b9f5174c6b226ca863 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
6baad9366a1975dbaba1786da91efeaa38533319 
  clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
ad2171f5417c93194f5f234bdc7fdd0b8d59a8a8 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
e67c4c8332cb1dd3d9cde5de687df7760045dfe6 
  clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java 
0057496228feeeccbc0c009a42f5268fa2cb8611 
  clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java 
8c50e9be534c61ecf56106bf2b68cf678ea50d66 
  clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java 
52b1803d8b558c1eeb978ba8821496c7d3c20a6b 
  
clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java 
cfac47a4a05dc8a535595542d93e55237b7d1e93 
  clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 
90f31413d7d80a06c0af359009cc271aa0c67be3 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
 4d3b9ececee4b4c0b50ba99da2ddbbb15f9cc08d 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java 
edbed5880dc44fc178737a5e298c106a00f38443 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
a00dcdf15d1c7bac7228be140647bd7d849deb9b 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
8f1a7a625e4eeafa44bbf9e5cff987de86c949be 
  core/src/main/scala/kafka/common/ErrorMapping.scala 
eedc2f5f21dd8755fba891998456351622e17047 
  core/src/main/scala/kafka/common/NoOffsetsCommittedException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 
4cabffeacea09a49913505db19a96a55d58c0909 
  core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 
21790a5059ee00d6610be6f0389445327b88db1d 
  core/src/main/scala/kafka/coordinator/ConsumerRegistry.scala 
b65c04d0a5d53bf92299d5f67f112be3da3bf77d 
  core/src/main/scala/kafka/coordinator/DelayedHeartbeat.scala 
b1248e95d8a648b461f604c154879cc95dc7b1cb 
  core/src/main/scala/kafka/coordinator/GroupRegistry.scala 
7d17e102235134b6312271c4061abd27d7177f7e 
  core/src/main/scala/kafka/server/KafkaServer.scala 
426e522fc9819a0fc0f4e8269033552d716eb066 
  core/src/test/scala/integration/kafka/api/ConsumerTest.scala PRE-CREATION 
  core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 
5650b4a7b950b48af3e272947bfb5e271c4238c9 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
ba48a636dd0b0ed06646d56bb36aa3d43228604f 
  core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 
dc0512b526e914df7e7581b27df18f498da428e2 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
a2bb8855c3c0586b6b45b53ce534edee31b3bd12 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
6ce18076f6b5deb05b51c25be5bed9957e6b4339 

Diff: https://reviews.apache.org/r/31650/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344174#comment-14344174
 ] 

Guozhang Wang commented on KAFKA-1910:
--

Created reviewboard https://reviews.apache.org/r/31650/diff/
 against branch origin/trunk

> Refactor KafkaConsumer
> --
>
> Key: KAFKA-1910
> URL: https://issues.apache.org/jira/browse/KAFKA-1910
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Attachments: KAFKA-1910.patch
>
>
> KafkaConsumer now contains all the logic on the consumer side, making it a 
> very huge class file, better re-factoring it to have multiple layers on top 
> of KafkaClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Unit tests in java7 vs java8

[Gwen Shapira]

Total time: 14 mins 57.037 secs

And I'm running with SSD.

On Mon, Mar 2, 2015 at 4:34 PM, Jay Kreps  wrote:
> Wow, 18 mins?
>
> I was seeing 8 mins a couple weeks ago and 12 mins now. Anyone know what's
> up? Not sure if the 12<=>18 is just because I have SSDs or what. It is
> really easy to make a small change that adds a few hundred ms of startup or
> shutdown time and that have that multiply by 500 server start and stops in
> the test execution.
>
> -Jay
>
> On Mon, Mar 2, 2015 at 2:47 PM, Guozhang Wang  wrote:
>
>> Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest
>> modified a bit):
>>
>>  JDK 8 
>>
>> Total time: 18 mins 3.649 secs
>>
>> real18m4.091s
>> user0m7.105s
>> sys0m0.426s
>>
>>  JDK 7 
>>
>> Total time: 18 mins 55.546 secs
>>
>> real18m55.997s
>> user0m4.157s
>> sys0m0.341s
>>
>> 
>>
>> Guozhang
>>
>>
>>
>> On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang > >
>> wrote:
>>
>> > I am working on the test handing / NPE / failure issues of ConsumerTest
>> > only.
>> >
>> > I currently run Java 8 and the unit test takes about 10 minutes, I can do
>> > "time ./gradlew test" with both versions and see if there is a clear
>> > difference.
>> >
>> > Guozhang
>> > 
>> > From: Jay Kreps [jay.kr...@gmail.com]
>> > Sent: Wednesday, February 25, 2015 4:53 PM
>> > To: dev@kafka.apache.org; Guozhang Wang
>> > Subject: Re: Unit tests in java7 vs java8
>> >
>> > Yeah, hey Guozhang, is that fix part of the larger consumer patch you
>> just
>> > posted or is that a separate issue?
>> >
>> > -Jay
>> >
>> > On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira > > > wrote:
>> > The Consumer tests are currently hanging :(
>> >
>> > I think Guozhang is working on a solution. I'm commenting them out until
>> > the problem is resolved...
>> >
>> >
>> >
>> > On Wed, Feb 25, 2015 at 4:00 PM, Tong Li > > liton...@us.ibm.com>> wrote:
>> >
>> > > Gwen,
>> > > I have not tried Java 8. Still on Java 7, but I always run into the
>> > > test hung problems (no errors on the screen and the system is
>> completely
>> > > idle), it may be a different problem. I can recreate that problem every
>> > > time when I run "gradle --daemon testAll", I recall that couple of
>> weeks
>> > > ago there was one patch saying fixed the problem, but I am still seeing
>> > the
>> > > problem with latest code. What I noticed is that seems tests always
>> stop
>> > at
>> > > one of the ConsumerTest test cases. What puzzled me the most is that it
>> > was
>> > > not always a particular test case. Being very new in this community, I
>> > > think that error must be something related to my env. Here is my
>> > > environment:
>> > >
>> > >  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
>> > > big enough max lock memory,
>> > >
>> > > not complaining, just some observations in case you wonder what other
>> > > developers may face.
>> > >
>> > > Thanks.
>> > >
>> > > Tong Li
>> > > OpenStack & Kafka Community Development
>> > > Building 501/B205
>> > > liton...@us.ibm.com
>> > >
>> > > [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
>> > > PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
>> > > Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java
>> 8? I
>> > > just noticed that they take almost twice
>> > >
>> > > From: Gwen Shapira mailto:gshap...@cloudera.com
>> >>
>> > > To: "dev@kafka.apache.org" <
>> > dev@kafka.apache.org>
>> > > Date: 02/25/2015 03:47 PM
>> > > Subject: Unit tests in java7 vs java8
>> > > --
>> > >
>> > >
>> > >
>> > > Hi,
>> > >
>> > > Anyone running tests on Java 8? I just noticed that they take almost
>> > twice
>> > > as long to run compared to Java 7 (at least on my box, and with Scala
>> > > 2.10.4).
>> > >
>> > > Anyone else noticed this? Maybe even did some digging on the causes?
>> > >
>> > > Gwen
>> > >
>> > >
>> >
>> >
>>
>>
>> --
>> -- Guozhang
>>

[jira] [Updated] (KAFKA-1882) Create extendable channel interface and default implementations

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1882:
-
Status: Patch Available  (was: Open)

> Create extendable channel interface and default implementations
> ---
>
> Key: KAFKA-1882
> URL: https://issues.apache.org/jira/browse/KAFKA-1882
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>Priority: Blocker
> Fix For: 0.8.3
>
>
> For the security infrastructure, we need an extendible interface to replace 
> SocketChannel.
> KAFKA-1684 suggests extending SocketChannel itself, but since SocketChannel 
> is part of Java's standard library, the interface changes between different 
> Java versions, so extending it directly can become a compatibility issue.
> Instead, we can implement a KafkaChannel interface, which will implement 
> connect(), read(), write() and possibly other methods we use. 
> We will replace direct use of SocketChannel in our code with use of 
> KafkaChannel.
> Different implementations of KafkaChannel will be instantiated based on the 
> port/SecurityProtocol configuration. 
> This patch will provide at least the PLAINTEXT implementation for 
> KafkaChannel.
> I will validate that the SSL implementation in KAFKA-1684 can be refactored 
> to use a KafkaChannel interface rather than extend SocketChannel directly. 
> However, the patch will not include the SSL channel itself.
> The interface should also include setters/getters for principal and remote 
> IP, which will be used for the authentication code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1882) Create extendable channel interface and default implementations

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1882:
-
 Priority: Blocker  (was: Major)
Fix Version/s: 0.8.3

supported in this patch https://issues.apache.org/jira/browse/KAFKA-1809 with 
PLAINTEXT as the default implementation. The KIP has been accepted too 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs

> Create extendable channel interface and default implementations
> ---
>
> Key: KAFKA-1882
> URL: https://issues.apache.org/jira/browse/KAFKA-1882
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>Priority: Blocker
> Fix For: 0.8.3
>
>
> For the security infrastructure, we need an extendible interface to replace 
> SocketChannel.
> KAFKA-1684 suggests extending SocketChannel itself, but since SocketChannel 
> is part of Java's standard library, the interface changes between different 
> Java versions, so extending it directly can become a compatibility issue.
> Instead, we can implement a KafkaChannel interface, which will implement 
> connect(), read(), write() and possibly other methods we use. 
> We will replace direct use of SocketChannel in our code with use of 
> KafkaChannel.
> Different implementations of KafkaChannel will be instantiated based on the 
> port/SecurityProtocol configuration. 
> This patch will provide at least the PLAINTEXT implementation for 
> KafkaChannel.
> I will validate that the SSL implementation in KAFKA-1684 can be refactored 
> to use a KafkaChannel interface rather than extend SocketChannel directly. 
> However, the patch will not include the SSL channel itself.
> The interface should also include setters/getters for principal and remote 
> IP, which will be used for the authentication code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31366: Patch for KAFKA-1461

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31366/#review74866
---



core/src/main/scala/kafka/server/AbstractFetcherThread.scala


OffsetAndDelay / OffsetAndState is a bit confusing, maybe we can just use 
PartitionFetchState?



core/src/main/scala/kafka/server/AbstractFetcherThread.scala


It seems we do not need to pass in the OffsetAndDelay object here as we 
will create new one anyways. We can still pass in Long, and with that 
OffsetAndDelay is just internal to AbstractFetcherThread.



core/src/main/scala/kafka/server/OffsetAndDelay.scala


Maybe we can just put this case class into AbstractFetcherThread and expose 
to AbstractFetcherManager.



core/src/main/scala/kafka/server/ReplicaFetcherThread.scala


Are these imports necessary?



core/src/main/scala/kafka/server/ReplicaFetcherThread.scala


Is this intentional?


- Guozhang Wang


On Feb. 24, 2015, 6:02 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31366/
> ---
> 
> (Updated Feb. 24, 2015, 6:02 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1461
> https://issues.apache.org/jira/browse/KAFKA-1461
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1461. Replica fetcher thread does not implement any back-off behavior.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/AbstractFetcherManager.scala 
> 20c00cb8cc2351950edbc8cb1752905a0c26e79f 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> 8c281d4668f92eff95a4a5df3c03c4b5b20e7095 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 14bf3216bae030331bdf76b3266ed0e73526c3de 
>   core/src/main/scala/kafka/server/OffsetAndDelay.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
> 6879e730282185bda3d6bc3659cb15af0672cecf 
>   core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala 
> da4bafc1e2a94a436efe395aab1888fc21e55748 
> 
> Diff: https://reviews.apache.org/r/31366/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344463#comment-14344463
 ] 

Grant Henke commented on KAFKA-902:
---

A few thoughts on the patch:
Should the "jitter" be added to 'reconnect.backoff.ms' too? 
Would there ever be a good reason to change the jitter value from 10? Should it 
be added to the CommonClientConfigs?





> Randomize backoff on the clients for metadata requests
> --
>
> Key: KAFKA-902
> URL: https://issues.apache.org/jira/browse/KAFKA-902
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Geoffrey Anderson
>Priority: Critical
>  Labels: newbie
> Attachments: KAFKA-902.patch
>
>
> If a Kafka broker dies and there are a large number of clients talking to the 
> Kafka cluster, each of the clients can end up shooting metadata requests at 
> around the same time. It is better to randomize the backoff on the clients so 
> the metadata requests are more evenly spread out



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29467: Patch for KAFKA-1660

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74884
---



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


It's probably worth adding an
  if(timeout > 0)
on this.



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


This seems to call initiateClose() twice, once in initiateClose and then 
again from forceClose. This seems like it depends on all the things getting 
closed being idempotent to repeated calls (e.g. record accumulator etc). Would 
it make more sense to have forceClose() just set the force flag?


Two minor changes I noted, but otherwise looks good to me. Needs some unit 
tests, as you mentioned.

- Jay Kreps


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344481#comment-14344481
 ] 

Jay Kreps commented on KAFKA-1660:
--

[~parth.brahmbhatt] Yeah this was exactly what I was thinking. It would be good 
to add some tests for it and kick off the KIP discussion.

[~guozhang] It looks to me like this should work if called from within a 
Callback, but I think you guys would have to specifically try that case or add 
a unit test for it. It would be good if you guys can do a pass on the code 
review once there are some tests.

> Ability to call close() with a timeout on the Java Kafka Producer. 
> ---
>
> Key: KAFKA-1660
> URL: https://issues.apache.org/jira/browse/KAFKA-1660
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2.0
>Reporter: Andrew Stein
>Assignee: Parth Brahmbhatt
> Fix For: 0.8.3
>
> Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch, 
> KAFKA-1660_2015-03-02_10:41:49.patch
>
>
> I would like the ability to call {{close}} with a timeout on the Java 
> Client's KafkaProducer.
> h6. Workaround
> Currently, it is possible to ensure that {{close}} will return quickly by 
> first doing a {{future.get(timeout)}} on the last future produced on each 
> partition, but this means that the user has to define the partitions up front 
> at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344500#comment-14344500
 ] 

Jay Kreps commented on KAFKA-902:
-

This looks good to me. I'd second Grant's comments:
1. I agree we should probably make it configurable and mark the configuration 
low importance. This kind of configuration is hyper-annoying because no one 
will ever set it but it's probably the right thing to do.
2. We should definitely apply the same thing to the reconnect backoff as well 
as metadata max age (if everyone disconnects at time X they will all expire 
their metadata at X+metadata.max.age.ms so jittering that will help too).

Another thing is that this jitter is only additive, so if you configure a 
backoff of 10 ms, your observed backoff time will be 15 ms. Also 10 ms will be 
a bit large if you configure a 1 ms backoff and zero ends up being kind of 
magical. I don't think this is really too terrible and it is simple, so maybe 
we should just leave it.

Another possibility would be something like using a jitter that is a random int 
in +/- min(20, 0.2 * backoff_ms). 

> Randomize backoff on the clients for metadata requests
> --
>
> Key: KAFKA-902
> URL: https://issues.apache.org/jira/browse/KAFKA-902
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Geoffrey Anderson
>Priority: Critical
>  Labels: newbie
> Attachments: KAFKA-902.patch
>
>
> If a Kafka broker dies and there are a large number of clients talking to the 
> Kafka cluster, each of the clients can end up shooting metadata requests at 
> around the same time. It is better to randomize the backoff on the clients so 
> the metadata requests are more evenly spread out



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31566: Patch for KAFKA-1988

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/#review74889
---

Ship it!


This looks good assuming the other patch, centralizes the scala code to all use 
this single abs function.

- Jay Kreps


On Feb. 27, 2015, 11:16 p.m., Tong Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31566/
> ---
> 
> (Updated Feb. 27, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1988
> https://issues.apache.org/jira/browse/KAFKA-1988
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
> for negative numbers
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
>  dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
>   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> 69530c187cd1c41b8173b61de6f982aafe65c9fe 
>   clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
> 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 
> 
> Diff: https://reviews.apache.org/r/31566/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tong Li
> 
>

[jira] [Updated] (KAFKA-1973) Remove the accidentally created LogCleanerManager.scala.orig

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-1973:
---
Status: Patch Available  (was: Open)

Removes core/src/main/scala/kafka/log/LogCleanerManager.scala.orig

> Remove the accidentally created LogCleanerManager.scala.orig
> 
>
> Key: KAFKA-1973
> URL: https://issues.apache.org/jira/browse/KAFKA-1973
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
> Attachments: KAFKA-1973.patch
>
>
> It seems there is a LogCleanerManager.scala.orig in the trunk now. Need to 
> remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1973) Remove the accidentally created LogCleanerManager.scala.orig

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-1973:
---
Attachment: KAFKA-1973.patch

> Remove the accidentally created LogCleanerManager.scala.orig
> 
>
> Key: KAFKA-1973
> URL: https://issues.apache.org/jira/browse/KAFKA-1973
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
> Attachments: KAFKA-1973.patch
>
>
> It seems there is a LogCleanerManager.scala.orig in the trunk now. Need to 
> remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31566: Patch for KAFKA-1988

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/#review74893
---


Thanks for the patch. A couple of minor comments below.


clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java


Perhaps we can change the comment to the following.

A cheap way to deterministically convert a number to a positive value. When 
the input number is negative, the returned positive value is not the absolute 
value of the input though.



clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java


We can just say it returns a positive number.


- Jun Rao


On Feb. 27, 2015, 11:16 p.m., Tong Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31566/
> ---
> 
> (Updated Feb. 27, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1988
> https://issues.apache.org/jira/browse/KAFKA-1988
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
> for negative numbers
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
>  dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
>   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> 69530c187cd1c41b8173b61de6f982aafe65c9fe 
>   clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
> 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 
> 
> Diff: https://reviews.apache.org/r/31566/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tong Li
> 
>

[jira] [Resolved] (KAFKA-952) a broker should unregister certain ZK watchers afte it is no longer the controller

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-952.
---
Resolution: Duplicate

This is already fixed in KAFKA-1578.

> a broker should unregister certain ZK watchers afte it is no longer the 
> controller
> --
>
> Key: KAFKA-952
> URL: https://issues.apache.org/jira/browse/KAFKA-952
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8.1
>Reporter: Jun Rao
>Assignee: Geoffrey Anderson
>  Labels: newbie
>
> It seems that we only register watchers in the controller logic, but never 
> deregister any watchers. Technically, after a broker stops becoming a 
> controller, the only watcher that it needs to keep registering is on the 
> controller path. The rest of the watchers can be deregistered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344572#comment-14344572
 ] 

Jun Rao commented on KAFKA-1994:


Ashish,

The code path for creating a topic may not be optimized. Could you just test 
the cost of createPersistent() with and w/o the patch? Thanks,

> Evaluate performance effect of chroot check on Topic creation
> -
>
> Key: KAFKA-1994
> URL: https://issues.apache.org/jira/browse/KAFKA-1994
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks 
> if namespace exists before trying to create a path in ZK. This raises a 
> concern that checking namespace for each path creation might be unnecessary 
> and can potentially make creations expensive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29467: Patch for KAFKA-1660

[Jay Kreps]

On March 3, 2015, 4:10 a.m., Parth Brahmbhatt wrote:
> > Two minor changes I noted, but otherwise looks good to me. Needs some unit 
> > tests, as you mentioned.

Actually one probably I didn't think of is that forceClose() leaves the 
in-flight requests forever incomplete. A better approach would be to fail them 
all with TimeoutException.


- Jay


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74884
---


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

Re: Review Request 29467: Patch for KAFKA-1660

[Jay Kreps]

> On March 2, 2015, 11:04 p.m., Jiangjie Qin wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java,
> >  line 219
> > 
> >
> > We probably need to release the caller threads that are waiting on 
> > flush() at this point.

Making flush a boolean method that you have to always check to see if someone 
called close() in another thead would be a really really really painful api to 
use in practice, right?

I think the issue here is actually what I pointed out in the other comment, 
namely that in-flight requests area actually left incomplete when you call 
close and hit the forceClose timeout. Any other thread blocking on these 
futures would block forever.

The right solution is just to fail all requests that haven't completed when 
forceClose kicks in. This then fullfills the criteria for flush which is that 
all the requests are completed or failed.


- Jay


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74843
---


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

Re: Review Request 29467: Patch for KAFKA-1660

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74897
---



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


Now there is a bit of duplicate code between the two close methods. Maybe 
this would be cleaner if we just made

public void close() {
close(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
}


- Jay Kreps


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

Re: Review Request 31369: Patch for KAFKA-1982

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31369/#review74895
---



clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java


Could we add a unit test for Integer Ser/DeSer?



clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java


Incorrect comment.



clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java


Incorrect comment.



examples/src/main/java/kafka/examples/Producer.java


We should handle the case when metadata is null.


- Jun Rao


On Feb. 27, 2015, 7:08 p.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31369/
> ---
> 
> (Updated Feb. 27, 2015, 7:08 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1982
> https://issues.apache.org/jira/browse/KAFKA-1982
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1982: change kafka.examples.Producer to use the new java producer
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java
>  PRE-CREATION 
>   examples/src/main/java/kafka/examples/Consumer.java 
> 13135b954f3078eeb7394822b0db25470b746f03 
>   examples/src/main/java/kafka/examples/Producer.java 
> 96e98933148d07564c1b30ba8e805e2433c2adc8 
> 
> Diff: https://reviews.apache.org/r/31369/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>

Re: Review Request 31369: Patch for KAFKA-1982

[Jun Rao]

> On Feb. 27, 2015, 7:29 p.m., Gwen Shapira wrote:
> > Thanks for the patch, Ashish. Its shaping up to be a very useful example. 
> > Two comments:
> > 
> > 1. I think the ser/de should be part of the example and not in "common", 
> > I'm not sure integer ser/de is useful enough to be distributed with Kafka 
> > (although Jun can correct me if I got this wrong).
> > 
> > 2. I saw a lot of discussion on the mailing list around using the new 
> > producer async vs. sync. This example shows the async path. Do we want to 
> > add another "sync" example where we do something like:
> > val future = producer.send(new ProducerRecord(topic,
> >  messageNo,
> >  messageStr), new DemoCallBack(startTime, messageNo, 
> > messageStr));
> > // this waits for send to complete
> > future.get

Gwen,

Integer may be a common type for keys. So, it probably makes sense to include 
Integer ser/de in common.

I agree that it would be useful to add a "sync" example.


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31369/#review74553
---


On Feb. 27, 2015, 7:08 p.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31369/
> ---
> 
> (Updated Feb. 27, 2015, 7:08 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1982
> https://issues.apache.org/jira/browse/KAFKA-1982
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1982: change kafka.examples.Producer to use the new java producer
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java
>  PRE-CREATION 
>   examples/src/main/java/kafka/examples/Consumer.java 
> 13135b954f3078eeb7394822b0db25470b746f03 
>   examples/src/main/java/kafka/examples/Producer.java 
> 96e98933148d07564c1b30ba8e805e2433c2adc8 
> 
> Diff: https://reviews.apache.org/r/31369/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>

Re: Review Request 29467: Patch for KAFKA-1660

[Jiangjie Qin]

> On March 2, 2015, 11:04 p.m., Jiangjie Qin wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java,
> >  line 219
> > 
> >
> > We probably need to release the caller threads that are waiting on 
> > flush() at this point.
> 
> Jay Kreps wrote:
> Making flush a boolean method that you have to always check to see if 
> someone called close() in another thead would be a really really really 
> painful api to use in practice, right?
> 
> I think the issue here is actually what I pointed out in the other 
> comment, namely that in-flight requests area actually left incomplete when 
> you call close and hit the forceClose timeout. Any other thread blocking on 
> these futures would block forever.
> 
> The right solution is just to fail all requests that haven't completed 
> when forceClose kicks in. This then fullfills the criteria for flush which is 
> that all the requests are completed or failed.

Yes, I agree that letting flush() return a boolean to just indicate whether 
someone called close is ugly. I'm thinking maybe we can make the return value 
to be more useful.
The idea of letting flush return a boolean comes when I was writing the mirror 
maker. When we call flush() followed by a consumer.commitOffsets(), we need to 
know the result of flush() in order to decide whether to commit offset or not. 
There might be three cases:
1. flush() succeeded on all batches.
2. flush() failed and some exception were thrown to caller thread (very rare, 
InterruptedException maybe)
3. flush() failed but are handled by sender thread in send callbacks.

For 1), no problem, everybody is happy.
For 2), caller thread knows something wrong happened and will not do next task 
(i.e. commit offsets).
For 3), caller thread has no idea about what happened and assumes everthing 
went well.

What I'm doing now is in send callback let the sender thread set a flag for the 
caller thread to check whether the flush succeeded or not when flush() returns. 
Otherwise, caller thread cannot decide whether to commit offset or not.

I'm thinking if in most cases people care about whether flush succeeded or not, 
they need to have this inter thread communication. If it is a common 
requirement, maybe we can let flush() return a boolean. 
>From API point of view, it is probably OK. If user cares about whether flush 
>succeeded or not, they check the return value, otherwise they ignore it. Just 
>like the what we do for send().


- Jiangjie


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74843
---


On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated March 2, 2015, 6:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> 
> Changing log levels as suggested.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 7397e565fd865214529ffccadd4222d835ac8110 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 6913090af03a455452b0b5c3df78f266126b3854 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>