Build failed in Jenkins: kafka-trunk-jdk8 #1288

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-4776; Implement graceful handling for improperly formed compressed

--
[...truncated 615.97 KB...]
org.apache.kafka.common.metrics.MetricsTest > testRemoveInactiveMetrics PASSED

org.apache.kafka.common.metrics.MetricsTest > testMetricName STARTED

org.apache.kafka.common.metrics.MetricsTest > testMetricName PASSED

org.apache.kafka.common.metrics.MetricsTest > testRateWindowing STARTED

org.apache.kafka.common.metrics.MetricsTest > testRateWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testTimeWindowing STARTED

org.apache.kafka.common.metrics.MetricsTest > testTimeWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testEventWindowing STARTED

org.apache.kafka.common.metrics.MetricsTest > testEventWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testRemoveMetric STARTED

org.apache.kafka.common.metrics.MetricsTest > testRemoveMetric PASSED

org.apache.kafka.common.metrics.MetricsTest > testBadSensorHierarchy STARTED

org.apache.kafka.common.metrics.MetricsTest > testBadSensorHierarchy PASSED

org.apache.kafka.common.metrics.MetricsTest > testRemoveSensor STARTED

org.apache.kafka.common.metrics.MetricsTest > testRemoveSensor PASSED

org.apache.kafka.common.metrics.MetricsTest > testPercentiles STARTED

org.apache.kafka.common.metrics.MetricsTest > testPercentiles PASSED

org.apache.kafka.common.metrics.MetricsTest > testSampledStatInitialValue 
STARTED

org.apache.kafka.common.metrics.MetricsTest > testSampledStatInitialValue PASSED

org.apache.kafka.common.metrics.MetricsTest > testDuplicateMetricName STARTED

org.apache.kafka.common.metrics.MetricsTest > testDuplicateMetricName PASSED

org.apache.kafka.common.metrics.MetricsTest > testQuotas STARTED

org.apache.kafka.common.metrics.MetricsTest > testQuotas PASSED

org.apache.kafka.common.metrics.MetricsTest > testHierarchicalSensors STARTED

org.apache.kafka.common.metrics.MetricsTest > testHierarchicalSensors PASSED

org.apache.kafka.common.utils.CrcTest > testUpdateInt STARTED

org.apache.kafka.common.utils.CrcTest > testUpdateInt PASSED

org.apache.kafka.common.utils.CrcTest > testUpdate STARTED

org.apache.kafka.common.utils.CrcTest > testUpdate PASSED

org.apache.kafka.common.utils.AbstractIteratorTest > testEmptyIterator STARTED

org.apache.kafka.common.utils.AbstractIteratorTest > testEmptyIterator PASSED

org.apache.kafka.common.utils.AbstractIteratorTest > testIterator STARTED

org.apache.kafka.common.utils.AbstractIteratorTest > testIterator PASSED

org.apache.kafka.common.utils.UtilsTest > testAbs STARTED

org.apache.kafka.common.utils.UtilsTest > testAbs PASSED

org.apache.kafka.common.utils.UtilsTest > testMin STARTED

org.apache.kafka.common.utils.UtilsTest > testMin PASSED

org.apache.kafka.common.utils.UtilsTest > testJoin STARTED

org.apache.kafka.common.utils.UtilsTest > testJoin PASSED

org.apache.kafka.common.utils.UtilsTest > 
testReadFullyOrFailWithPartialFileChannelReads STARTED

org.apache.kafka.common.utils.UtilsTest > 
testReadFullyOrFailWithPartialFileChannelReads PASSED

org.apache.kafka.common.utils.UtilsTest > testReadBytes STARTED

org.apache.kafka.common.utils.UtilsTest > testReadBytes PASSED

org.apache.kafka.common.utils.UtilsTest > testGetHost STARTED

org.apache.kafka.common.utils.UtilsTest > testGetHost PASSED

org.apache.kafka.common.utils.UtilsTest > testGetPort STARTED

org.apache.kafka.common.utils.UtilsTest > testGetPort PASSED

org.apache.kafka.common.utils.UtilsTest > 
testReadFullyWithPartialFileChannelReads STARTED

org.apache.kafka.common.utils.UtilsTest > 
testReadFullyWithPartialFileChannelReads PASSED

org.apache.kafka.common.utils.UtilsTest > testReadFullyOrFailWithRealFile 
STARTED

org.apache.kafka.common.utils.UtilsTest > testReadFullyOrFailWithRealFile PASSED

org.apache.kafka.common.utils.UtilsTest > testReadFullyIfEofIsReached STARTED

org.apache.kafka.common.utils.UtilsTest > testReadFullyIfEofIsReached PASSED

org.apache.kafka.common.utils.UtilsTest > testCloseAll STARTED

org.apache.kafka.common.utils.UtilsTest > testCloseAll PASSED

org.apache.kafka.common.utils.UtilsTest > testFormatAddress STARTED

org.apache.kafka.common.utils.UtilsTest > testFormatAddress PASSED

org.apache.kafka.common.config.ConfigDefTest > testBasicTypes STARTED

org.apache.kafka.common.config.ConfigDefTest > testBasicTypes PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefault STARTED

org.apache.kafka.common.config.ConfigDefTest > testNullDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testParseForValidate STARTED

org.apache.kafka.common.config.ConfigDefTest > testParseForValidate PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultRange STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultRange PASSED

org.apache.kafka.common.config.ConfigDefTest > 

[jira] [Commented] (KAFKA-4776) Implement graceful handling for improperly formed compressed message sets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874046#comment-15874046
 ] 

ASF GitHub Bot commented on KAFKA-4776:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2572


> Implement graceful handling for improperly formed compressed message sets
> -
>
> Key: KAFKA-4776
> URL: https://issues.apache.org/jira/browse/KAFKA-4776
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1, 0.10.2.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Minor
> Fix For: 0.10.3.0
>
>
> This affects validation of compressed message sets. It is possible for a 
> buggy client to send both a null compressed message set (i.e. a wrapper 
> message with a null value), and an empty compressed message set (i.e. a 
> wrapper message with valid compressed data in the value field, but no actual 
> records). In both cases, this causes an unexpected exception raised from the 
> deep iteration, which is returned to the client as an UNKNOWN_ERROR. It would 
> be better to return a CORRUPT_MESSAGE error.
> Note also that the behavior of the empty case was potentially more 
> problematic in versions prior to 0.10.2.0. Although we properly handled the 
> null case, the broker would accept the empty message set and write it to the 
> log. The impact of this appears to be minor, but may cause unexpected 
> behavior in cases where we assume compressed message sets would contain some 
> records.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2572: KAFKA-4776: Implement graceful handling for improp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2572


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4776) Implement graceful handling for improperly formed compressed message sets

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-4776.

   Resolution: Fixed
Fix Version/s: 0.10.3.0

Issue resolved by pull request 2572
[https://github.com/apache/kafka/pull/2572]

> Implement graceful handling for improperly formed compressed message sets
> -
>
> Key: KAFKA-4776
> URL: https://issues.apache.org/jira/browse/KAFKA-4776
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1, 0.10.2.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Minor
> Fix For: 0.10.3.0
>
>
> This affects validation of compressed message sets. It is possible for a 
> buggy client to send both a null compressed message set (i.e. a wrapper 
> message with a null value), and an empty compressed message set (i.e. a 
> wrapper message with valid compressed data in the value field, but no actual 
> records). In both cases, this causes an unexpected exception raised from the 
> deep iteration, which is returned to the client as an UNKNOWN_ERROR. It would 
> be better to return a CORRUPT_MESSAGE error.
> Note also that the behavior of the empty case was potentially more 
> problematic in versions prior to 0.10.2.0. Although we properly handled the 
> null case, the broker would accept the empty message set and write it to the 
> log. The impact of this appears to be minor, but may cause unexpected 
> behavior in cases where we assume compressed message sets would contain some 
> records.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-2319) After controlled shutdown: IllegalStateException: Kafka scheduler has not been started

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872967#comment-15872967
 ] 

Michał Woś edited comment on KAFKA-2319 at 2/20/17 4:26 AM:


I've got the same problem after sending SIGTERM to a broker. Moreover: each 
time I restart gracefully brokers I'm loosing data. I'm pretty sure not in the 
producer side, I see from producer metrics that there are retries during 
initializing of broker stop, but no records dropped recorded. Producer thinks 
that all records were stored. However they are missing on the server side.
Moreover between "kafka.log.LogManager: Shutting down." and "INFO 
kafka.log.LogManager: Shutdown complete." there is 4 minute pause (nothing is 
printed in logs). So yes - stopping broker takes >4 minutes.
I'm running kafka from cloudera which is based on 0.8.2.0 + (I belive) 127 
commits. To be exact, Cloudera version is named: 0.8.2.0+kafka1.4.0+127. 
Each broker leads ~200 topics.

Full log:
{code}
...
2017-02-17 17:55:54,059 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutting down
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Stopped 
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutdown completed
2017-02-17 17:55:54,062 INFO kafka.server.ReplicaFetcherManager: 
[ReplicaFetcherManager on broker 721] shutdown completed
2017-02-17 17:55:54,090 INFO kafka.server.ReplicaManager: [Replica Manager on 
Broker 721]: Shut down completely
2017-02-17 17:55:54,091 INFO kafka.log.LogManager: Shutting down.
// note it is still not killed, 4 minutes break and finally:
2017-02-17 17:59:56,174 INFO kafka.log.LogManager: Shutdown complete.
2017-02-17 17:59:56,179 WARN kafka.utils.Utils$: Kafka scheduler has not been 
started
java.lang.IllegalStateException: Kafka scheduler has not been started
at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
at 
kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:354)
at kafka.controller.KafkaController.shutdown(KafkaController.scala:677)
at 
kafka.server.KafkaServer$$anonfun$shutdownapply$mcV$sp(KafkaServer.scala:285)
at kafka.utils.Utils$.swallow(Utils.scala:172)
at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
at kafka.utils.Logging$class.swallow(Logging.scala:94)
at kafka.utils.Utils$.swallow(Utils.scala:45)
at kafka.server.KafkaServer.shutdown(KafkaServer.scala:285)
at 
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:42)
at kafka.Kafka$$anonrun(Kafka.scala:42)
2017-02-17 17:59:56,179 INFO org.I0Itec.zkclient.ZkEventThread: Terminate 
ZkClient event thread.
2017-02-17 17:59:56,190 INFO org.apache.zookeeper.ZooKeeper: Session: 
0x45a1d0f178a1267 closed
2017-02-17 17:59:56,190 INFO org.apache.zookeeper.ClientCnxn: EventThread shut 
down
2017-02-17 17:59:56,199 INFO kafka.server.KafkaServer: [Kafka Server 721], shut 
down completed
{code}

Can someone confirm that kafka 0.9.0.0 resolves that issue for sure? 
[~hachikuji] Did you upgrade? Did it help?



was (Author: wosiu):
I've got the same problem after seinding SIGTERM to a broker. Moreover: each 
time I restart gracefully brokers I'm loosing data. I'm pretty sure not in the 
producer side, I see from producer metrics that there are retries during 
initializing of broker stop, but no records dropped recorded. Producer thinks 
that all records were stored. However they are missing on the server side.
Moreover between "kafka.log.LogManager: Shutting down." and "INFO 
kafka.log.LogManager: Shutdown complete." there is 4 minute pause (nothing is 
printed in logs). So yes - stopping broker takes >4 minutes.
I'm running kafka from cloudera which is based on 0.8.2.0 + (I belive) 127 
commits. To be exact, Cloudera version is named: 0.8.2.0+kafka1.4.0+127. 
Each broker leads ~200 topics.

Full log:
{code}
...
2017-02-17 17:55:54,059 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutting down
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Stopped 
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutdown completed
2017-02-17 17:55:54,062 INFO kafka.server.ReplicaFetcherManager: 
[ReplicaFetcherManager on broker 721] shutdown completed
2017-02-17 17:55:54,090 INFO kafka.server.ReplicaManager: [Replica Manager on 
Broker 721]: Shut down completely
2017-02-17 17:55:54,091 INFO kafka.log.LogManager: Shutting down.
// note it is still not killed, 4 minutes break and finally:
2017-02-17 17:59:56,174 INFO kafka.log.LogManager: Shutdown complete.

[jira] [Comment Edited] (KAFKA-2319) After controlled shutdown: IllegalStateException: Kafka scheduler has not been started

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872967#comment-15872967
 ] 

Michał Woś edited comment on KAFKA-2319 at 2/20/17 4:26 AM:


I've got the same problem after seinding SIGTERM to a broker. Moreover: each 
time I restart gracefully brokers I'm loosing data. I'm pretty sure not in the 
producer side, I see from producer metrics that there are retries during 
initializing of broker stop, but no records dropped recorded. Producer thinks 
that all records were stored. However they are missing on the server side.
Moreover between "kafka.log.LogManager: Shutting down." and "INFO 
kafka.log.LogManager: Shutdown complete." there is 4 minute pause (nothing is 
printed in logs). So yes - stopping broker takes >4 minutes.
I'm running kafka from cloudera which is based on 0.8.2.0 + (I belive) 127 
commits. To be exact, Cloudera version is named: 0.8.2.0+kafka1.4.0+127. 
Each broker leads ~200 topics.

Full log:
{code}
...
2017-02-17 17:55:54,059 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutting down
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Stopped 
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutdown completed
2017-02-17 17:55:54,062 INFO kafka.server.ReplicaFetcherManager: 
[ReplicaFetcherManager on broker 721] shutdown completed
2017-02-17 17:55:54,090 INFO kafka.server.ReplicaManager: [Replica Manager on 
Broker 721]: Shut down completely
2017-02-17 17:55:54,091 INFO kafka.log.LogManager: Shutting down.
// note it is still not killed, 4 minutes break and finally:
2017-02-17 17:59:56,174 INFO kafka.log.LogManager: Shutdown complete.
2017-02-17 17:59:56,179 WARN kafka.utils.Utils$: Kafka scheduler has not been 
started
java.lang.IllegalStateException: Kafka scheduler has not been started
at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
at 
kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:354)
at kafka.controller.KafkaController.shutdown(KafkaController.scala:677)
at 
kafka.server.KafkaServer$$anonfun$shutdownapply$mcV$sp(KafkaServer.scala:285)
at kafka.utils.Utils$.swallow(Utils.scala:172)
at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
at kafka.utils.Logging$class.swallow(Logging.scala:94)
at kafka.utils.Utils$.swallow(Utils.scala:45)
at kafka.server.KafkaServer.shutdown(KafkaServer.scala:285)
at 
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:42)
at kafka.Kafka$$anonrun(Kafka.scala:42)
2017-02-17 17:59:56,179 INFO org.I0Itec.zkclient.ZkEventThread: Terminate 
ZkClient event thread.
2017-02-17 17:59:56,190 INFO org.apache.zookeeper.ZooKeeper: Session: 
0x45a1d0f178a1267 closed
2017-02-17 17:59:56,190 INFO org.apache.zookeeper.ClientCnxn: EventThread shut 
down
2017-02-17 17:59:56,199 INFO kafka.server.KafkaServer: [Kafka Server 721], shut 
down completed
{code}

Can someone confirm that kafka 0.9.0.0 resolves that issue for sure? 
[~hachikuji] Did you upgrade? Did it help?



was (Author: wosiu):
I've got the same problem after SIGTERM. Moreover: each time I restart 
gracefully brokers I'm loosing data. I'm pretty sure not in the producer side, 
I see from producer metrics that there are retries during initializing of 
broker stop, but no records dropped recorded. Producer thinks that all records 
were stored. However they are missing on the server side.
Moreover between "kafka.log.LogManager: Shutting down." and "INFO 
kafka.log.LogManager: Shutdown complete." there is 4 minute pause (nothing is 
printed in logs). So yes - stopping broker takes >4 minutes.
I'm running kafka from cloudera which is based on 0.8.2.0 + (I belive) 127 
commits. To be exact, Cloudera version is named: 0.8.2.0+kafka1.4.0+127. 
Each broker leads ~200 topics.

Full log:
{code}
...
2017-02-17 17:55:54,059 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutting down
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Stopped 
2017-02-17 17:55:54,061 INFO kafka.server.ReplicaFetcherThread: 
[ReplicaFetcherThread-0-2032], Shutdown completed
2017-02-17 17:55:54,062 INFO kafka.server.ReplicaFetcherManager: 
[ReplicaFetcherManager on broker 721] shutdown completed
2017-02-17 17:55:54,090 INFO kafka.server.ReplicaManager: [Replica Manager on 
Broker 721]: Shut down completely
2017-02-17 17:55:54,091 INFO kafka.log.LogManager: Shutting down.
// note it is still not killed, 4 minutes break and finally:
2017-02-17 17:59:56,174 INFO kafka.log.LogManager: Shutdown complete.
2017-02-17 

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

[Guozhang Wang]

Thanks Becket.

Actually sequence is associated with a message, not a message set. For
example if a message set sent by producer contains 100 messages, and the
first message's sequence is 5, then the last message's sequence number
would be 104, and the next message set's first sequence is expected to be
105.


Guozhang


On Sun, Feb 19, 2017 at 4:48 PM, Becket Qin  wrote:

> +1. Thanks for the great work on the KIP!
>
> I have only one minor question, in the wiki (and the doc) the new message
> set format has a "FirstSequence" field, should it just be "Sequence" if the
> sequence is always associated with a message set?
>
> On Fri, Feb 17, 2017 at 3:28 AM, Michael Pearce 
> wrote:
>
> > +0
> >
> > I think need some unified agreement on the VarInts.
> >
> > Would this also change in all other area’s of the protocol, e.g. value
> and
> > key length in message protocol, to keep this uniform across all protocols
> > going forwards?
> >
> >
> >
> > On 17/02/2017, 00:23, "Apurva Mehta"  wrote:
> >
> > Hi Jun,
> >
> > Thanks for the reply. Comments inline.
> >
> > On Thu, Feb 16, 2017 at 2:29 PM, Jun Rao  wrote:
> >
> > > Hi, Apurva,
> > >
> > > Thanks for the reply. A couple of comment below.
> > >
> > > On Wed, Feb 15, 2017 at 9:45 PM, Apurva Mehta  >
> > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Answers inline:
> > > >
> > > > 210. Pid snapshots: Is the number of pid snapshot configurable or
> > > hardcoded
> > > > > with 2? When do we decide to roll a new snapshot? Based on
> time,
> > byte,
> > > or
> > > > > offset? Is that configurable too?
> > > > >
> > >
> >
> >
> > > When a replica becomes a follower, we do a bit log truncation.
> > Having an
> > > older snapshot allows us to recover the PID->sequence mapping much
> > quicker
> > > than rescanning the whole log.
> >
> >
> > This is a good point. I have updated the doc with a more detailed
> > proposal.
> > Essentially, snapshots will be created on a periodic basis. A
> > reasonable
> > period would be every 30 or 60 seconds. We will keep at most 2 copies
> > of
> > the snapshot file. With this setup, we would have to replay at most
> 60
> > or
> > 120 seconds of the log in the event of log truncation during leader
> > failover.
> >
> > If we need to make any of this configurable, we can expose a config
> in
> > the
> > future. It would be easier to add a config we need than remove one
> with
> > marginal utility.
> >
> >
> > >
> > > > >
> > > > > 211. I am wondering if we should store ExpirationTime in the
> > producer
> > > > > transactionalId mapping message as we do in the producer
> > transaction
> > > > status
> > > > > message. If a producer only calls initTransactions(), but never
> > > publishes
> > > > > any data, we still want to be able to expire and remove the
> > producer
> > > > > transactionalId mapping message.
> > > > >
> > > > >
> > > > Actually, the document was inaccurate. The transactionalId will
> be
> > > expired
> > > > only if there is no active transaction, and the age of the last
> > > transaction
> > > > with that transactionalId is older than the transactioanlId
> > expiration
> > > > time. With these semantics, storing the expiration time in the
> > > > transactionalId mapping message won't be useful, since the
> > expiration
> > > time
> > > > is a moving target based on transaction activity.
> > > >
> > > > I have updated the doc with a clarification.
> > > >
> > > >
> > > >
> > > Currently, the producer transactionalId mapping message doesn't
> carry
> > > ExpirationTime, but the producer transaction status message does.
> > It would
> > > be useful if they are consistent.
> > >
> > >
> > You are right. The document has been updated to remove the
> > ExpirationTime
> > from the transaction status messages as well. Any utility for this
> > field
> > can be achieved by using the timestamp of the message itself along
> with
> > another expiration time (like transactionalId expiration time,
> > transaction
> > expiration time, etc.).
> >
> > Thanks,
> > Apurva
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a 

[GitHub] kafka-site issue #45: Manual edits needed for 0.10.2 release

[guozhangwang]

Github user guozhangwang commented on the issue:

https://github.com/apache/kafka-site/pull/45
  
@ewencp Can I merge this patch now since 0.10.2 is released already? I will 
rebase the patch of course.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4781) Kafka should return its advertised host name before any protocol verification is done

[Stephane Maarek (JIRA)]

Stephane Maarek created KAFKA-4781:
--

 Summary: Kafka should return its advertised host name before any 
protocol verification is done
 Key: KAFKA-4781
 URL: https://issues.apache.org/jira/browse/KAFKA-4781
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.10.1.1
Reporter: Stephane Maarek


We have a Kafka cluster and each broker advertises its hostname 
e.g.
kafka1.example.com
kafka2.example.com
kafka3.example.com

We have an SSL certificate for *.example.com and we have SASL principals for 
kafka/kafka[1,2,3].example.com 

All works well using SASL_SSL if we set the bootstrap servers as 
kafka1.example.com:9095,kafka2.example.com:9095,kafka3.example.com:9095

As soon as we set the bootstrap server as localhost:9095, it doesn't work. 
Kerberos can't authenticate.

Also, we like to have one CNAME that points to all the brokers in a round robin 
fashion, say kafka.example.com. In that case, if we use kafka.example.com:9095 
as our bootstrap, we get a Server not found in Kerberos database error as it 
tries to look up kafka.example.com

I think Kafka communicates its advertised hostname after the handshake (SASL / 
SSL) is done, which is a problem in our case. 

Would it be beneficial that on connection opening (on any port), Kafka first 
sends its advertised hostname. Then the SASL / SSL protocols use that 
advertised hostname as a starting point to do the authentication, etc?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-4767) KafkaProducer is not joining its IO thread properly

[huxi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huxi reassigned KAFKA-4767:
---

Assignee: huxi

> KafkaProducer is not joining its IO thread properly
> ---
>
> Key: KAFKA-4767
> URL: https://issues.apache.org/jira/browse/KAFKA-4767
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.11.0.0
>Reporter: Buğra Gedik
>Assignee: huxi
>Priority: Minor
>
> The {{KafkaProducer}} is not properly joining the thread it creates. The code 
> is like this:
> {code}
> try {
> this.ioThread.join(timeUnit.toMillis(timeout));
> } catch (InterruptedException t) {
> firstException.compareAndSet(null, t);
> log.error("Interrupted while joining ioThread", t);
> }
> {code}
> If the code is interrupted while performing the join, it will end up leaving 
> the io thread running. The correct way of handling this is a follows:
> {code}
> try {
> this.ioThread.join(timeUnit.toMillis(timeout));
> } catch (InterruptedException t) {
> // propagate the interrupt
> this.ioThread.interrupt();
> try { 
>  this.ioThread.join();
> } catch (InterruptedException t) {
> firstException.compareAndSet(null, t);
> log.error("Interrupted while joining ioThread", t);
> } finally {
> // make sure we maintain the interrupted status
> Thread.currentThread.interrupt();
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2576: kafka-4767: KafkaProducer is not joining its IO th...

[amethystic]

GitHub user amethystic opened a pull request:

https://github.com/apache/kafka/pull/2576

kafka-4767: KafkaProducer is not joining its IO thread properly

KafkaProducer#close swallows the InterruptedException which might be 
acceptable when it's invoked from within the main thread or user is extending 
Thread and therefore control all the code higher up on the call stack. For 
other cases, it'd better retstore the interupted status after capturing the 
exception.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amethystic/kafka 
kafka-4767_KafkaProducer_not_joining_IO_thread

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2576.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2576


commit d10417802d58d4282fe4a39377e38df1a6d4a7db
Author: huxi 
Date:   2017-02-20T02:27:27Z

kafka-4767: KafkaProducer is not joining its IO thread properly

KafkaProducer#close swallows the InterruptedException which might be 
acceptable when it's invoked from within the main thread or user is extending 
Thread and therefore control all the code higher up on the call stack. For 
other cases, it'd better retstore the interupted status after capturing the 
exception.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4780) ReplicaFetcherThread.fetch could not get any reponse

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873942#comment-15873942
 ] 

huxi commented on KAFKA-4780:
-

Seems it is a duplicate of 
[KAFKA-4477|https://issues.apache.org/jira/browse/KAFKA-4477?jql=project%20%3D%20KAFKA%20AND%20text%20~%20%22was%20disconnected%20before%20the%20response%20was%20read%22]?

> ReplicaFetcherThread.fetch could not get any reponse
> 
>
> Key: KAFKA-4780
> URL: https://issues.apache.org/jira/browse/KAFKA-4780
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: mashudong
> Attachments: capture.png, log.png, partition.png
>
>
> All partitions with broker 3 as leader has just broker 3 in its isr
>  !partition.png!
> Many IOException in server.log on broker 1 and 2
> !log.png!
> According to network packet capture, ReplicaFetcherThread of broker 1 could 
> not get any response from broker 3, and after 30 seconds the connection was 
> closed by broker 1.
> !capture.png!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2575: MINOR: update AWS test setup guide

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2575

MINOR: update AWS test setup guide



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka minor-update-system-test-readme

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2575.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2575


commit 3e0f6a9d1ef602537c1d31d98420804e111a4432
Author: Matthias J. Sax 
Date:   2017-02-20T01:33:23Z

MINOR: update AWS test setup guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

[Becket Qin]

+1. Thanks for the great work on the KIP!

I have only one minor question, in the wiki (and the doc) the new message
set format has a "FirstSequence" field, should it just be "Sequence" if the
sequence is always associated with a message set?

On Fri, Feb 17, 2017 at 3:28 AM, Michael Pearce 
wrote:

> +0
>
> I think need some unified agreement on the VarInts.
>
> Would this also change in all other area’s of the protocol, e.g. value and
> key length in message protocol, to keep this uniform across all protocols
> going forwards?
>
>
>
> On 17/02/2017, 00:23, "Apurva Mehta"  wrote:
>
> Hi Jun,
>
> Thanks for the reply. Comments inline.
>
> On Thu, Feb 16, 2017 at 2:29 PM, Jun Rao  wrote:
>
> > Hi, Apurva,
> >
> > Thanks for the reply. A couple of comment below.
> >
> > On Wed, Feb 15, 2017 at 9:45 PM, Apurva Mehta 
> wrote:
> >
> > > Hi Jun,
> > >
> > > Answers inline:
> > >
> > > 210. Pid snapshots: Is the number of pid snapshot configurable or
> > hardcoded
> > > > with 2? When do we decide to roll a new snapshot? Based on time,
> byte,
> > or
> > > > offset? Is that configurable too?
> > > >
> >
>
>
> > When a replica becomes a follower, we do a bit log truncation.
> Having an
> > older snapshot allows us to recover the PID->sequence mapping much
> quicker
> > than rescanning the whole log.
>
>
> This is a good point. I have updated the doc with a more detailed
> proposal.
> Essentially, snapshots will be created on a periodic basis. A
> reasonable
> period would be every 30 or 60 seconds. We will keep at most 2 copies
> of
> the snapshot file. With this setup, we would have to replay at most 60
> or
> 120 seconds of the log in the event of log truncation during leader
> failover.
>
> If we need to make any of this configurable, we can expose a config in
> the
> future. It would be easier to add a config we need than remove one with
> marginal utility.
>
>
> >
> > > >
> > > > 211. I am wondering if we should store ExpirationTime in the
> producer
> > > > transactionalId mapping message as we do in the producer
> transaction
> > > status
> > > > message. If a producer only calls initTransactions(), but never
> > publishes
> > > > any data, we still want to be able to expire and remove the
> producer
> > > > transactionalId mapping message.
> > > >
> > > >
> > > Actually, the document was inaccurate. The transactionalId will be
> > expired
> > > only if there is no active transaction, and the age of the last
> > transaction
> > > with that transactionalId is older than the transactioanlId
> expiration
> > > time. With these semantics, storing the expiration time in the
> > > transactionalId mapping message won't be useful, since the
> expiration
> > time
> > > is a moving target based on transaction activity.
> > >
> > > I have updated the doc with a clarification.
> > >
> > >
> > >
> > Currently, the producer transactionalId mapping message doesn't carry
> > ExpirationTime, but the producer transaction status message does.
> It would
> > be useful if they are consistent.
> >
> >
> You are right. The document has been updated to remove the
> ExpirationTime
> from the transaction status messages as well. Any utility for this
> field
> can be achieved by using the timestamp of the message itself along with
> another expiration time (like transactionalId expiration time,
> transaction
> expiration time, etc.).
>
> Thanks,
> Apurva
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

[GitHub] kafka pull request #2574: MINOR: Fixed 3 inner classes without instance refe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2574


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4774) Inner classes which don't need a reference to the outer class should be static

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4774.

   Resolution: Fixed
Fix Version/s: 0.10.3.0

Issue resolved by pull request 2558
[https://github.com/apache/kafka/pull/2558]

> Inner classes which don't need a reference to the outer class should be static
> --
>
> Key: KAFKA-4774
> URL: https://issues.apache.org/jira/browse/KAFKA-4774
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.10.3.0
>
>
> Inner classes which don't need a reference to the outer class should be 
> static.  This takes up less space in memory, generates less load on the 
> garbage collector, and eliminates a findbugs warning.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Jason Gustafson]

Re; having a dedicated Headers object. I think it makes sense. Maybe it
could implement Iterable?

One thing I'm not too sure about is mutability. Both the ProducerRecord and
ConsumerRecord are immutable currently. It would be nice to keep it that
way. Perhaps Headers could itself be immutable and passed through the
constructors?

Just going through how this is,  I don’t believe this would provide much
> saving. If issue trying to resolve is around object/garbage creation of
> Map.Entry objects.


It's definitely better because we only allocate the space we will use, but
I assume most of the actual memory cost is in the header data itself which
we can't do much about. Maybe we can start with the simple array
representation and optimize from there after we find it's worthwhile to do
so? If we use the Headers object, we can change this later without breaking
the API.

-Jason

On Sun, Feb 19, 2017 at 2:39 PM, radai  wrote:

> "(By the way, doesn't it feel a bit odd that we seem to be designing a
> feature which is optimized for people not using?)"
>
> very. this (i claim :-) ) is the result of very intense opposition to the
> usefulness of the feature early on, and not a real design goal
>
> On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce 
> wrote:
>
> > Whilst if having an array of a Header[] would mean getting this through,
> > im happy changing this, as already done.
> >
> > Just going through how this is,  I don’t believe this would provide much
> > saving. If issue trying to resolve is around object/garbage creation of
> > Map.Entry objects.
> >
> > All you’re doing here is replacing the equiv of a HashMap.Node
> (Map.Entry)
> > which simply holds reference to key and value objects with a custom
> variant.
> >
> >
> >
> >
> >
> > On 19/02/2017, 20:22, "Michael Pearce"  wrote:
> >
> > On point 1 & 2 Ive updated KIP to show varints (and removed the bit
> > flag). (on the assumption KIP 98 is getting the agreement the protocol is
> > moving from int32 to varInts as standard)
> >
> > On point 3 ive updated to use an array of Header class, instead of a
> > MultiMap in the Headers class object
> >
> >
> >
> > On 19/02/2017, 20:06, "Michael Pearce" 
> wrote:
> >
> > On points 1 and 2 I agree.
> >
> > This also affects kip-98, I should expect this resolved before
> > that vote also passes. If it is accepted there (I’m assuming this is
> > getting discussed on that KIP? As you’re driving the move to VarInts), I
> am
> > happy to make this KIP will simply follow suit to whatever is agreed in
> > KIP-98.
> >
> > On 3) Agreed this is a simpler form, as long as no one is
> > expecting hashmap lookup performance ( O(1) ), I still would prefer an
> > encapsulated class, so if we find that holding it as an array in future
> is
> > causing some perf issues, the internals are not exposed to end users,
> > allowing the internal structure to move to a map.
> >
> > Can we compromise on?
> >
> > class ConsumerRecord {
> > K key;
> > V value;
> > Headers headers;
> > }
> >
> > class Headers {
> >   Header[] headers;
> >
> >   add(String key, byte[] value)
> >   Collection get(String key)
> > }
> >
> > class Header {
> >   String key;
> >   byte[] value;
> > }
> >
> >
> >
> > On 19/02/2017, 18:54, "Jason Gustafson" 
> > wrote:
> >
> > >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> >
> >
> > This is exactly what I have been trying to get at. The use
> > cases documented
> > here are middleware:
> > https://cwiki.apache.org/confluence/display/KAFKA/A+
> > Case+for+Kafka+Headers.
> > If headers are intended for the application space as well,
> the
> > document
> > should be updated accordingly so that it is an explicit
> design
> > goal and not
> > an unfortunate design byproduct. There may be some
> > disagreement on whether
> > it _should_ be a design goal, but it may not matter much
> since
> > the current
> > interceptor API is probably insufficient for middleware
> > applications (I
> > haven't thought of a way to do this that isn't cumbersome).
> >
> > That aside, the three unresolved points for me are the
> > following:
> >
> > 1. The use of varints. As a one-off for use only with record
> > headers, the
> > case was weak, but if we are using them throughout the
> message
> > format, then
> > we should do so here as well. The additional complexity is
> > minimal and
> > early performance testing fully justifies 

Re: [DISCUSS] KIP-124: Request rate quotas

[Becket Qin]

Thanks for the KIP, Rajini,

If I understand correctly the proposal was essentially trying to quota the
CPU usage (that is probably why time slice is used instead of request rate)
while the existing quota we have is for network bandwidth.

Given we are trying to throttle both CPU and Network, that implies the
following patterns for the clients:
1. High CPU usage, high network usage.
2. High CPU usage, low network usage.
3. Low CPU usage, high network usage.
4. Low CPU usage, low network usage

Theoretically the existing quota addresses case 3 & 4. And this KIP seems
trying to address case 1 & 2. However, it might be helpful to understand
what we want to achieve with CPU and network quotas.

People mainly use quota for two different purposes:
a) protecting the broker from misbehaving clients, and
b) resource distribution for multi-tenancy.

I agree that generally speaking CPU time is a suitable metric to quota on
for CPU usage and would work for a). However, as Dong and Onur noticed, it
is not easy to quantify the impact for the end users at application level
with a throttled CPU time. If the purpose of the CPU quota is only for
protection, maybe we don't need a user facing CPU quota.

That said, a user facing CPU quota could be useful for virtualization,
which maybe related to multi-tenancy but is a little different. Imagine
there are 10 services sharing the same physical Kafka cluster. With CPU
time quota and network bandwidth quota, each service can provision a
logical Kafka cluster with some reserved CPU time and network bandwidth.
And in this case the quota will be on per logic cluster. Not sure if this
is what the KIP is intended in the future, though. It would be good if the
KIP can be more clear on what exact scenarios the CPU quota is trying to
address.

As of the request rate quota, while it seems easy to enforce and intuitive,
there are some caveats.
1. Users do not have direct control over the request rate, i.e. users do
not known when a request will be sent by the clients.
2. Each request may require different amount of CPU resources to handle.
That may depends on many things, e.g. whether acks = 1 or acks = -1,
whether a request is addressing 1000 partitions or 1 partition, whether a
fetch request requires message format down conversion or not, etc.
So the result of using request rate quota could be quite unpredictable.

Thanks,

Jiangjie (Becket) Qin

On Sat, Feb 18, 2017 at 9:35 PM, Dong Lin  wrote:

> I realized the main concern with this proposal is how user can interpret
> this CPU-percentage based quota. Since this quota is exposed to user, we
> need to explain to user how this quota is going to impact their application
> performance and convince them that the quota is now too low for their
> application. We can able to do this with byte-rate based quota. But I am
> not sure how we can do this with CPU-percentage based quota. For example,
> how is user going to understand whether 1% CPU is OK?
>
> On Fri, Feb 17, 2017 at 10:11 AM, Onur Karaman <
> onurkaraman.apa...@gmail.com
> > wrote:
>
> > Overall a big fan of the KIP.
> >
> > I'd have to agree with Dong. I'm not sure about the decision of using the
> > percentage over the window as opposed to request rate. It's pretty hard
> to
> > reason about. I just spoke to one of our SRE's and he agrees.
> >
> > Also I may have missed it, but I couldn't find information in the KIP on
> > where this window would be configured.
> >
> > On Fri, Feb 17, 2017 at 9:45 AM, Dong Lin  wrote:
> >
> > > To correct the typo above: It seems to me that determination of request
> > > rate is not any more difficult than determination of *byte* rate as
> both
> > > metrics are commonly used to measure performance and provide guarantee
> to
> > > user.
> > >
> > > On Fri, Feb 17, 2017 at 9:40 AM, Dong Lin  wrote:
> > >
> > > > Hey Rajini,
> > > >
> > > > Thanks for the KIP. I have some questions:
> > > >
> > > > - I am wondering why throttling based on request rate is listed as a
> > > > rejected alternative. Can you provide more specific reason why it is
> > > > difficult for administrators to decide request rates to allocate? It
> > > seems
> > > > to me that determination of request rate is not any more difficult
> than
> > > > determination of request rate as both metrics are commonly used to
> > > measure
> > > > performance and provide guarantee to user. On the other hand, the
> > > > percentage of processing time provides a vague guarantee to user. For
> > > > example, what performance can user expect if you provide 1%
> processing
> > > time
> > > > quota to this user? How is administrator going to decide this quota?
> > > Should
> > > > Kafka administrator continues to reduce this percentage quota as
> number
> > > of
> > > > users grow?
> > > >
> > > > - The KIP suggests that LeaderAndIsrRequest and MetadataRequest will
> > also
> > > > be throttled by this quota. What is the motivation for 

Re: [DISCUSS] KIP-82 - Add Record Headers

[radai]

"(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)"

very. this (i claim :-) ) is the result of very intense opposition to the
usefulness of the feature early on, and not a real design goal

On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce 
wrote:

> Whilst if having an array of a Header[] would mean getting this through,
> im happy changing this, as already done.
>
> Just going through how this is,  I don’t believe this would provide much
> saving. If issue trying to resolve is around object/garbage creation of
> Map.Entry objects.
>
> All you’re doing here is replacing the equiv of a HashMap.Node (Map.Entry)
> which simply holds reference to key and value objects with a custom variant.
>
>
>
>
>
> On 19/02/2017, 20:22, "Michael Pearce"  wrote:
>
> On point 1 & 2 Ive updated KIP to show varints (and removed the bit
> flag). (on the assumption KIP 98 is getting the agreement the protocol is
> moving from int32 to varInts as standard)
>
> On point 3 ive updated to use an array of Header class, instead of a
> MultiMap in the Headers class object
>
>
>
> On 19/02/2017, 20:06, "Michael Pearce"  wrote:
>
> On points 1 and 2 I agree.
>
> This also affects kip-98, I should expect this resolved before
> that vote also passes. If it is accepted there (I’m assuming this is
> getting discussed on that KIP? As you’re driving the move to VarInts), I am
> happy to make this KIP will simply follow suit to whatever is agreed in
> KIP-98.
>
> On 3) Agreed this is a simpler form, as long as no one is
> expecting hashmap lookup performance ( O(1) ), I still would prefer an
> encapsulated class, so if we find that holding it as an array in future is
> causing some perf issues, the internals are not exposed to end users,
> allowing the internal structure to move to a map.
>
> Can we compromise on?
>
> class ConsumerRecord {
> K key;
> V value;
> Headers headers;
> }
>
> class Headers {
>   Header[] headers;
>
>   add(String key, byte[] value)
>   Collection get(String key)
> }
>
> class Header {
>   String key;
>   byte[] value;
> }
>
>
>
> On 19/02/2017, 18:54, "Jason Gustafson" 
> wrote:
>
> >
> > headers dont "leak" into application code. they are useful
> to application
> > code as well.
>
>
> This is exactly what I have been trying to get at. The use
> cases documented
> here are middleware:
> https://cwiki.apache.org/confluence/display/KAFKA/A+
> Case+for+Kafka+Headers.
> If headers are intended for the application space as well, the
> document
> should be updated accordingly so that it is an explicit design
> goal and not
> an unfortunate design byproduct. There may be some
> disagreement on whether
> it _should_ be a design goal, but it may not matter much since
> the current
> interceptor API is probably insufficient for middleware
> applications (I
> haven't thought of a way to do this that isn't cumbersome).
>
> That aside, the three unresolved points for me are the
> following:
>
> 1. The use of varints. As a one-off for use only with record
> headers, the
> case was weak, but if we are using them throughout the message
> format, then
> we should do so here as well. The additional complexity is
> minimal and
> early performance testing fully justifies their use.
>
> 2. If varints are used, the case for using attributes to
> indicate null
> headers also becomes weak. We only add one additional byte in
> each message
> if there are no headers. Whatever the case, let us be
> consistent with how
> we treat null keys and values.
>
> 3. We have apparently agreed that the broker will validate
> headers (please
> add this to the KIP). That being the case, I would prefer to
> use Kafka's
> conventional format for arrays. The downside is that it takes
> more work to
> skip over the headers on the consumer, though it's unclear if
> the cost
> would matter in practice. Some concern was previously
> expressed about the
> allocation of maps. An alternative would be to use arrays, i.e.
>
> class ConsumerRecord {
> K key;
> V value;
> Header[] headers;
> }
>
> class Header {
>   String key;
>   byte[] value;
> }
>
> This would work nicely with the conventional array format and
> my 

[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection

[Dipen Patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873850#comment-15873850
 ] 

Dipen Patel commented on KAFKA-1935:


Hi,

I think that this is an interesting first issue for me to work on. 
Please correct me if I am wrong, but from my understanding the issue we are 
trying to solve here is ensuring that each coordinator node that we register 
with the consumer's NetworkClient has a unique Id such that there are no 
conflicts with the Id's of existing nodes already registered with the 
consumer's NetworkClient. By ensuring that each coordinator node's Id is 
unique, we will have successfully mimicked unique sockets for each coordinator 
node. 

If that is the issue, then I think that implementing a mechanism to 
check whether an Id is currently assigned to a node would be an appropriate 
solution. This mechanism could be implemented by writing a method in the 
NetworkClient that queries the ClusterConnectionStates (a class containing a 
Map of all nodes) to check whether the String Id is assigned. We would then 
have to modify the KafkaClient interface to account for the new method added to 
the NetworkClient.  These changes would allow the AbstractCoordinator class to 
use its ConsumerNetworkClient field to call the newly implemented method in the 
NetworkClient. In the case that the Id is not available we increment the Id and 
repeat the process of checking the Id's uniqueness. Does this proposal sound 
viable?   

Thanks


> Consumer should use a separate socket for Coordinator connection
> 
>
> Key: KAFKA-1935
> URL: https://issues.apache.org/jira/browse/KAFKA-1935
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>  Labels: newbie
>
> KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be 
> able to create separate sockets for the same server for coordinator / broker 
> roles.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

On point 1 & 2 Ive updated KIP to show varints (and removed the bit flag). (on 
the assumption KIP 98 is getting the agreement the protocol is moving from 
int32 to varInts as standard)

On point 3 ive updated to use an array of Header class, instead of a MultiMap 
in the Headers class object



On 19/02/2017, 20:06, "Michael Pearce"  wrote:

On points 1 and 2 I agree.

This also affects kip-98, I should expect this resolved before that vote 
also passes. If it is accepted there (I’m assuming this is getting discussed on 
that KIP? As you’re driving the move to VarInts), I am happy to make this KIP 
will simply follow suit to whatever is agreed in KIP-98.

On 3) Agreed this is a simpler form, as long as no one is expecting hashmap 
lookup performance ( O(1) ), I still would prefer an encapsulated class, so if 
we find that holding it as an array in future is causing some perf issues, the 
internals are not exposed to end users, allowing the internal structure to move 
to a map.

Can we compromise on?

class ConsumerRecord {
K key;
V value;
Headers headers;
}

class Headers {
  Header[] headers;

  add(String key, byte[] value)
  Collection get(String key)
}

class Header {
  String key;
  byte[] value;
}



On 19/02/2017, 18:54, "Jason Gustafson"  wrote:

>
> headers dont "leak" into application code. they are useful to 
application
> code as well.


This is exactly what I have been trying to get at. The use cases 
documented
here are middleware:

https://cwiki.apache.org/confluence/display/KAFKA/A+Case+for+Kafka+Headers.
If headers are intended for the application space as well, the document
should be updated accordingly so that it is an explicit design goal and 
not
an unfortunate design byproduct. There may be some disagreement on 
whether
it _should_ be a design goal, but it may not matter much since the 
current
interceptor API is probably insufficient for middleware applications (I
haven't thought of a way to do this that isn't cumbersome).

That aside, the three unresolved points for me are the following:

1. The use of varints. As a one-off for use only with record headers, 
the
case was weak, but if we are using them throughout the message format, 
then
we should do so here as well. The additional complexity is minimal and
early performance testing fully justifies their use.

2. If varints are used, the case for using attributes to indicate null
headers also becomes weak. We only add one additional byte in each 
message
if there are no headers. Whatever the case, let us be consistent with 
how
we treat null keys and values.

3. We have apparently agreed that the broker will validate headers 
(please
add this to the KIP). That being the case, I would prefer to use Kafka's
conventional format for arrays. The downside is that it takes more work 
to
skip over the headers on the consumer, though it's unclear if the cost
would matter in practice. Some concern was previously expressed about 
the
allocation of maps. An alternative would be to use arrays, i.e.

class ConsumerRecord {
K key;
V value;
Header[] headers;
}

class Header {
  String key;
  byte[] value;
}

This would work nicely with the conventional array format and my guess 
is
it would obviate the need do any lazy initialization. If we use the map 
as
is currently documented, then it is possible with either representation 
to
slice the headers and initialize them lazily. Either way, it might be a
good idea to use a separate object to represent the headers in case we 
need
to modify it in the future in some way.

(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)


If we can resolve these points, then at least you will get my vote.

Thanks,
Jason

On Sun, Feb 19, 2017 at 7:30 AM, radai  
wrote:

> headers dont "leak" into application code. they are useful to 
application
> code as well.
> IIUC samze currently has headers "in-V" and would just switch over to 
kafka
> headers if they exist.
> im sure plenty of other users of kafka would have a use for headers.
> im pretty sure use cases exist around shuffling data into/out-of kafka
> (kafka connect or equivalent) where metadata from one end could 
copied over
> to the other (S3, for example uses http headers for user-accessible
> metadata). 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

On points 1 and 2 I agree.

This also affects kip-98, I should expect this resolved before that vote also 
passes. If it is accepted there (I’m assuming this is getting discussed on that 
KIP? As you’re driving the move to VarInts), I am happy to make this KIP will 
simply follow suit to whatever is agreed in KIP-98.

On 3) Agreed this is a simpler form, as long as no one is expecting hashmap 
lookup performance ( O(1) ), I still would prefer an encapsulated class, so if 
we find that holding it as an array in future is causing some perf issues, the 
internals are not exposed to end users, allowing the internal structure to move 
to a map.

Can we compromise on?

class ConsumerRecord {
K key;
V value;
Headers headers;
}

class Headers {
  Header[] headers;

  add(String key, byte[] value)
  Collection get(String key)
}

class Header {
  String key;
  byte[] value;
}



On 19/02/2017, 18:54, "Jason Gustafson"  wrote:

>
> headers dont "leak" into application code. they are useful to application
> code as well.


This is exactly what I have been trying to get at. The use cases documented
here are middleware:
https://cwiki.apache.org/confluence/display/KAFKA/A+Case+for+Kafka+Headers.
If headers are intended for the application space as well, the document
should be updated accordingly so that it is an explicit design goal and not
an unfortunate design byproduct. There may be some disagreement on whether
it _should_ be a design goal, but it may not matter much since the current
interceptor API is probably insufficient for middleware applications (I
haven't thought of a way to do this that isn't cumbersome).

That aside, the three unresolved points for me are the following:

1. The use of varints. As a one-off for use only with record headers, the
case was weak, but if we are using them throughout the message format, then
we should do so here as well. The additional complexity is minimal and
early performance testing fully justifies their use.

2. If varints are used, the case for using attributes to indicate null
headers also becomes weak. We only add one additional byte in each message
if there are no headers. Whatever the case, let us be consistent with how
we treat null keys and values.

3. We have apparently agreed that the broker will validate headers (please
add this to the KIP). That being the case, I would prefer to use Kafka's
conventional format for arrays. The downside is that it takes more work to
skip over the headers on the consumer, though it's unclear if the cost
would matter in practice. Some concern was previously expressed about the
allocation of maps. An alternative would be to use arrays, i.e.

class ConsumerRecord {
K key;
V value;
Header[] headers;
}

class Header {
  String key;
  byte[] value;
}

This would work nicely with the conventional array format and my guess is
it would obviate the need do any lazy initialization. If we use the map as
is currently documented, then it is possible with either representation to
slice the headers and initialize them lazily. Either way, it might be a
good idea to use a separate object to represent the headers in case we need
to modify it in the future in some way.

(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)


If we can resolve these points, then at least you will get my vote.

Thanks,
Jason

On Sun, Feb 19, 2017 at 7:30 AM, radai  wrote:

> headers dont "leak" into application code. they are useful to application
> code as well.
> IIUC samze currently has headers "in-V" and would just switch over to 
kafka
> headers if they exist.
> im sure plenty of other users of kafka would have a use for headers.
> im pretty sure use cases exist around shuffling data into/out-of kafka
> (kafka connect or equivalent) where metadata from one end could copied 
over
> to the other (S3, for example uses http headers for user-accessible
> metadata). it will be kafka client code getting/setting those headers. not
> an interceptor.
>
> On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce 
> wrote:
>
> > For APM single event tracing, need access to the header at the point of
> > processing on the processing thread.
> >
> > As such interceptors will not work/be suitable for these, due to the 
fact
> > they act on the ConsumerRecords as a batch, before the handling thread
> can
> > split out and process per message which is the point these tools will
> need
> > to continue to transaction tracing.
> >
> > Like wise tools and infra pieces will need 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Jason Gustafson]

>
> headers dont "leak" into application code. they are useful to application
> code as well.


This is exactly what I have been trying to get at. The use cases documented
here are middleware:
https://cwiki.apache.org/confluence/display/KAFKA/A+Case+for+Kafka+Headers.
If headers are intended for the application space as well, the document
should be updated accordingly so that it is an explicit design goal and not
an unfortunate design byproduct. There may be some disagreement on whether
it _should_ be a design goal, but it may not matter much since the current
interceptor API is probably insufficient for middleware applications (I
haven't thought of a way to do this that isn't cumbersome).

That aside, the three unresolved points for me are the following:

1. The use of varints. As a one-off for use only with record headers, the
case was weak, but if we are using them throughout the message format, then
we should do so here as well. The additional complexity is minimal and
early performance testing fully justifies their use.

2. If varints are used, the case for using attributes to indicate null
headers also becomes weak. We only add one additional byte in each message
if there are no headers. Whatever the case, let us be consistent with how
we treat null keys and values.

3. We have apparently agreed that the broker will validate headers (please
add this to the KIP). That being the case, I would prefer to use Kafka's
conventional format for arrays. The downside is that it takes more work to
skip over the headers on the consumer, though it's unclear if the cost
would matter in practice. Some concern was previously expressed about the
allocation of maps. An alternative would be to use arrays, i.e.

class ConsumerRecord {
K key;
V value;
Header[] headers;
}

class Header {
  String key;
  byte[] value;
}

This would work nicely with the conventional array format and my guess is
it would obviate the need do any lazy initialization. If we use the map as
is currently documented, then it is possible with either representation to
slice the headers and initialize them lazily. Either way, it might be a
good idea to use a separate object to represent the headers in case we need
to modify it in the future in some way.

(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)


If we can resolve these points, then at least you will get my vote.

Thanks,
Jason

On Sun, Feb 19, 2017 at 7:30 AM, radai  wrote:

> headers dont "leak" into application code. they are useful to application
> code as well.
> IIUC samze currently has headers "in-V" and would just switch over to kafka
> headers if they exist.
> im sure plenty of other users of kafka would have a use for headers.
> im pretty sure use cases exist around shuffling data into/out-of kafka
> (kafka connect or equivalent) where metadata from one end could copied over
> to the other (S3, for example uses http headers for user-accessible
> metadata). it will be kafka client code getting/setting those headers. not
> an interceptor.
>
> On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce 
> wrote:
>
> > For APM single event tracing, need access to the header at the point of
> > processing on the processing thread.
> >
> > As such interceptors will not work/be suitable for these, due to the fact
> > they act on the ConsumerRecords as a batch, before the handling thread
> can
> > split out and process per message which is the point these tools will
> need
> > to continue to transaction tracing.
> >
> > Like wise tools and infra pieces will need access to the message outside
> > the interceptor realm.
> >
> >
> >
> > On 17/02/2017, 21:26, "Jason Gustafson"  wrote:
> >
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> >
> >
> > So the broker will parse the headers and validate them. Good. The
> only
> > reason remaining that I can see to leave the headers as a byte array
> > is to
> > make it easier for the client to skip past them. Are we sure this is
> > not
> > premature optimization? Are there any performance results which show
> > that
> > this is worthwhile?
> >
> > What’s the issue with exposing a method getHeaders on the
> > producer/consumer
> > > record? It doesn’t break anything. We don’t need any special
> version.
> >
> >
> > See my previous 

[jira] [Comment Edited] (KAFKA-1524) Implement transactional producer

[Andrew Olson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873783#comment-15873783
 ] 

Andrew Olson edited comment on KAFKA-1524 at 2/19/17 6:16 PM:
--

Can this Jira be closed as obsolete? It appears to have been superseded by the 
design in 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging

Same comment applies to several other open Jiras with the "transactions" label.


was (Author: noslowerdna):
Can this Jira be closed as obsolete? It appears to have been superseded by the 
design in 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging

> Implement transactional producer
> 
>
> Key: KAFKA-1524
> URL: https://issues.apache.org/jira/browse/KAFKA-1524
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1524_2014-08-18_09:39:34.patch, 
> KAFKA-1524_2014-08-20_09:14:59.patch, KAFKA-1524.patch, KAFKA-1524.patch, 
> KAFKA-1524.patch
>
>
> Implement the basic transactional producer functionality as outlined in 
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
> The scope of this jira is basic functionality (i.e., to be able to begin and 
> commit or abort a transaction) without the failure scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-1524) Implement transactional producer

[Andrew Olson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873783#comment-15873783
 ] 

Andrew Olson commented on KAFKA-1524:
-

Can this Jira be closed as obsolete? It appears to have been superseded by the 
design in 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging

> Implement transactional producer
> 
>
> Key: KAFKA-1524
> URL: https://issues.apache.org/jira/browse/KAFKA-1524
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1524_2014-08-18_09:39:34.patch, 
> KAFKA-1524_2014-08-20_09:14:59.patch, KAFKA-1524.patch, KAFKA-1524.patch, 
> KAFKA-1524.patch
>
>
> Implement the basic transactional producer functionality as outlined in 
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
> The scope of this jira is basic functionality (i.e., to be able to begin and 
> commit or abort a transaction) without the failure scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-82 - Add Record Headers

[radai]

headers dont "leak" into application code. they are useful to application
code as well.
IIUC samze currently has headers "in-V" and would just switch over to kafka
headers if they exist.
im sure plenty of other users of kafka would have a use for headers.
im pretty sure use cases exist around shuffling data into/out-of kafka
(kafka connect or equivalent) where metadata from one end could copied over
to the other (S3, for example uses http headers for user-accessible
metadata). it will be kafka client code getting/setting those headers. not
an interceptor.

On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce 
wrote:

> For APM single event tracing, need access to the header at the point of
> processing on the processing thread.
>
> As such interceptors will not work/be suitable for these, due to the fact
> they act on the ConsumerRecords as a batch, before the handling thread can
> split out and process per message which is the point these tools will need
> to continue to transaction tracing.
>
> Like wise tools and infra pieces will need access to the message outside
> the interceptor realm.
>
>
>
> On 17/02/2017, 21:26, "Jason Gustafson"  wrote:
>
> >
> > That’s exactly what we’re doing the headers are a slice of bytes,
> which
> > then gets parsed later if needed, or can be parsed right away, the
> headers
> > is part of the protocol, so can still be validated if wanted.
> > If you had a header count then you would have to go through each
> header
> > key and value length value to work out how much to skip to get to
> say the
> > value or any future component in the message after the headers.
> Having it
> > as a byte[] with length value makes this a lot easier to skip.
>
>
> So the broker will parse the headers and validate them. Good. The only
> reason remaining that I can see to leave the headers as a byte array
> is to
> make it easier for the client to skip past them. Are we sure this is
> not
> premature optimization? Are there any performance results which show
> that
> this is worthwhile?
>
> What’s the issue with exposing a method getHeaders on the
> producer/consumer
> > record? It doesn’t break anything. We don’t need any special version.
>
>
> See my previous explanation. What I am trying to resist is the headers
> becoming a general application-level facility. The primary use case as
> far
> as I can tell is middleware, which is the use case that the
> interceptors
> are providing.
>
> Current batch consumer model and consumer interceptors don’t work where
> > headers need to be acted on at per message level at time of
> processing,
> > very case is APM (the core one), where the header value is used to
> continue
> > tracing.
>
>
> I still don't understand the point about batching. The consumer
> records are
> exposed as a batch in the consumer interceptor, but you can still
> iterate
> them individually. It is no different for the consumer API itself.
>
> -Jason
>
> On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> michael.pea...@ig.com>
> wrote:
>
> > Re:
> >
> > “The point about creation of maps seems orthogonal. We can still
> > represent
> > the headers as a slice of bytes until the time it is accessed.”
> >
> > That’s exactly what we’re doing the headers are a slice of bytes,
> which
> > then gets parsed later if needed, or can be parsed right away, the
> headers
> > is part of the protocol, so can still be validated if wanted.
> >
> > If you had a header count then you would have to go through each
> header
> > key and value length value to work out how much to skip to get to
> say the
> > value or any future component in the message after the headers.
> Having it
> > as a byte[] with length value makes this a lot easier to skip.
> >
> >
> > On 17/02/2017, 20:37, "Michael Pearce" 
> wrote:
> >
> > What’s the issue with exposing a method getHeaders on the
> > producer/consumer record? It doesn’t break anything. We don’t need
> any
> > special version.
> >
> > Current batch consumer model and consumer interceptors don’t work
> > where headers need to be acted on at per message level at time of
> > processing, very case is APM (the core one), where the header value
> is used
> > to continue tracing. JMS/HTTP etc all expose these, without issues.
> I would
> > NOT want to lock this down to only be usable accessible via
> interceptors,
> > as we’d fail on one of the main goals.
> >
> > Regards
> > Mike
> >
> >
> >
> >
> > On 17/02/2017, 20:21, "Jason Gustafson" 
> wrote:
> >
> > The point about creation of maps seems orthogonal. We can
> still
> > represent
> > the headers as a slice of 

[jira] [Updated] (KAFKA-4780) ReplicaFetcherThread.fetch could not get any reponse

[mashudong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mashudong updated KAFKA-4780:
-
Attachment: capture.png
log.png

> ReplicaFetcherThread.fetch could not get any reponse
> 
>
> Key: KAFKA-4780
> URL: https://issues.apache.org/jira/browse/KAFKA-4780
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: mashudong
> Attachments: capture.png, log.png, partition.png
>
>
> All partitions with broker 3 as leader has just broker 3 in its isr
>  !partition.png!
> Many IOException in server.log on broker 1 and 2
> !log.png!
> According to network packet capture, ReplicaFetcherThread of broker 1 could 
> not get any response from broker 3, and after 30 seconds the connection was 
> closed by broker 1.
> !capture.png!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4780) ReplicaFetcherThread.fetch could not get any reponse

[mashudong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mashudong updated KAFKA-4780:
-
Attachment: (was: capture.png)

> ReplicaFetcherThread.fetch could not get any reponse
> 
>
> Key: KAFKA-4780
> URL: https://issues.apache.org/jira/browse/KAFKA-4780
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: mashudong
> Attachments: partition.png
>
>
> All partitions with broker 3 as leader has just broker 3 in its isr
>  !partition.png!
> Many IOException in server.log on broker 1 and 2
> !log.png!
> According to network packet capture, ReplicaFetcherThread of broker 1 could 
> not get any response from broker 3, and after 30 seconds the connection was 
> closed by broker 1.
> !capture.png!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4780) ReplicaFetcherThread.fetch could not get any reponse

[mashudong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mashudong updated KAFKA-4780:
-
Attachment: (was: log.png)

> ReplicaFetcherThread.fetch could not get any reponse
> 
>
> Key: KAFKA-4780
> URL: https://issues.apache.org/jira/browse/KAFKA-4780
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: mashudong
> Attachments: partition.png
>
>
> All partitions with broker 3 as leader has just broker 3 in its isr
>  !partition.png!
> Many IOException in server.log on broker 1 and 2
> !log.png!
> According to network packet capture, ReplicaFetcherThread of broker 1 could 
> not get any response from broker 3, and after 30 seconds the connection was 
> closed by broker 1.
> !capture.png!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-4780) ReplicaFetcherThread.fetch could not get any reponse

[mashudong (JIRA)]

mashudong created KAFKA-4780:


 Summary: ReplicaFetcherThread.fetch could not get any reponse
 Key: KAFKA-4780
 URL: https://issues.apache.org/jira/browse/KAFKA-4780
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 0.10.1.0
Reporter: mashudong
 Attachments: capture.png, log.png, partition.png

All partitions with broker 3 as leader has just broker 3 in its isr
 !partition.png!

Many IOException in server.log on broker 1 and 2
!log.png!

According to network packet capture, ReplicaFetcherThread of broker 1 could not 
get any response from broker 3, and after 30 seconds the connection was closed 
by broker 1.
!capture.png!




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2574: MINOR: Fixed 3 inner classes without instance refe...

[original-brownbear]

GitHub user original-brownbear opened a pull request:

https://github.com/apache/kafka/pull/2574

MINOR: Fixed 3 inner classes without instance reference to be static

* Turned 3 inner classes that weren't static but could be into `static` 
ones.
* Turned one `public` inner class that wasn't used publicly into a 
`private`.

Trivial but imo worthwhile to explicitly keep visibility and "staticness" 
correct in syntax (if only to be nice to the GC) :)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/original-brownbear/kafka 
cleanup-inner-nonstatic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2574.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2574


commit 9b0bf675f46e8a628ecb4ed4a56ba2e4bf184bfc
Author: Armin Braun 
Date:   2017-02-19T09:19:37Z

MINOR: Fixed 3 inner classes without instance reference to be static




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---