[GitHub] kafka pull request #3712: MINOR: Fix transient failure in SocketServerTest.t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3712


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Replace Broker

[Put C]

Trying to add a new broker to the Kafka cluster and decommission the
existing broker. Using the reassignment-partitions script, the partitions
are migrated successfully to the new broker. But the consumer offsets are
not migrating from existing broker to the new broker (Consumer offsets are
stored in kafka and not in zookeeper). So the existing consumers are not
receiving any messages with only the new node running. But able to publish
messages to the new node.

Appreciate if someone can provide some insights on how to handle this.

Jenkins build is back to normal : kafka-0.11.0-jdk7 #284

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #3733: [WIP] MINOR: KIP-160 docs

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/3733

[WIP] MINOR: KIP-160 docs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka KMinor-kip160-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3733.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3733


commit c13b3161cee31da24e301fe5399eaf79a01a924e
Author: Guozhang Wang 
Date:   2017-08-24T23:54:06Z

doc changes for KIP-138

commit 265a82fe5fd1f7f26f7e5e012dd9dacb61aaea5d
Author: Guozhang Wang 
Date:   2017-08-25T00:16:41Z

kip 160 docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #1940

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-5342; Clarify producer fatal/abortable errors and fix

--
[...truncated 2.46 MB...]
org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
shouldAddStateStoreToRegexDefinedSource STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
shouldAddStateStoreToRegexDefinedSource PASSED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache FAILED
java.lang.AssertionError: Condition not met within timeout 6. Expecting 
5 records from topic map-one-join-output-1 while only received 0: []
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:275)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:201)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.receiveMessages(KStreamRepartitionJoinTest.java:375)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.verifyCorrectOutput(KStreamRepartitionJoinTest.java:296)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.verifyRepartitionOnJoinOperations(KStreamRepartitionJoinTest.java:141)
at 
org.apache.kafka.streams.integration.KStreamRepartitionJoinTest.shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache(KStreamRepartitionJoinTest.java:119)

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithNonZeroSizedCache STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithNonZeroSizedCache PASSED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldCompactTopicsForStateChangelogs STARTED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldCompactTopicsForStateChangelogs PASSED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldUseCompactAndDeleteForWindowStoreChangelogs STARTED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldUseCompactAndDeleteForWindowStoreChangelogs PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldGroupByKey STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldGroupByKey PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceWindowed STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceWindowed PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountSessionWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountSessionWindows PASSED

[GitHub] kafka pull request #3732: MINOR: doc changes for KIP-138

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/3732

MINOR: doc changes for KIP-138

1. Core concepts (added the stream time definition), upgrade guide and 
developer guide.
2. Related Java docs changes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka KMinor-kip138-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3732.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3732


commit c13b3161cee31da24e301fe5399eaf79a01a924e
Author: Guozhang Wang 
Date:   2017-08-24T23:54:06Z

doc changes for KIP-138




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3731: KAFKA-5720: Fix AdminClientIntegrationTest#testCal...

[cmccabe]

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/3731

KAFKA-5720: Fix AdminClientIntegrationTest#testCallInFlightTimeouts

* When a call is aborted, that should count as a "try" in the failure log 
message.
* FailureInjectingTimeoutProcessorFactory should fail the first request it 
is asked about.
* testCallTimeouts should expect the first request it makes to fail because 
of the timeout we injected.
* FailureInjectingTimeoutProcessorFactory should track how many failures it 
has injected, and the test should verify that one has been injected.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-5720

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3731.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3731


commit 3749bda7d3ee5d50cbc10081a8d7cbaef30617b3
Author: Colin P. Mccabe 
Date:   2017-08-24T23:22:44Z

KAFKA-5720: Fix AdminClientIntegrationTest#testCallInFlightTimeouts




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5342) Distinguish abortable failures in transactional producer

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-5342.

   Resolution: Fixed
Fix Version/s: 1.0.0

Issue resolved by pull request 3716
[https://github.com/apache/kafka/pull/3716]

> Distinguish abortable failures in transactional producer
> 
>
> Key: KAFKA-5342
> URL: https://issues.apache.org/jira/browse/KAFKA-5342
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 1.0.0, 0.11.0.1
>
>
> The transactional producer distinguishes two classes of user-visible errors:
> 1. Abortable errors: these are errors which are fatal to the ongoing 
> transaction, but which can be successfully aborted. Essentially any error in 
> which the producer can still expect to successfully send EndTxn to the 
> transaction coordinator is abortable.
> 2. Fatal errors: any error which is not abortable is fatal. For example, a 
> transactionalId authorization error is fatal because it would also prevent 
> the TC from receiving the EndTxn request.
> At the moment, it's not clear how the user would know how they should handle 
> a given failure. One option is to add an exception type to indicate which 
> errors are abortable (e.g. AbortableKafkaException). Then any other exception 
> could be considered fatal.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3716: KAFKA-5342: Clarify fatal/abortable exceptions use...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3716


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5784) Add a sensor for dropped records in window stores

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-5784:


 Summary: Add a sensor for dropped records in window stores
 Key: KAFKA-5784
 URL: https://issues.apache.org/jira/browse/KAFKA-5784
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang


Today when a {{put(record)}} call on a windowed store does not find the 
corresponding segment, i.e. its corresponding's window has expired, we simply 
returns a {{null}} and hence silently drops it.

We should consider 1) add log4j entries when it happens, 2) add metrics (we can 
discuss whether it should be a processor-node level, or store level sensor) for 
such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5783) Implement KafkaPrincipalBuilder interface with support for SASL (KIP-189)

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-5783:
--

 Summary: Implement KafkaPrincipalBuilder interface with support 
for SASL (KIP-189)
 Key: KAFKA-5783
 URL: https://issues.apache.org/jira/browse/KAFKA-5783
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson


This issue covers the implementation of 
[KIP-189|https://cwiki.apache.org/confluence/display/KAFKA/KIP-189%3A+Improve+principal+builder+interface+and+add+support+for+SASL].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Sriram Subramanian]

+1

On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang  wrote:

> +1. Thanks Damian!
>
> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck  wrote:
>
> > Thanks for the KIP!
> >
> > +1
> >
> > Thanks,
> > Bill
> >
> > On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy 
> wrote:
> >
> > > Hi,
> > >
> > > I'd like to kick off the voting thread for KIP-182:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> > > use+of+custom+storage+engines
> > >
> > > Thanks,
> > > Damian
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Jason Gustafson]

Hey Becket,

Yeah, my point is that we should never retry a batch using a different
PID/sequence. In the current implementation, the PID/sequence is assigned
when the batch is dequeued and it is never changed after that. Even if we
reset the PID due to an error on another partition, we will continue
retrying batches with the old PID/sequence until we receive a failure. I
think the client needs to guarantee this or duplicates will be possible.

-Jason

On Thu, Aug 24, 2017 at 12:52 PM, Becket Qin  wrote:

> Hi Jason,
>
> delivery.timeout.ms sounds good to me.
>
> I was referring to the case that we are resetting the PID/sequence after
> expire a batch. This is more about the sending the batches after the
> expired batch.
>
> The scenario being discussed is expiring one of the batches in a in-flight
> request and retry the other batches in the that in-flight request. So
> consider the following case:
> 1. Producer sends request_0 with two batches (batch_0_tp0 and batch_0_tp1).
> 2. Broker receives the request enqueued the request to the log.
> 3. Before the producer receives the response from the broker, batch_0_tp0
> expires. The producer will expire batch_0_tp0 immediately, resets PID, and
> then resend batch_0_tp1, and maybe send batch_1_tp0 (i.e. the next batch to
> the expired batch) as well.
>
> For batch_0_tp1, it is OK to reuse PID and and sequence number. The problem
> is for batch_1_tp0, If we reuse the same PID and the broker has already
> appended batch_0_tp0, the broker will think batch_1_tp0 is a duplicate with
> the same sequence number. As a result broker will drop batch_0_tp1. That is
> why we have to either bump up sequence number or reset PID. To avoid this
> complexity, I was suggesting not expire the in-flight batch immediately,
> but wait for the produce response. If the batch has been successfully
> appended, we do not expire it. Otherwise, we expire it.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Thu, Aug 24, 2017 at 11:26 AM, Jason Gustafson 
> wrote:
>
> > @Becket
> >
> > Good point about unnecessarily resetting the PID in cases where we know
> the
> > request has failed. Might be worth opening a JIRA to try and improve
> this.
> >
> > So if we expire the batch prematurely and resend all
> > > the other batches in the same request, chances are there will be
> > > duplicates. If we wait for the response instead, it is less likely to
> > > introduce duplicates, and we may not need to reset the PID.
> >
> >
> > Not sure I follow this. Are you assuming that we change the batch
> > PID/sequence of the retried batches after resetting the PID? I think we
> > probably need to ensure that when we retry a batch, we always use the
> same
> > PID/sequence.
> >
> > By the way, as far as naming, `max.message.delivery.wait.ms` is quite a
> > mouthful. Could we shorten it? Perhaps `delivery.timeout.ms`?
> >
> > -Jason
> >
> > On Wed, Aug 23, 2017 at 8:51 PM, Becket Qin 
> wrote:
> >
> > > Hi Jun,
> > >
> > > If TCP timeout is longer than request.timeout.ms, the producer will
> > always
> > > hit request.timeout.ms before hitting TCP timeout, right? That is why
> we
> > > added request.timeout.ms in the first place.
> > >
> > > You are right. Currently we are reset the PID and resend the batches to
> > > avoid OutOfOrderSequenceException when the expired batches are in
> retry.
> > >
> > > This does not distinguish the reasons that caused the retry. There are
> > two
> > > cases:
> > > 1. If the batch was in retry because it received an error response
> (e.g.
> > > NotLeaderForPartition), we actually don't need to reset PID in this
> case
> > > because we know that broker did not accept it.
> > > 2. If the batch was in retry because it hit a timeout earlier, then we
> > > should reset the PID (or optimistically send and only reset PID when
> > > receive OutOfOrderSequenceException?)
> > > Case 1 is probably the most common case, so it looks that we are
> > resetting
> > > the PID more often than necessary. But because in case 1 the broker
> does
> > > not have the batch, there isn't much impact on resting PID and resend
> > other
> > > than the additional round trip.
> > >
> > > Now we are introducing another case:
> > > 3. A batch is in retry because we expired an in-flight request before
> it
> > > hits request.timeout.ms.
> > >
> > > The difference between 2 and 3 is that in case 3 likely the broker has
> > > appended the messages. So if we expire the batch prematurely and resend
> > all
> > > the other batches in the same request, chances are there will be
> > > duplicates. If we wait for the response instead, it is less likely to
> > > introduce duplicates, and we may not need to reset the PID.
> > >
> > > That said, given that batch expiration is probably already rare enough,
> > so
> > > it may not be necessary to optimize for that.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On Wed, Aug 

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Matthias J. Sax]

Thanks for clarification. I see your point. Java varargs are problematic
in general IMHO as they force you to put them as last argument making
parameter ordering unnatural for some cases (as we have it currently in
the API).

Nevertheless, I think that reading a single topic is the most common
case and thus I would love to see the overloads as mentioned in my last
email in addition to the overloads taking a Collection of topics. Maybe
it's just personal taste -- I agree that the overhead of specifying a
singleton on not severe, but to me it still feels like a "step backward"
as reading a single topic should be the pattern for like 90% or more of
the cases.


-Matthias


On 8/24/17 12:03 PM, Guozhang Wang wrote:
> Matthias,
> 
> I think it's my bad that I did not post another comment on the mailing list
> while syncing with Damian. Here it is:
> 
> Regarding 1) above, a second thought on varargs: though I have not heard
> from anyone using multiple topics, it is also true that people will just
> keep silent until their APIs gets removed. So instead of keeping a single
> topic name in the constructor, it'd better to still allow users to pass
> multiple topics, as a Collection topic.
> 
> It does mean that users who would only want a single topic would feel
> inconvenient with "Collections.singleton(topic)", but I felt it is not too
> big of an issue. On the other hand KafkaConsumer also only allow
> `subscribe(Collection topics)` so I'd suggest in this KIP we do not
> have two overloads of "stream(topic)" and "stream(topics)" and consider
> adding that as a syntax-sugar if it does become a big complaint.
> 
> 
> Guozhang
> 
> 
> 
> On Thu, Aug 24, 2017 at 11:32 AM, Matthias J. Sax 
> wrote:
> 
>> We now have
>>
>>> public synchronized  KStream stream(final Collection
>> topic, final Consumed options)
>>
>> This would prevent so write code like
>>
>> builder.stream("topic", Consumers.with(...));
>>
>> I think, we need methods
>>
>> StreamsBuilder#stream(String topic);
>> StreamsBuilder#stream(String topic, Consumed options);
>>
>> Or do I miss anything?
>>
>>
>> -Matthias
>>
>>
>> On 8/24/17 1:53 AM, Damian Guy wrote:
>>> I've updated the kip to reflect Bill's comment and also to make
>>> StreamBuilder methods have topic as the first param, i.e.,
>>> StreamBuilder#stream no longer accepts varargs.
>>>
>>> On Thu, 24 Aug 2017 at 09:12 Damian Guy  wrote:
>>>
 On Thu, 24 Aug 2017 at 02:49 Guozhang Wang  wrote:

> I have a couple of comments but otherwise it LGTM:
>
> 1. For these two functions in StreamsBuilder, the topic String is set
>> as
> the second parameter in between of two options. Would that be better
>> to be
> set as the first or the last one instead?
>
> It would be better as the first, but then it is different to the
 #streams() methods due to varargs.


> public synchronized  KTable table(final Consumed
> consumed, final String topic, final Materialized materialized)
>
> public synchronized  GlobalKTable globalTable(final
> Consumed V> consumed, final String topic, final Materialized materialized)
>
> I understand that we cannot do it for the first parameter because of
>> the
> vararg type. So I'd suggest either
>
> a) set it as the last parameter, but then it is inconsistent with other
> functions like these:
>
> void to(final String topic, final Produced options);
>
> KTable through(final String topic, final Materialized
> options);
>
> b) only allow one single topic name parameter in
>> StreamsBuilder.stream()
> since in practice we do not see too many usages of multiple topics,
>> plus
> it
> can be semi-supported with "merge" as we move it from StreamsBuilder to
> KStream (KAFKA-5765),
>
> Perhaps this is the better approach


> 2. KGroupedStream's function:
>
>  KTable aggregate(final Initializer initializer,
>  final Aggregator
> aggregator,
>  final Serde aggValueSerde,
>  final Materialized VR>> materialized);
>
> The "aggValueSerde" seems not needed?
>
> 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think KGroupedStream
> was
> a bad name as a hind-sight. I personally feel we should just correct it
> with a new class and deprecate / remove the old one before 1.0.0, but
>> that
> could be in its own KIP.
>
>
 The problem with this is that we'd need to add new `groupBy` and
 `groupByKey` methods that return `GroupedKStream`, we can't change the
 existing ones as that would break compatibility. So what would we name
 these methods?


>
> Guozhang
>
>
>
> 

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Jun Rao]

Congratulations, Jiangjie!

Jun

On Wed, Aug 23, 2017 at 10:20 PM, Joel Koshy  wrote:

> Hi everyone,
>
> Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has
> contributed significantly to several major patches, reviews and discussions
> since. I am glad to announce that Becket is now a member of the Apache
> Kafka
>  PMC.
>
> Congratulations Becket!
>
> Joel
>

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Becket Qin]

Hi Jason,

delivery.timeout.ms sounds good to me.

I was referring to the case that we are resetting the PID/sequence after
expire a batch. This is more about the sending the batches after the
expired batch.

The scenario being discussed is expiring one of the batches in a in-flight
request and retry the other batches in the that in-flight request. So
consider the following case:
1. Producer sends request_0 with two batches (batch_0_tp0 and batch_0_tp1).
2. Broker receives the request enqueued the request to the log.
3. Before the producer receives the response from the broker, batch_0_tp0
expires. The producer will expire batch_0_tp0 immediately, resets PID, and
then resend batch_0_tp1, and maybe send batch_1_tp0 (i.e. the next batch to
the expired batch) as well.

For batch_0_tp1, it is OK to reuse PID and and sequence number. The problem
is for batch_1_tp0, If we reuse the same PID and the broker has already
appended batch_0_tp0, the broker will think batch_1_tp0 is a duplicate with
the same sequence number. As a result broker will drop batch_0_tp1. That is
why we have to either bump up sequence number or reset PID. To avoid this
complexity, I was suggesting not expire the in-flight batch immediately,
but wait for the produce response. If the batch has been successfully
appended, we do not expire it. Otherwise, we expire it.

Thanks,

Jiangjie (Becket) Qin



On Thu, Aug 24, 2017 at 11:26 AM, Jason Gustafson 
wrote:

> @Becket
>
> Good point about unnecessarily resetting the PID in cases where we know the
> request has failed. Might be worth opening a JIRA to try and improve this.
>
> So if we expire the batch prematurely and resend all
> > the other batches in the same request, chances are there will be
> > duplicates. If we wait for the response instead, it is less likely to
> > introduce duplicates, and we may not need to reset the PID.
>
>
> Not sure I follow this. Are you assuming that we change the batch
> PID/sequence of the retried batches after resetting the PID? I think we
> probably need to ensure that when we retry a batch, we always use the same
> PID/sequence.
>
> By the way, as far as naming, `max.message.delivery.wait.ms` is quite a
> mouthful. Could we shorten it? Perhaps `delivery.timeout.ms`?
>
> -Jason
>
> On Wed, Aug 23, 2017 at 8:51 PM, Becket Qin  wrote:
>
> > Hi Jun,
> >
> > If TCP timeout is longer than request.timeout.ms, the producer will
> always
> > hit request.timeout.ms before hitting TCP timeout, right? That is why we
> > added request.timeout.ms in the first place.
> >
> > You are right. Currently we are reset the PID and resend the batches to
> > avoid OutOfOrderSequenceException when the expired batches are in retry.
> >
> > This does not distinguish the reasons that caused the retry. There are
> two
> > cases:
> > 1. If the batch was in retry because it received an error response (e.g.
> > NotLeaderForPartition), we actually don't need to reset PID in this case
> > because we know that broker did not accept it.
> > 2. If the batch was in retry because it hit a timeout earlier, then we
> > should reset the PID (or optimistically send and only reset PID when
> > receive OutOfOrderSequenceException?)
> > Case 1 is probably the most common case, so it looks that we are
> resetting
> > the PID more often than necessary. But because in case 1 the broker does
> > not have the batch, there isn't much impact on resting PID and resend
> other
> > than the additional round trip.
> >
> > Now we are introducing another case:
> > 3. A batch is in retry because we expired an in-flight request before it
> > hits request.timeout.ms.
> >
> > The difference between 2 and 3 is that in case 3 likely the broker has
> > appended the messages. So if we expire the batch prematurely and resend
> all
> > the other batches in the same request, chances are there will be
> > duplicates. If we wait for the response instead, it is less likely to
> > introduce duplicates, and we may not need to reset the PID.
> >
> > That said, given that batch expiration is probably already rare enough,
> so
> > it may not be necessary to optimize for that.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Wed, Aug 23, 2017 at 5:01 PM, Jun Rao  wrote:
> >
> > > Hi, Becket,
> > >
> > > If a message expires while it's in an inflight produce request, the
> > > producer will get a new PID if idempotent is enabled. This will prevent
> > > subsequent messages from hitting OutOfOrderSequenceException. The issue
> > of
> > > not expiring an inflight request is that if a broker server goes down
> > hard
> > > (e.g. power outage), the time that it takes for the client to detect
> the
> > > socket level error (this will be sth like 8+ minutes with the default
> TCP
> > > setting) is much longer than the default request.timeout.ms.
> > >
> > > Hi, Sumant,
> > >
> > > We can probably just default max.message.delivery.wait.ms to 30 secs,
> > the
> > > 

[jira] [Created] (KAFKA-5782) Avoid unnecessary PID reset when expire batches.

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-5782:
---

 Summary: Avoid unnecessary PID reset when expire batches.
 Key: KAFKA-5782
 URL: https://issues.apache.org/jira/browse/KAFKA-5782
 Project: Kafka
  Issue Type: Improvement
  Components: producer 
Affects Versions: 0.11.0.0
Reporter: Jiangjie Qin
 Fix For: 1.0.0


This is more of an efficiency optimization. Currently we will reset PID when 
batch expiration happens and one of the expired batches is in retry mode. This 
is assuming that we don't know if the batch in retry has been appended to the 
broker or not. However, if the batch was in retry due to a retriable exception 
returned by the broker, the batch is not appended. In this case, we do not need 
to reset the PID.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

[Vahid S Hashemian]

+1

Thanks Rajini.

--Vahid



From:   Edoardo Comar 
To: dev@kafka.apache.org
Date:   08/24/2017 10:55 AM
Subject:Re: [VOTE] KIP-152 - Improve diagnostics for SASL 
authentication failures



Thanks Rajini!

+1 (non-binding)
--

Edoardo Comar

IBM Message Hub

IBM UK Ltd, Hursley Park, SO21 2JN



From:   Rajini Sivaram 
To: dev 
Date:   24/08/2017 18:30
Subject:[VOTE] KIP-152 - Improve diagnostics for SASL 
authentication failures



Hi all,

I would like to start vote on KIP-152 to improve diagnostics of
authentication failures and to update clients to treat authentication
failures as fatal exceptions rather than transient errors:
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D152-2B-2D-2BImprove-2Bdiagnostics-2Bfor-2BSASL-2Bauthentication-2Bfailures=DwIBAg=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=oomurFvX7GYTMP0ZlfXU8eQSS68MfeCQYhvtB7zXrCw=AaHp43w6RWNlh_1HZc6_rdangxC-oXMWHz7XR6i0suA=
 



Thank you...

Rajini



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Becket Qin]

Thanks everyone!

On Thu, Aug 24, 2017 at 11:27 AM, Jason Gustafson 
wrote:

> Congrats Becket!
>
> On Thu, Aug 24, 2017 at 11:15 AM, Ismael Juma  wrote:
>
> > Congratulations Becket!
> >
> > On 24 Aug 2017 6:20 am, "Joel Koshy"  wrote:
> >
> > Hi everyone,
> >
> > Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has
> > contributed significantly to several major patches, reviews and
> discussions
> > since. I am glad to announce that Becket is now a member of the Apache
> > Kafka
> >  PMC.
> >
> > Congratulations Becket!
> >
> > Joel
> >
>

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Guozhang Wang]

Matthias,

I think it's my bad that I did not post another comment on the mailing list
while syncing with Damian. Here it is:

Regarding 1) above, a second thought on varargs: though I have not heard
from anyone using multiple topics, it is also true that people will just
keep silent until their APIs gets removed. So instead of keeping a single
topic name in the constructor, it'd better to still allow users to pass
multiple topics, as a Collection topic.

It does mean that users who would only want a single topic would feel
inconvenient with "Collections.singleton(topic)", but I felt it is not too
big of an issue. On the other hand KafkaConsumer also only allow
`subscribe(Collection topics)` so I'd suggest in this KIP we do not
have two overloads of "stream(topic)" and "stream(topics)" and consider
adding that as a syntax-sugar if it does become a big complaint.


Guozhang



On Thu, Aug 24, 2017 at 11:32 AM, Matthias J. Sax 
wrote:

> We now have
>
> > public synchronized  KStream stream(final Collection
> topic, final Consumed options)
>
> This would prevent so write code like
>
> builder.stream("topic", Consumers.with(...));
>
> I think, we need methods
>
> StreamsBuilder#stream(String topic);
> StreamsBuilder#stream(String topic, Consumed options);
>
> Or do I miss anything?
>
>
> -Matthias
>
>
> On 8/24/17 1:53 AM, Damian Guy wrote:
> > I've updated the kip to reflect Bill's comment and also to make
> > StreamBuilder methods have topic as the first param, i.e.,
> > StreamBuilder#stream no longer accepts varargs.
> >
> > On Thu, 24 Aug 2017 at 09:12 Damian Guy  wrote:
> >
> >> On Thu, 24 Aug 2017 at 02:49 Guozhang Wang  wrote:
> >>
> >>> I have a couple of comments but otherwise it LGTM:
> >>>
> >>> 1. For these two functions in StreamsBuilder, the topic String is set
> as
> >>> the second parameter in between of two options. Would that be better
> to be
> >>> set as the first or the last one instead?
> >>>
> >>> It would be better as the first, but then it is different to the
> >> #streams() methods due to varargs.
> >>
> >>
> >>> public synchronized  KTable table(final Consumed
> >>> consumed, final String topic, final Materialized materialized)
> >>>
> >>> public synchronized  GlobalKTable globalTable(final
> >>> Consumed >>> V> consumed, final String topic, final Materialized materialized)
> >>>
> >>> I understand that we cannot do it for the first parameter because of
> the
> >>> vararg type. So I'd suggest either
> >>>
> >>> a) set it as the last parameter, but then it is inconsistent with other
> >>> functions like these:
> >>>
> >>> void to(final String topic, final Produced options);
> >>>
> >>> KTable through(final String topic, final Materialized
> >>> options);
> >>>
> >>> b) only allow one single topic name parameter in
> StreamsBuilder.stream()
> >>> since in practice we do not see too many usages of multiple topics,
> plus
> >>> it
> >>> can be semi-supported with "merge" as we move it from StreamsBuilder to
> >>> KStream (KAFKA-5765),
> >>>
> >>> Perhaps this is the better approach
> >>
> >>
> >>> 2. KGroupedStream's function:
> >>>
> >>>  KTable aggregate(final Initializer initializer,
> >>>  final Aggregator
> >>> aggregator,
> >>>  final Serde aggValueSerde,
> >>>  final Materialized >>> VR>> materialized);
> >>>
> >>> The "aggValueSerde" seems not needed?
> >>>
> >>> 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think KGroupedStream
> >>> was
> >>> a bad name as a hind-sight. I personally feel we should just correct it
> >>> with a new class and deprecate / remove the old one before 1.0.0, but
> that
> >>> could be in its own KIP.
> >>>
> >>>
> >> The problem with this is that we'd need to add new `groupBy` and
> >> `groupByKey` methods that return `GroupedKStream`, we can't change the
> >> existing ones as that would break compatibility. So what would we name
> >> these methods?
> >>
> >>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>>
> >>> On Wed, Aug 23, 2017 at 1:01 PM, Damian Guy 
> wrote:
> >>>
>  We already have GlobalKTable and i can't rename KGroupedStream, which
>  really should be GroupedKStream. So I think we should name new things
>  correctly, i.e., WindowedKStream etc and fix the others when we can.
> 
>  On Wed, 23 Aug 2017 at 20:38 Matthias J. Sax 
>  wrote:
> 
> > About KGroupedStream vs GroupedKStream: shouldn't we keep the naming
> > convention consistent? And if we change the naming schema just change
> > all at once? I personally don't care which naming scheme is better,
> >>> but
> > I think consistency is super important!
> >
> > About Bill's comment: I agree, and had a similar thought.
> >
> >
> 

Re: [DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

[Jason Gustafson]

Seems reasonable. I don't recall any specific reason for not providing this
method initially.

-Jason

On Thu, Aug 24, 2017 at 5:50 AM, Attila Kreiner  wrote:

> Hi All,
>
> I created KIP-191:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 191%3A+KafkaConsumer.subscribe%28%29+overload+that+takes+just+Pattern
>
> Jira: https://issues.apache.org/jira/browse/KAFKA-5726
> PR: https://github.com/apache/kafka/pull/3669
>
> Please check it.
>
> Thanks,
> Attila
>

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Matthias J. Sax]

We now have

> public synchronized  KStream stream(final Collection 
> topic, final Consumed options)

This would prevent so write code like

builder.stream("topic", Consumers.with(...));

I think, we need methods

StreamsBuilder#stream(String topic);
StreamsBuilder#stream(String topic, Consumed options);

Or do I miss anything?


-Matthias


On 8/24/17 1:53 AM, Damian Guy wrote:
> I've updated the kip to reflect Bill's comment and also to make
> StreamBuilder methods have topic as the first param, i.e.,
> StreamBuilder#stream no longer accepts varargs.
> 
> On Thu, 24 Aug 2017 at 09:12 Damian Guy  wrote:
> 
>> On Thu, 24 Aug 2017 at 02:49 Guozhang Wang  wrote:
>>
>>> I have a couple of comments but otherwise it LGTM:
>>>
>>> 1. For these two functions in StreamsBuilder, the topic String is set as
>>> the second parameter in between of two options. Would that be better to be
>>> set as the first or the last one instead?
>>>
>>> It would be better as the first, but then it is different to the
>> #streams() methods due to varargs.
>>
>>
>>> public synchronized  KTable table(final Consumed
>>> consumed, final String topic, final Materialized materialized)
>>>
>>> public synchronized  GlobalKTable globalTable(final
>>> Consumed>> V> consumed, final String topic, final Materialized materialized)
>>>
>>> I understand that we cannot do it for the first parameter because of the
>>> vararg type. So I'd suggest either
>>>
>>> a) set it as the last parameter, but then it is inconsistent with other
>>> functions like these:
>>>
>>> void to(final String topic, final Produced options);
>>>
>>> KTable through(final String topic, final Materialized
>>> options);
>>>
>>> b) only allow one single topic name parameter in StreamsBuilder.stream()
>>> since in practice we do not see too many usages of multiple topics, plus
>>> it
>>> can be semi-supported with "merge" as we move it from StreamsBuilder to
>>> KStream (KAFKA-5765),
>>>
>>> Perhaps this is the better approach
>>
>>
>>> 2. KGroupedStream's function:
>>>
>>>  KTable aggregate(final Initializer initializer,
>>>  final Aggregator
>>> aggregator,
>>>  final Serde aggValueSerde,
>>>  final Materialized>> VR>> materialized);
>>>
>>> The "aggValueSerde" seems not needed?
>>>
>>> 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think KGroupedStream
>>> was
>>> a bad name as a hind-sight. I personally feel we should just correct it
>>> with a new class and deprecate / remove the old one before 1.0.0, but that
>>> could be in its own KIP.
>>>
>>>
>> The problem with this is that we'd need to add new `groupBy` and
>> `groupByKey` methods that return `GroupedKStream`, we can't change the
>> existing ones as that would break compatibility. So what would we name
>> these methods?
>>
>>
>>>
>>> Guozhang
>>>
>>>
>>>
>>> On Wed, Aug 23, 2017 at 1:01 PM, Damian Guy  wrote:
>>>
 We already have GlobalKTable and i can't rename KGroupedStream, which
 really should be GroupedKStream. So I think we should name new things
 correctly, i.e., WindowedKStream etc and fix the others when we can.

 On Wed, 23 Aug 2017 at 20:38 Matthias J. Sax 
 wrote:

> About KGroupedStream vs GroupedKStream: shouldn't we keep the naming
> convention consistent? And if we change the naming schema just change
> all at once? I personally don't care which naming scheme is better,
>>> but
> I think consistency is super important!
>
> About Bill's comment: I agree, and had a similar thought.
>
>
> -Matthias
>
> On 8/23/17 12:24 PM, Bill Bejeck wrote:
>> Thanks for all the work on this KIP Damian.
>>
>> Both `Produced` and `Joined` have a `with` method accepting all
> parameters,
>> but `Consumed` doesn't. Should we add one for consistency?
>>
>> Thanks,
>> Bill
>>
>> On Wed, Aug 23, 2017 at 4:15 AM, Damian Guy 
> wrote:
>>
>>> KIP has been updated. thanks
>>>
>>> On Wed, 23 Aug 2017 at 09:10 Damian Guy 
>>> wrote:
>>>
 Hi Matthias,


> KStream:
> leftJoin and outerJoin for KStream/KTable join should not have
> `JoinWindows` parameter
>
> Thanks!


>
> Nit: TopologyBuilder -> Topology
>
> Ack


> Nit: new class Serialized list static method #with twice
>
> Ack


> WindowedKStream -> for consistency we should either have
> GroupedKStream
> or KWindowedStream... (similar argument for
>>> SessionWindowedKStream)
>
> We can't rename KGroupedStream -> 

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Jason Gustafson]

Congrats Becket!

On Thu, Aug 24, 2017 at 11:15 AM, Ismael Juma  wrote:

> Congratulations Becket!
>
> On 24 Aug 2017 6:20 am, "Joel Koshy"  wrote:
>
> Hi everyone,
>
> Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has
> contributed significantly to several major patches, reviews and discussions
> since. I am glad to announce that Becket is now a member of the Apache
> Kafka
>  PMC.
>
> Congratulations Becket!
>
> Joel
>

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Jason Gustafson]

@Becket

Good point about unnecessarily resetting the PID in cases where we know the
request has failed. Might be worth opening a JIRA to try and improve this.

So if we expire the batch prematurely and resend all
> the other batches in the same request, chances are there will be
> duplicates. If we wait for the response instead, it is less likely to
> introduce duplicates, and we may not need to reset the PID.


Not sure I follow this. Are you assuming that we change the batch
PID/sequence of the retried batches after resetting the PID? I think we
probably need to ensure that when we retry a batch, we always use the same
PID/sequence.

By the way, as far as naming, `max.message.delivery.wait.ms` is quite a
mouthful. Could we shorten it? Perhaps `delivery.timeout.ms`?

-Jason

On Wed, Aug 23, 2017 at 8:51 PM, Becket Qin  wrote:

> Hi Jun,
>
> If TCP timeout is longer than request.timeout.ms, the producer will always
> hit request.timeout.ms before hitting TCP timeout, right? That is why we
> added request.timeout.ms in the first place.
>
> You are right. Currently we are reset the PID and resend the batches to
> avoid OutOfOrderSequenceException when the expired batches are in retry.
>
> This does not distinguish the reasons that caused the retry. There are two
> cases:
> 1. If the batch was in retry because it received an error response (e.g.
> NotLeaderForPartition), we actually don't need to reset PID in this case
> because we know that broker did not accept it.
> 2. If the batch was in retry because it hit a timeout earlier, then we
> should reset the PID (or optimistically send and only reset PID when
> receive OutOfOrderSequenceException?)
> Case 1 is probably the most common case, so it looks that we are resetting
> the PID more often than necessary. But because in case 1 the broker does
> not have the batch, there isn't much impact on resting PID and resend other
> than the additional round trip.
>
> Now we are introducing another case:
> 3. A batch is in retry because we expired an in-flight request before it
> hits request.timeout.ms.
>
> The difference between 2 and 3 is that in case 3 likely the broker has
> appended the messages. So if we expire the batch prematurely and resend all
> the other batches in the same request, chances are there will be
> duplicates. If we wait for the response instead, it is less likely to
> introduce duplicates, and we may not need to reset the PID.
>
> That said, given that batch expiration is probably already rare enough, so
> it may not be necessary to optimize for that.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Wed, Aug 23, 2017 at 5:01 PM, Jun Rao  wrote:
>
> > Hi, Becket,
> >
> > If a message expires while it's in an inflight produce request, the
> > producer will get a new PID if idempotent is enabled. This will prevent
> > subsequent messages from hitting OutOfOrderSequenceException. The issue
> of
> > not expiring an inflight request is that if a broker server goes down
> hard
> > (e.g. power outage), the time that it takes for the client to detect the
> > socket level error (this will be sth like 8+ minutes with the default TCP
> > setting) is much longer than the default request.timeout.ms.
> >
> > Hi, Sumant,
> >
> > We can probably just default max.message.delivery.wait.ms to 30 secs,
> the
> > current default for request.timeout.ms.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Aug 23, 2017 at 3:38 PM, Sumant Tambe  wrote:
> >
> > > OK. Looks like starting the clock after closing the batch has quite a
> few
> > > pitfalls. I can't think of a way of to work around it without adding
> yet
> > > another config. So I won't discuss that here. Anyone? As I said
> earlier,
> > > I'm not hung up on super-accurate notification times.
> > >
> > > If we are going down the max.message.delievery.wait.ms route, what
> would
> > > be
> > > the default? There seem to be a few options.
> > >
> > > 1. max.message.delievery.wait.ms=null. Nothing changes for those who
> > don't
> > > set it. I.e., batches expire after request.timeout.ms in accumulator.
> If
> > > they are past the accumulator stage, timeout after retries*(
> > > request.timeout.ms+backoff).
> > >
> > > 2. max.message.delivery.wait.ms=request.timeout.ms. No obervable
> > > behavioral
> > > change at the accumulator level as timeout value is same as before.
> > Retries
> > > will be done if as long as batch is under max.message.delivery.wait.ms
> .
> > > However, a batch can expire just after one try. That's ok IMO because
> > > request.timeout.ms tend to be large (Default 3).
> > >
> > > 3. max.message.delivery.wait.ms=2*request.timeout.ms. Give opportunity
> > for
> > > two retries but warn that retries may not happen at all in some rare
> > > cases and a batch could expire before any attempt.
> > >
> > > 4. max.message.delivery.wait.ms=something else (a constant?)
> > >
> > > Thoughts?
> > >
> > > On 23 August 2017 at 09:01, 

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Ismael Juma]

Congratulations Becket!

On 24 Aug 2017 6:20 am, "Joel Koshy"  wrote:

Hi everyone,

Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has
contributed significantly to several major patches, reviews and discussions
since. I am glad to announce that Becket is now a member of the Apache Kafka
 PMC.

Congratulations Becket!

Joel

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

[Edoardo Comar]

Thanks Rajini!

+1 (non-binding)
--

Edoardo Comar

IBM Message Hub

IBM UK Ltd, Hursley Park, SO21 2JN



From:   Rajini Sivaram 
To: dev 
Date:   24/08/2017 18:30
Subject:[VOTE] KIP-152 - Improve diagnostics for SASL 
authentication failures



Hi all,

I would like to start vote on KIP-152 to improve diagnostics of
authentication failures and to update clients to treat authentication
failures as fatal exceptions rather than transient errors:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures


Thank you...

Rajini



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[VOTE] KIP-188 - Add new metrics to support health checks

[Rajini Sivaram]

Hi all,

I would like to start the vote on KIP-188 that adds additional metrics to
support health checks for Kafka Ops. Details are here:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-188+-+Add+new+metrics+to+support+health+checks

Thank you,

Rajini

[VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

[Rajini Sivaram]

Hi all,

I would like to start the vote on KIP-187 that adds a cumulative count
metric associated with each Kafka rate metric to improve downstream
processing of rate metrics. Details are here:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics


Thank you,

Rajini

Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Mickael Maison]

Congratulations Becket!

On Thu, Aug 24, 2017 at 6:23 PM, Rajini Sivaram  wrote:
> Congratulations, Becket!
>
> Regards,
>
> Rajini
>
> On Thu, Aug 24, 2017 at 10:02 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
>> Congrats Becket!
>>
>> --Vahid
>>
>>
>>
>> From:   Abhishek Mendhekar 
>> To: dev@kafka.apache.org
>> Date:   08/24/2017 09:53 AM
>> Subject:Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie
>> (Becket) Qin
>>
>>
>>
>> Congrats Becket!
>>
>> On Thu, Aug 24, 2017 at 9:48 AM, Damian Guy  wrote:
>>
>> > Congratulations.
>> >
>> > On Thu, 24 Aug 2017 at 17:44 Matthias J. Sax 
>> > wrote:
>> >
>> > > Congrats!
>> > >
>> > > On 8/23/17 10:40 PM, Hu Xi wrote:
>> > > > Congrats Becket!
>> > > >
>> > > >
>> > > > 
>> > > > 发件人: Guozhang Wang 
>> > > > 发送时间: 2017年8月24日 13:32
>> > > > 收件人: dev@kafka.apache.org
>> > > > 主题: Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin
>> > > >
>> > > > Congrats Jiangjie!
>> > > >
>> > > >
>> > > > Guozhang
>> > > >
>> > > > On Wed, Aug 23, 2017 at 10:20 PM, Joel Koshy 
>> > > wrote:
>> > > >
>> > > >> Hi everyone,
>> > > >>
>> > > >> Jiangjie (Becket) Qin has been a Kafka committer in October 2016
>> and
>> > has
>> > > >> contributed significantly to several major patches, reviews and
>> > > discussions
>> > > >> since. I am glad to announce that Becket is now a member of the
>> Apache
>> > > >> Kafka
>> > > >>  PMC.
>> > > >>
>> > > >> Congratulations Becket!
>> > > >>
>> > > >> Joel
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > -- Guozhang
>> > > >
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Abhishek Mendhekar
>> abhishek.mendhe...@gmail.com | 818.263.7030
>>
>>
>>
>>
>>

Re: [DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

[Florian Hussonnois]

Hi Randall,

Thank you for your answer.

I will update the KIP and the PR with your last approach which sounds
better.

Thanks.

Le 16 août 2017 00:53, "Randall Hauch"  a écrit :

Sorry it's taken me so long to come back to this.

Have you considered creating a `SourceConnectorContext` interface that
extends `ConnectorContext` and that adds the method to access the offset
storage? This would very closely match the existing `SourceTaskContext`.

`SourceConnector` implementations could always cast the `context` field in
the superclass to `SourceConnectorContext`, but perhaps a slightly better
way to do this is to add the following method to the `SourceConnector`
class:


public SourceConnectorContext context() {
return (SourceConnectorContext)context;
}


Now, `SourceConnector` implementations can either cast themselves or use
this additional method to obtain the correctly cast context.

In fact, it might be good to do this for `SinkConnector` as well, and we
could even add a `context()` method in the `Connector` interface, since
subinterfaces can change the return type to be a subtype of that returned
by the interface:

ConnectorContext context();

One advantage of this approach is that `SourceConnectorContext` and
`SinkConnectorContext` remain interfaces. Another is not adding new method
to `SourceConnector` that implementers may need to learn that they should
not override or implement them. A third is that now we have a
`SourceConnectorContext` and `SinkConnectorContext` to which we can add
more methods if needed, and they are very similar to `SourceTaskContext`
and `SinkTaskContext`.

Thoughts?

On Wed, Apr 5, 2017 at 3:59 PM, Florian Hussonnois 
wrote:

> Hi All,
>
> Is there any feedback regarding that KIP ?
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+
> OffsetStorageReader+from+SourceConnector
>
> Thanks,
>
> 2017-03-14 22:51 GMT+01:00 Florian Hussonnois :
>
> > Hi Matthias,
> >
> > Sorry I didn't know this page. Ths KIP has been added to it.
> >
> > Thanks,
> >
> > 2017-03-13 21:30 GMT+01:00 Matthias J. Sax :
> >
> >> Can you please add the KIP to this table:
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+
> >> Improvement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion
> >>
> >> Thanks,
> >>
> >>  Matthias
> >>
> >>
> >> On 3/7/17 1:24 PM, Florian Hussonnois wrote:
> >> > Hi all,
> >> >
> >> > I've created a new KIP to add access to OffsetStorageReader from
> >> > SourceConnector
> >> >
> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+
> >> Add+access+to+OffsetStorageReader+from+SourceConnector
> >> >
> >> > Thanks.
> >> >
> >>
> >>
> >
> >
> > --
> > Florian HUSSONNOIS
> >
>
>
>
> --
> Florian HUSSONNOIS
>

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Xavier Léauté]

A few comments on the KIP:
- I'm a bit confused about the BytesStoreSupplier interface. Nothing in its
definition is really specific to Bytes, and
when I see return types like BytesStoreSupplier>, it seems redundant to have "Bytes" in the supplier name.
Why can't we reuse the existing StateStoreSupplier interface and move the
extra logConfig and loggingEnabled methods elsewhere?
- I don't really see any mention of the motivation behind the Materialized
interface and what the implications are for the user, i.e. what does it
mean for a store to be materialized.
- Until now, serialization implementation details were decoupled from the
state store interfaces. With this KIP we are now bubbling up the
assumptions that state store going to be using Bytes or byte[] into the
API. I'm not a fan of this, because it precludes us from providing more
efficient implementations, e.g. using ByteBuffers, that can avoid costly
byte array copying and extraneous byte array allocations during serde
operations.
A better approach might be to provide a first class ByteStore interface
that could help abstract the different types of buffers we might want to
use, or alternatively use a buffer agnostic type in the state store
definition (similar to what LMDB

 does)

On Thu, Aug 24, 2017 at 1:53 AM Damian Guy  wrote:

> I've updated the kip to reflect Bill's comment and also to make
> StreamBuilder methods have topic as the first param, i.e.,
> StreamBuilder#stream no longer accepts varargs.
>
> On Thu, 24 Aug 2017 at 09:12 Damian Guy  wrote:
>
> > On Thu, 24 Aug 2017 at 02:49 Guozhang Wang  wrote:
> >
> >> I have a couple of comments but otherwise it LGTM:
> >>
> >> 1. For these two functions in StreamsBuilder, the topic String is set as
> >> the second parameter in between of two options. Would that be better to
> be
> >> set as the first or the last one instead?
> >>
> >> It would be better as the first, but then it is different to the
> > #streams() methods due to varargs.
> >
> >
> >> public synchronized  KTable table(final Consumed
> >> consumed, final String topic, final Materialized materialized)
> >>
> >> public synchronized  GlobalKTable globalTable(final
> >> Consumed >> V> consumed, final String topic, final Materialized materialized)
> >>
> >> I understand that we cannot do it for the first parameter because of the
> >> vararg type. So I'd suggest either
> >>
> >> a) set it as the last parameter, but then it is inconsistent with other
> >> functions like these:
> >>
> >> void to(final String topic, final Produced options);
> >>
> >> KTable through(final String topic, final Materialized
> >> options);
> >>
> >> b) only allow one single topic name parameter in StreamsBuilder.stream()
> >> since in practice we do not see too many usages of multiple topics, plus
> >> it
> >> can be semi-supported with "merge" as we move it from StreamsBuilder to
> >> KStream (KAFKA-5765),
> >>
> >> Perhaps this is the better approach
> >
> >
> >> 2. KGroupedStream's function:
> >>
> >>  KTable aggregate(final Initializer initializer,
> >>  final Aggregator
> >> aggregator,
> >>  final Serde aggValueSerde,
> >>  final Materialized >> VR>> materialized);
> >>
> >> The "aggValueSerde" seems not needed?
> >>
> >> 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think KGroupedStream
> >> was
> >> a bad name as a hind-sight. I personally feel we should just correct it
> >> with a new class and deprecate / remove the old one before 1.0.0, but
> that
> >> could be in its own KIP.
> >>
> >>
> > The problem with this is that we'd need to add new `groupBy` and
> > `groupByKey` methods that return `GroupedKStream`, we can't change the
> > existing ones as that would break compatibility. So what would we name
> > these methods?
> >
> >
> >>
> >> Guozhang
> >>
> >>
> >>
> >> On Wed, Aug 23, 2017 at 1:01 PM, Damian Guy 
> wrote:
> >>
> >> > We already have GlobalKTable and i can't rename KGroupedStream, which
> >> > really should be GroupedKStream. So I think we should name new things
> >> > correctly, i.e., WindowedKStream etc and fix the others when we can.
> >> >
> >> > On Wed, 23 Aug 2017 at 20:38 Matthias J. Sax 
> >> > wrote:
> >> >
> >> > > About KGroupedStream vs GroupedKStream: shouldn't we keep the naming
> >> > > convention consistent? And if we change the naming schema just
> change
> >> > > all at once? I personally don't care which naming scheme is better,
> >> but
> >> > > I think consistency is super important!
> >> > >
> >> > > About Bill's comment: I agree, and had a similar thought.
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> 

[VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

[Rajini Sivaram]

Hi all,

I would like to start vote on KIP-152 to improve diagnostics of
authentication failures and to update clients to treat authentication
failures as fatal exceptions rather than transient errors:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures

Thank you...

Rajini

Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Rajini Sivaram]

Congratulations, Becket!

Regards,

Rajini

On Thu, Aug 24, 2017 at 10:02 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Congrats Becket!
>
> --Vahid
>
>
>
> From:   Abhishek Mendhekar 
> To: dev@kafka.apache.org
> Date:   08/24/2017 09:53 AM
> Subject:Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie
> (Becket) Qin
>
>
>
> Congrats Becket!
>
> On Thu, Aug 24, 2017 at 9:48 AM, Damian Guy  wrote:
>
> > Congratulations.
> >
> > On Thu, 24 Aug 2017 at 17:44 Matthias J. Sax 
> > wrote:
> >
> > > Congrats!
> > >
> > > On 8/23/17 10:40 PM, Hu Xi wrote:
> > > > Congrats Becket!
> > > >
> > > >
> > > > 
> > > > 发件人: Guozhang Wang 
> > > > 发送时间: 2017年8月24日 13:32
> > > > 收件人: dev@kafka.apache.org
> > > > 主题: Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin
> > > >
> > > > Congrats Jiangjie!
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Aug 23, 2017 at 10:20 PM, Joel Koshy 
> > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> Jiangjie (Becket) Qin has been a Kafka committer in October 2016
> and
> > has
> > > >> contributed significantly to several major patches, reviews and
> > > discussions
> > > >> since. I am glad to announce that Becket is now a member of the
> Apache
> > > >> Kafka
> > > >>  PMC.
> > > >>
> > > >> Congratulations Becket!
> > > >>
> > > >> Joel
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> >
>
>
>
> --
> Abhishek Mendhekar
> abhishek.mendhe...@gmail.com | 818.263.7030
>
>
>
>
>

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Mayuresh Gharat]

Sure.

Thanks,

Mayuresh

On Wed, Aug 23, 2017 at 5:07 PM, Jun Rao  wrote:

> Hi, Mayuresh,
>
> Since this KIP covers the requirement in KIP-111, could you review it too?
>
> Thanks,
>
> Jun
>
>
> On Tue, Aug 22, 2017 at 3:04 PM, Jason Gustafson 
> wrote:
>
>> Bump. I'll open a vote in a few days if there are no comments.
>>
>> Thanks,
>> Jason
>>
>> On Sat, Aug 19, 2017 at 12:28 AM, Ismael Juma  wrote:
>>
>> > Thanks for the KIP Jason. It seems reasonable and cleans up some
>> > inconsistencies in that area. It would be great to get some feedback
>> from
>> > Mayuresh and others who worked on KIP-111.
>> >
>> > Ismael
>> >
>> > On Thu, Aug 17, 2017 at 1:21 AM, Jason Gustafson 
>> > wrote:
>> >
>> > > Hi All,
>> > >
>> > > I've added a new KIP to improve and extend the principal building API
>> > that
>> > > Kafka exposes:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 189%3A+Improve+principal+builder+interface+and+add+support+for+SASL
>> > > .
>> > >
>> > > As always, feedback is appreciated.
>> > >
>> > > Thanks,
>> > > Jason
>> > >
>> >
>>
>
>


-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Guozhang Wang]

+1. Thanks Damian!

On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck  wrote:

> Thanks for the KIP!
>
> +1
>
> Thanks,
> Bill
>
> On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy  wrote:
>
> > Hi,
> >
> > I'd like to kick off the voting thread for KIP-182:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> > use+of+custom+storage+engines
> >
> > Thanks,
> > Damian
> >
>



-- 
-- Guozhang

[jira] [Resolved] (KAFKA-4350) Can't mirror from Kafka 0.9 to Kafka 0.10.1

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4350.
--
Resolution: Won't Fix

Closing as per comments.

> Can't mirror from Kafka 0.9 to Kafka 0.10.1
> ---
>
> Key: KAFKA-4350
> URL: https://issues.apache.org/jira/browse/KAFKA-4350
> Project: Kafka
>  Issue Type: Bug
>Reporter: Emanuele Cesena
>
> I'm running 2 clusters: K9 with Kafka 0.9 and K10 with Kafka 0.10.1.
> In K10, I've set up mirror maker to clone a topic from K9 to K10.
> Mirror maker immediately fails while starting, any suggestion? Following 
> error message and configs.
> Error message:
> {code:java} 
> [2016-10-26 23:54:01,663] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
> 'cluster_id': Error reading string of length 418, only 43 bytes available
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
> at 
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:193)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
> at 
> kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:582)
> at 
> kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:431)
> [2016-10-26 23:54:01,679] FATAL [mirrormaker-thread-0] Mirror maker thread 
> exited abnormally, stopping the whole mirror maker. 
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> {code} 
> Consumer:
> {code:} 
> group.id=mirrormaker001
> client.id=mirrormaker001
> bootstrap.servers=...K9...
> security.protocol=PLAINTEXT
> auto.offset.reset=earliest
> {code} 
> (note that I first run without client.id, then tried adding a client.id 
> because -- same error in both cases)
> Producer:
> {code:}
> bootstrap.servers=...K10...
> security.protocol=PLAINTEXT
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-trunk-jdk7 #2671

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-4109) kafka client send msg exception

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4109.
--
Resolution: Cannot Reproduce

 Pl reopen if you think the issue still exists


> kafka client send msg exception
> ---
>
> Key: KAFKA-4109
> URL: https://issues.apache.org/jira/browse/KAFKA-4109
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.0.0
> Environment: java8
> kafka cluster
>Reporter: frank
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Batch Expired 
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
>  
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
>  
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
>  
>   at 
> com.longsheng.basicCollect.kafka.KafkaProducer.publishMessage(KafkaProducer.java:44)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.ParentBasic.publish(ParentBasic.java:60)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.ParentBasic.parseIncJson(ParentBasic.java:119)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.coke.price.SecondPrice.collectData(SecondPrice.java:41)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.coke.price.SecondPrice.process(SecondPrice.java:49)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.DzwProcessor.process(DzwProcessor.java:33)
>  
>   at 
> com.longsheng.basicCollect.timer.CollectDzwBigData.execute(CollectDzwBigData.java:14)
>  
>   at org.quartz.core.JobRunShell.run(JobRunShell.java:202) 
>   at 
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) 
> Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
> the exception is arbitrarily!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk8 #1939

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: add upgrade section for 1.0.0

--
[...truncated 2.02 MB...]
org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFirstMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
invalidServerFirstMessage PASSED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFinalMessage STARTED

org.apache.kafka.common.security.scram.ScramMessagesTest > 
validServerFinalMessage PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryWithoutPasswordConfiguration STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryWithoutPasswordConfiguration PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration STARTED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration PASSED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse STARTED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameOverride PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testSingleOption STARTED

org.apache.kafka.common.security.JaasContextTest > testSingleOption PASSED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes STARTED

org.apache.kafka.common.security.JaasContextTest > 
testNumericOptionWithoutQuotes PASSED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testConfigNoOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithWrongListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
STARTED

org.apache.kafka.common.security.JaasContextTest > testNumericOptionWithQuotes 
PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionValue PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingLoginModule PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingSemicolon PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleOptions PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForClientWithListenerName PASSED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMultipleLoginModules 
PASSED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag 
STARTED

org.apache.kafka.common.security.JaasContextTest > testMissingControlFlag PASSED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback STARTED

org.apache.kafka.common.security.JaasContextTest > 
testLoadForServerWithListenerNameAndFallback PASSED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName STARTED

org.apache.kafka.common.security.JaasContextTest > testQuotedOptionName PASSED

org.apache.kafka.common.security.JaasContextTest > testControlFlag STARTED

org.apache.kafka.common.security.JaasContextTest > testControlFlag PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslScramMechanisms STARTED

[GitHub] kafka pull request #3730: MINOR: stateful docs for aggregates [WiP]

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/3730

MINOR: stateful docs for aggregates [WiP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka minor-docs-aggregates

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3730.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3730


commit 1dcb2ca6c70673d28a65917baf09bc4d88dce444
Author: Eno Thereska 
Date:   2017-08-24T17:02:48Z

Checkpoint




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Vahid S Hashemian]

Congrats Becket!

--Vahid



From:   Abhishek Mendhekar 
To: dev@kafka.apache.org
Date:   08/24/2017 09:53 AM
Subject:Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie 
(Becket) Qin



Congrats Becket!

On Thu, Aug 24, 2017 at 9:48 AM, Damian Guy  wrote:

> Congratulations.
>
> On Thu, 24 Aug 2017 at 17:44 Matthias J. Sax 
> wrote:
>
> > Congrats!
> >
> > On 8/23/17 10:40 PM, Hu Xi wrote:
> > > Congrats Becket!
> > >
> > >
> > > 
> > > 发件人: Guozhang Wang 
> > > 发送时间: 2017年8月24日 13:32
> > > 收件人: dev@kafka.apache.org
> > > 主题: Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin
> > >
> > > Congrats Jiangjie!
> > >
> > >
> > > Guozhang
> > >
> > > On Wed, Aug 23, 2017 at 10:20 PM, Joel Koshy 
> > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> Jiangjie (Becket) Qin has been a Kafka committer in October 2016 
and
> has
> > >> contributed significantly to several major patches, reviews and
> > discussions
> > >> since. I am glad to announce that Becket is now a member of the 
Apache
> > >> Kafka
> > >>  PMC.
> > >>
> > >> Congratulations Becket!
> > >>
> > >> Joel
> > >>
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
>



-- 
Abhishek Mendhekar
abhishek.mendhe...@gmail.com | 818.263.7030

[jira] [Resolved] (KAFKA-3885) Kafka new producer cannot failover

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3885.
--
Resolution: Duplicate

> Kafka new producer cannot failover
> --
>
> Key: KAFKA-3885
> URL: https://issues.apache.org/jira/browse/KAFKA-3885
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.2, 0.9.0.0, 0.9.0.1, 0.10.0.0
>Reporter: wateray
>
> This bug can reproduce by the following steps.
> The cluster has 2 brokers.
>  a) start a new producer, then send messages, it works well.
>  b) Then kill one broker,  it works well.
>  c) Then restart the broker,  it works well.
>  d) Then kill the other broker,  the producer can't failover.
> The the producer print log infinity.
> org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) 
> expired due to timeout while requesting metadata from brokers for 
> lwb_test_p50_r2-29
> 
> When producer sends msg, it detected that metadata should update.
> But at this code, class: NetworkClient ,method: leastLoadedNode
> List nodes = this.metadataUpdater.fetchNodes();
> nodes only return one result, and the returned node is the killed node, so 
> the producer cannot failover!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Damian Guy]

Congratulations.

On Thu, 24 Aug 2017 at 17:44 Matthias J. Sax  wrote:

> Congrats!
>
> On 8/23/17 10:40 PM, Hu Xi wrote:
> > Congrats Becket!
> >
> >
> > 
> > 发件人: Guozhang Wang 
> > 发送时间: 2017年8月24日 13:32
> > 收件人: dev@kafka.apache.org
> > 主题: Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin
> >
> > Congrats Jiangjie!
> >
> >
> > Guozhang
> >
> > On Wed, Aug 23, 2017 at 10:20 PM, Joel Koshy 
> wrote:
> >
> >> Hi everyone,
> >>
> >> Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has
> >> contributed significantly to several major patches, reviews and
> discussions
> >> since. I am glad to announce that Becket is now a member of the Apache
> >> Kafka
> >>  PMC.
> >>
> >> Congratulations Becket!
> >>
> >> Joel
> >>
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Bill Bejeck]

Thanks for the KIP!

+1

Thanks,
Bill

On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy  wrote:

> Hi,
>
> I'd like to kick off the voting thread for KIP-182:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> use+of+custom+storage+engines
>
> Thanks,
> Damian
>

Re: 答复: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

[Matthias J. Sax]

Congrats!

On 8/23/17 10:40 PM, Hu Xi wrote:
> Congrats Becket!
> 
> 
> 
> 发件人: Guozhang Wang 
> 发送时间: 2017年8月24日 13:32
> 收件人: dev@kafka.apache.org
> 主题: Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin
> 
> Congrats Jiangjie!
> 
> 
> Guozhang
> 
> On Wed, Aug 23, 2017 at 10:20 PM, Joel Koshy  wrote:
> 
>> Hi everyone,
>>
>> Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has
>> contributed significantly to several major patches, reviews and discussions
>> since. I am glad to announce that Becket is now a member of the Apache
>> Kafka
>>  PMC.
>>
>> Congratulations Becket!
>>
>> Joel
>>
> 
> 
> 
> --
> -- Guozhang
> 



signature.asc
Description: OpenPGP digital signature

[GitHub] kafka pull request #3722: KAFKA-5603: Don't abort TX for zombie tasks

[mjsax]

Github user mjsax closed the pull request at:

https://github.com/apache/kafka/pull/3722


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Damian Guy]

Hi,

I'd like to kick off the voting thread for KIP-182:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+use+of+custom+storage+engines

Thanks,
Damian

[GitHub] kafka pull request #3687: MINOR: add upgrade section for 1.0.0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3687


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-2414) Running kafka-producer-perf-test.sh with " --messages 10000000 --message-size 1000 --new-producer" will get WARN Error in I/O.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2414.
--
Resolution: Cannot Reproduce

 This might have been fixed in latest versions. Pl reopen if you think the 
issue still exists


> Running kafka-producer-perf-test.sh  with " --messages 1000 
> --message-size 1000  --new-producer" will get WARN Error in I/O.
> 
>
> Key: KAFKA-2414
> URL: https://issues.apache.org/jira/browse/KAFKA-2414
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.8.2.1
> Environment: Linux
>Reporter: Bo Wang
>  Labels: performance
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Running kafka-producer-perf-test.sh  with " --messages 1000 
> --message-size 1000  --new-producer" will get WARN Error in I/O：
> java.io.EOFException
>   at 
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
>   at org.apache.kafka.common.network.Selector.poll(Selector.java:248)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5781) Frequent long produce latency periods that result in reduced produce rate.

[Raoufeh Hashemian (JIRA)]

Raoufeh Hashemian created KAFKA-5781:


 Summary: Frequent long produce latency periods that result in 
reduced produce rate.
 Key: KAFKA-5781
 URL: https://issues.apache.org/jira/browse/KAFKA-5781
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.11.0.0
 Environment: CentOS Linux release 7.3.1611 , Kernel 3.10, java version 
"1.8.0_121"
Reporter: Raoufeh Hashemian
 Attachments: frequent_latency_increase_diskactivity.png, 
frequent_latency_increase.png, frequent_latency_increase_zoomed.png

When we upgraded from Kafka 0.10,2 to 0.11.0 , I started to see frequent 
throughput drops with a predictable pattern (attached file shows the pattern in 
a 14 hour period). This resulted in an overall degradation of up to 30% in our 
overall produce throughput.

The drops can be correlated to the significant increase in 99th percentile 
latency (up to 4 seconds). We have a cluster of 6 brokers and a single topic. 
The problem happens both with/without consumers running so I only included a 
case without consumers.

There is no specific message in the broker logs when the latency surge happens. 
 However, I found a correlation between the log rotation messages in the log 
and the the longer cycles in the pattern (details shown in the first attached 
graph)

Each increased latency period takes 5 to 20 minutes to finish (shown in the 
zoomed graph in the attached files). 

The broker cpu utilization goes down during this time and some read disk 
activity is observed (see attached graph)

This pattern started to appear in our environment exactly at the time when we 
switched to kafka 0.11.0. We kept the idempotence as false and didn`t make any 
configuration change as we switched. So I was wondering if it could be a bug or 
configuration that needs to be changed after upgrade?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2451) Exception logged but not managed

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2451.
--
Resolution: Cannot Reproduce

 Pl reopen if you think the issue still exists


> Exception logged but not managed
> 
>
> Key: KAFKA-2451
> URL: https://issues.apache.org/jira/browse/KAFKA-2451
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.1
> Environment: Windows + Java
>Reporter: Gwenhaël PASQUIERS
>Assignee: Jun Rao
>
> We've been having issues with java-snappy and it's native dll.
> To make it short : we have exceptions when serializing the message.
> We are using kafka producer it in Camel.
> The problem is that kafka thinks that the message was worrectly sent, and 
> returns no error: camel consumes the files even though kafka coult not send 
> the messages.
> Where the issue lies (if i'm correct):
> In DefaultEventHandler line 115 with tag 0.8.1 the exception that is thrown 
> by groupMessageToSet() is catched and logged. The return value of the 
> function dispatchSerializedData() is used to determine if the send was 
> successfull (if (outstandingProduceRequest.size >0) { ...}).
> BUT in this case I'm suspecting that, not even one message could be 
> serialized and added to  "failedProduceRequests". So the code that called 
> "dispatchSerializedData" thinks everything is OK though it's not.
> The producer could behave better and propagate the error properly. Since, it 
> could lead to pure data loss.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2577) one node of Kafka cluster cannot process produce request

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2577.
--
Resolution: Cannot Reproduce

This might have fixed in latest versions. Pl reopen if you think the issue 
still exists


> one node of Kafka cluster cannot process produce request
> 
>
> Key: KAFKA-2577
> URL: https://issues.apache.org/jira/browse/KAFKA-2577
> Project: Kafka
>  Issue Type: Bug
>  Components: producer , replication
>Affects Versions: 0.8.1.1
>Reporter: Ray
>Assignee: Jun Rao
>
> We had 3 nodes for kafka cluster, suddenly one node cannot accept produce r 
> request, here is the log:
> [2015-09-21 04:56:32,413] WARN [KafkaApi-0] Produce request with correlation 
> id 9178992 from client  on partition [topic_name,3] failed due to Leader not 
> local for partition [topic_name,3] on broker 0 (kafka.server.KafkaApis)
> after restarting that node, it still cannot work and I saw different log:
> [2015-09-21 20:38:16,791] WARN [KafkaApi-0] Produce request with correlation 
> id 9661337 from client  on partition [topic_name,3] failed due to Topic 
> topic_name either doesn't exist or is in the process of being deleted 
> (kafka.server.KafkaApis)
> it got fixed after rolling all the kafka nodes



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2560) Fatal error during KafkaServer startup because of Map failed error.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2560.
--
Resolution: Fixed

This is due to java.lang.OutOfMemoryError.  Pl reopen if you think the issue 
still exists


> Fatal error during KafkaServer startup because of Map failed error.
> ---
>
> Key: KAFKA-2560
> URL: https://issues.apache.org/jira/browse/KAFKA-2560
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2.1
> Environment: Linux 
>Reporter: Bo Wang
>Assignee: Jay Kreps
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> I have 3 kafka nodes,  
> create 30 topics ,every topic has 100 pations, and replica factor is 2.
> Kafka server start failed,
> 2015-09-21 10:28:35,668 | INFO  | pool-2-thread-1 | Recovering unflushed 
> segment 0 in log testTopic_14-34. | 
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,942 | ERROR | main | There was an error in one of the 
> threads during logs loading: java.io.IOException: Map failed | 
> kafka.utils.Logging$class.error(Logging.scala:97)
> 2015-09-21 10:28:35,943 | INFO  | pool-2-thread-5 | Recovering unflushed 
> segment 0 in log testTopic_17-23. | 
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,944 | INFO  | pool-2-thread-5 | Completed load of log 
> testTopic_17-23 with log end offset 0 | 
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,945 | FATAL | main | [Kafka Server 54], Fatal error 
> during KafkaServer startup. Prepare to shutdown | 
> kafka.utils.Logging$class.fatal(Logging.scala:116)
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286)
> at 
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
> at kafka.log.Log.loadSegments(Log.scala:179)
> at kafka.log.Log.(Log.scala:67)
> at 
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
> at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
> ... 13 more
> 2015-09-21 10:28:35,946 | INFO  | pool-2-thread-5 | Recovering unflushed 
> segment 0 in log testTopic_25-77. | 
> kafka.utils.Logging$class.info(Logging.scala:68)
> 2015-09-21 10:28:35,949 | INFO  | main | [Kafka Server 54], shutting down | 
> kafka.utils.Logging$class.info(Logging.scala:68)
> Kafka server host's top infomation below:
> top - 17:16:23 up 53 min,  6 users,  load average: 0.42, 0.99, 1.19
> Tasks: 215 total,   1 running, 214 sleeping,   0 stopped,   0 zombie
> Cpu(s):  4.5%us,  2.4%sy,  0.0%ni, 92.9%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem: 40169M total, 6118M used,34050M free,9M buffers
> Swap:0M total,0M used,0M free,  431M cached



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2565) Offset Commit is not working if multiple consumers try to commit the offset

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2565.
--
Resolution: Cannot Reproduce

may be related to deployment issue. Pl reopen if you think the issue still 
exists


> Offset Commit is not working if multiple consumers try to commit the offset
> ---
>
> Key: KAFKA-2565
> URL: https://issues.apache.org/jira/browse/KAFKA-2565
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1, 0.8.2.1, 0.8.2.2
>Reporter: Sreenivasulu Nallapati
>Assignee: Neha Narkhede
>
> We are seeing some strange behaviour with commitOffsets() method of 
> kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to 
> zookeeper at the end of the consumer batch. We are running multiple consumers 
> for the same topic.
> Test details: 
> 1.Created a topic with three partitions
> 2.Started three consumers (cronjob) at the same time. The aim is that 
> each consumer to process one partition.
> 3.Each consumer at the end of the batch, it will call the commitOffsets() 
> method on kafka.javaapi.consumer.ConsumerConnector
> 4.The offsets are getting properly updated in zookeeper if we run the 
> consumers for small set (say 1000 messages) of messages.
> 5.But for larger number of messages, commit offset is not working as 
> expected…sometimes only two offsets are properly committing and other one 
> remains as it was.
> 6.Please see the below example
> Partition: 0 Latest Offset: 1057585
> Partition: 1 Latest Offset: 1057715
> Partition: 2 Latest Offset: 1057590
> Earliest Offset after all consumers completed: {0=1057585, 1=724375, 
> 2=1057590}
> Highlighted in red supposed to be committed as 1057715 but it did not.
> Please check if it is bug with multiple consumers. When multiple consumers 
> are trying to update the same path in Zookeper, is there any synchronization 
> issue?
> Kafka Cluster details
> 1 zookeeper
> 3 brokers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2829) Inconsistent naming in {Producer,Consumer} Callback interfaces

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2829.
--
Resolution: Won't Fix

These are public interfaces heavily used by users. It's not appropriate to 
change now.  Please reopen if you think otherwise.


> Inconsistent naming in {Producer,Consumer} Callback interfaces
> --
>
> Key: KAFKA-2829
> URL: https://issues.apache.org/jira/browse/KAFKA-2829
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Mathias Söderberg
>Assignee: Neha Narkhede
>Priority: Minor
>
> The Callback interface for the "new" producer has a method called 
> "onCompletion" while the OffsetCommitCallback for the new consumer has a 
> method called "onComplete".
> Perhaps they should be using the same naming convention to avoid confusion?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-2966) 0.9.0 docs missing upgrade notes regarding replica lag

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2966.
--
Resolution: Fixed

> 0.9.0 docs missing upgrade notes regarding replica lag
> --
>
> Key: KAFKA-2966
> URL: https://issues.apache.org/jira/browse/KAFKA-2966
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Aditya Auradkar
>
> We should document that:
> * replica.lag.max.messages is gone
> * replica.lag.time.max.ms has a new meaning
> In the upgrade section. People can get caught by surprise.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-627) Make UnknownTopicOrPartitionException a WARN in broker

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-627.
-
Resolution: Fixed

Not observed on latest versions

> Make UnknownTopicOrPartitionException a WARN in broker
> --
>
> Key: KAFKA-627
> URL: https://issues.apache.org/jira/browse/KAFKA-627
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
> Environment: Kafka 0.8, RHEL6, Java 1.6
>Reporter: Chris Riccomini
>
> Currently, when sending messages to a topic that doesn't yet exist, the 
> broker spews out these "errors" as it tries to auto-create new topics. I 
> spoke with Neha, and she said that this should be a warning, not an error.
> Could you please change it to something less scary, if, in fact, it's not 
> scary.
> 2012/11/14 22:38:53.238 INFO [LogManager] [kafka-request-handler-6] [kafka] 
> []  [Log Manager on Broker 464] Created log for 'firehoseReads'-5
> 2012/11/14 22:38:53.241 WARN [HighwaterMarkCheckpoint] 
> [kafka-request-handler-6] [kafka] []  No previously checkpointed 
> highwatermark value found for topic firehoseReads partition 5. Returning 0 as 
> the highwatermark
> 2012/11/14 22:38:53.242 INFO [Log] [kafka-request-handler-6] [kafka] []  
> [Kafka Log on Broker 464], Truncated log segment 
> /export/content/kafka/i001_caches/firehoseReads-5/.log to 
> target offset 0
> 2012/11/14 22:38:53.242 INFO [ReplicaFetcherManager] 
> [kafka-request-handler-6] [kafka] []  [ReplicaFetcherManager on broker 464] 
> adding fetcher on topic firehoseReads, partion 5, initOffset 0 to broker 466 
> with fetcherId 0
> 2012/11/14 22:38:53.248 ERROR [ReplicaFetcherThread] 
> [ReplicaFetcherThread-466-0-on-broker-464] [kafka] []  
> [ReplicaFetcherThread-466-0-on-broker-464], error for firehoseReads 5 to 
> broker 466
> kafka.common.UnknownTopicOrPartitionException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:68)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$doWork$5$$anonfun$apply$3.apply(AbstractFetcherThread.scala:124)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$doWork$5$$anonfun$apply$3.apply(AbstractFetcherThread.scala:124)
> at kafka.utils.Logging$class.error(Logging.scala:102)
> at kafka.utils.ShutdownableThread.error(ShutdownableThread.scala:23)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:123)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$doWork$5.apply(AbstractFetcherThread.scala:99)
> at 
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:125)
> at 
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
> at 
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:344)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:99)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:50)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-3332) Consumer can't consume messages from zookeeper chroot

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3332.
--
Resolution: Cannot Reproduce

> Consumer can't consume messages from zookeeper chroot
> -
>
> Key: KAFKA-3332
> URL: https://issues.apache.org/jira/browse/KAFKA-3332
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.2.2, 0.9.0.1
> Environment: RHEL 6.X, OS X
>Reporter: Sergey Vergun
>Assignee: Neha Narkhede
>
> I have faced issue when consumer can't consume messages from zookeeper 
> chroot. It was tested on Kafka 0.8.2.2 and Kafka 0.9.0.1
> My zookeeper options into server.properties:
> $cat config/server.properties | grep zookeeper
> zookeeper.connect=localhost:2181/kafka-cluster/kafka-0.9.0.1
> zookeeper.session.timeout.ms=1
> zookeeper.connection.timeout.ms=1
> zookeeper.sync.time.ms=2000
> I can create successfully a new topic
> $kafka-topics.sh --create --partition 3 --replication-factor 1 --topic 
> __TEST-Topic_1 --zookeeper localhost:2181/kafka-cluster/kafka-0.9.0.1
> Created topic "__TEST-Topic_1".
> and produce messages into it
> $kafka-console-producer.sh --topic __TEST-Topic_1 --broker-list localhost:9092
> 1
> 2
> 3
> 4
> 5
> In Kafka Manager I see that messages was delivered:
> Sum of partition offsets  5
> But I can't consume the messages via kafka-console-consumer
> $kafka-console-consumer.sh --topic TEST-Topic_1 --zookeeper 
> localhost:2181/kafka-cluster/kafka-0.9.0.1 --from-beginning
> The consumer is present in zookeeper
> [zk: localhost:2181(CONNECTED) 10] ls /kafka-cluster/kafka-0.9.0.1/consumers
> [console-consumer-62895] 
> [zk: localhost:2181(CONNECTED) 12] ls 
> /kafka-cluster/kafka-0.9.0.1/consumers/console-consumer-62895/ids
> [console-consumer-62895_SV-Macbook-1457097451996-64640cc1] 
> If I reconfigure kafka cluster with zookeeper chroot "/" then everything is 
> ok.
> $cat config/server.properties | grep zookeeper
> zookeeper.connect=localhost:2181
> zookeeper.session.timeout.ms=1
> zookeeper.connection.timeout.ms=1
> zookeeper.sync.time.ms=2000
> $kafka-console-producer.sh --topic __TEST-Topic_1 --broker-list localhost:9092
> 1
> 2
> 3
> 4
> 5
> $kafka-console-consumer.sh --topic TEST-Topic_1 --zookeeper localhost:2181 
> --from-beginning
> 1
> 2
> 3
> 4
> 5
> Is it bug or my mistake?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3729: KAFKA-5749: Add MeteredSessionStore and Changelogg...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3729

KAFKA-5749: Add MeteredSessionStore and ChangeloggingSessionBytesStore.

Make MeteredSessionStore the outermost store.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-5749

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3729.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3729


commit 1dabd6a7ee8675d6e69df38a30c456e4e913339c
Author: Damian Guy 
Date:   2017-08-22T17:46:42Z

Add MeteredSessionStore and ChangeloggingSessionBytesStore.
Make MeteredSessionStore the outermost store




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-958) Please compile list of key metrics on the broker and client side and put it on a wiki

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-958.
-
Resolution: Fixed

Key metrics are listed on monitoring section of  Kafka documentation page

> Please compile list of key metrics on the broker and client side and put it 
> on a wiki
> -
>
> Key: KAFKA-958
> URL: https://issues.apache.org/jira/browse/KAFKA-958
> Project: Kafka
>  Issue Type: Bug
>  Components: website
>Affects Versions: 0.8.0
>Reporter: Vadim
>Assignee: Joel Koshy
>Priority: Minor
>
> Please compile list of important metrics that need to be monitored by 
> companies  to insure healthy operation of the kafka service



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk7 #2670

[Apache Jenkins Server]

See 


Changes:

[damian.guy] MINOR: Stateless transformation documentation

--
[...truncated 915.81 KB...]

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection STARTED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection PASSED

kafka.controller.ControllerIntegrationTest > testTopicCreation STARTED

kafka.controller.ControllerIntegrationTest > testTopicCreation PASSED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment STARTED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment PASSED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion STARTED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled PASSED

kafka.controller.ControllerIntegrationTest > testEmptyCluster STARTED

kafka.controller.ControllerIntegrationTest > testEmptyCluster PASSED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
STARTED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
PASSED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException 
STARTED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException PASSED

kafka.tools.ConsumerPerformanceTest > testHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testHeaderMatchBody PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetSmallest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetSmallest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured STARTED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest STARTED

Re: Confluence access rights

[Kreiner Attila]

Hi Guozhang,

Thanks!

Attila

2017-08-22 20:15 GMT+02:00 Guozhang Wang :

> Attila,
>
> I have added you to the wiki page list. Cheers.
>
>
> Guozhang
>
> On Tue, Aug 22, 2017 at 4:29 AM, Attila Kreiner  wrote:
>
> > Hey All,
> >
> > I need to create a KIP for this:
> > https://issues.apache.org/jira/browse/KAFKA-5726
> > https://github.com/apache/kafka/pull/3669
> >
> > However, I can't edit the page https://cwiki.apache.org/
> > confluence/display/KAFKA/Kafka+Improvement+Proposals and when I click
> > Create button I do not see the KIP template. So I can't create the page
> for
> > the KIP.
> >
> > Can someone provide me with proper access rights to Confluence?
> >
> > Thanks,
> > Attila
> >
>
>
>
> --
> -- Guozhang
>

Build failed in Jenkins: kafka-trunk-jdk8 #1938

[Apache Jenkins Server]

See 


Changes:

[damian.guy] MINOR: Stateless transformation documentation

--
[...truncated 922.42 KB...]

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
STARTED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics STARTED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithInvalidReplication 
STARTED

[jira] [Resolved] (KAFKA-962) Add list topics to ClientUtils

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-962.
-
Resolution: Fixed

Topic management methods are added to new admin client.

> Add list topics to ClientUtils
> --
>
> Key: KAFKA-962
> URL: https://issues.apache.org/jira/browse/KAFKA-962
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
>
> Currently there is no programmatic way to get a list of topics supported 
> directly by Kafka (one can talk to ZooKeeper directly).  There is a CLI tool 
> for this ListTopicCommand, but it'd be good to provide this directly to 
> clients as an API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-1339) Time based offset retrieval seems broken

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1339.
--
Resolution: Fixed

Time-based offset retrieval is improved with the introduction of message 
timestamp.  Pl reopen if you think the issue still exists


> Time based offset retrieval seems broken
> 
>
> Key: KAFKA-1339
> URL: https://issues.apache.org/jira/browse/KAFKA-1339
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
> Environment: Linux
>Reporter: Frank Varnavas
>Priority: Minor
>
> The kafka PartitionOffsetRequest takes a time parameter.  It seems broken to 
> me.
> There are two magic values
>   -2 returns the oldest  available offset
>   -1 returns the newest available offset
>   Otherwise the value is time since epoch in millisecs 
> (System.currentTimeMillis())
> The granularity is limited to the granularity of the log files
> These are the log segments for the partition I tested
>   Time now is about 17:07
>   Time shown is last modify time
>   File name has the starting offset number
>   You can see that the current one started about 13:40
> 1073742047 Mar 24 02:52 04740823.log
> 1073759588 Mar 24 11:25 04831581.log
> 1073782532 Mar 24 16:31 04916313.log
> 1073741985 Mar 25 09:11 05066939.log
> 1073743756 Mar 25 13:39 05158529.log
>  778424349 Mar 25 17:07 05214225.log
> The below shows the returned offset for an input time = (current time - 
> [0..23] hours)
> Even 1 second less than the current time returns the previous segment, even 
> though that segment ended 2.5 hours earlier.
> I think the result is off by 1 log segment. i.e. offset 1-3 should have been 
> from 5214225, 4-7 should have been from 5158529
> 0 -> 5214225
> 1 -> 5158529
> 2 -> 5158529
> 3 -> 5158529
> 4 -> 5066939
> 5 -> 5066939
> 6 -> 5066939
> 7 -> 5066939
> 8 -> 4973490
> 9 -> 4973490
> 10 -> 4973490
> 11 -> 4973490
> 12 -> 4973490
> 13 -> 4973490
> 14 -> 4973490
> 15 -> 4973490
> 16 -> 4916313
> 17 -> 4916313
> 18 -> 4916313
> 19 -> 4916313
> 20 -> 4916313
> 21 -> 4916313
> 22 -> 4916313
> 23 -> 4916313



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5780) Long shutdown time when updated to 0.11.0

[Raoufeh Hashemian (JIRA)]

Raoufeh Hashemian created KAFKA-5780:


 Summary: Long shutdown time when updated to 0.11.0
 Key: KAFKA-5780
 URL: https://issues.apache.org/jira/browse/KAFKA-5780
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.11.0.0
 Environment: CentOS Linux release 7.3.1611 , Kernel 3.10
Reporter: Raoufeh Hashemian
 Attachments: broker_shutdown.png

When we switched from Kafka 0.10.2 to Kafka 0.11.0 , We faced a problem with 
stopping the kafka service on a broker node.

Our cluster consists of 6 broker nodes. We had an existing topic when switched 
to Kafka 0.11.0 . Since then, gracefully stoping the service on a Kafka broker 
node results in the following warning message being repeated every 100 ms in 
the broker log, and the shutdown takes approximately 45 minutes to complete.

{code:java}
@4000599714da1e582e4c [2017-08-18 16:24:48,509] WARN Connection to node 
1002 could not be established. Broker may not be available. 
(org.apache.kafka.clients.NetworkClient)
@4000599714da245483a4 [2017-08-18 16:24:48,609] WARN Connection to node 
1002 could not be established. Broker may not be available. 
(org.apache.kafka.clients.NetworkClient)
@4000599714da2a51177c [2017-08-18 16:24:48,709] WARN Connection to node 
1002 could not be established. Broker may not be available. 
(org.apache.kafka.clients.NetworkClient)
{code}

Below is the last log lines when the shutdown is complete :

{code:java}
@400059971afd31113dbc [2017-08-18 16:50:59,823] WARN Connection to node 
1002 could not be established. Broker may not be available. 
(org.apache.kafka.clients.NetworkClient)
@400059971afd361200bc [2017-08-18 16:50:59,907] INFO Shutdown complete. 
(kafka.log.LogManager)
@400059971afd36afa04c [2017-08-18 16:50:59,917] INFO Terminate ZkClient 
event thread. (org.I0Itec.zkclient.ZkEventThread)
@400059971afd36dd6edc [2017-08-18 16:50:59,920] INFO Session: 
0x35d68c9e76702a4 closed (org.apache.zookeeper.ZooKeeper)
@400059971afd36deca84 [2017-08-18 16:50:59,920] INFO EventThread shut down 
for session: 0x35d68c9e76702a4 (org.apache.zookeeper.ClientCnxn)
@400059971afd36f6afb4 [2017-08-18 16:50:59,922] INFO [Kafka Server 1002], 
shut down completed (kafka.server.KafkaServer)
{code}

I should note that I stopped the producers before shutting down the broker.
If I repeat the process after brining up the service, the shutdown takes less 
than a minute. However, if I start the producers even for a short time and 
repeat the process, it will again take around 45 minutes to do a graceful 
shutdown.

Attached files shows the brokers CPU usage during the shutdown period (light 
blue curve is the node in which the broker service is shutting down).
The size of the topic is 2.3 TB per broker.

I was wondering if this is an expected new normal in Kafka 0.11.0 or It is a 
bug or a mis configuration?




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-1497) Change producer load-balancing algorithm in MirrorMaker

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1497.
--
Resolution: Fixed

MIrrorMaker now uses single producer instance.

> Change producer load-balancing algorithm in MirrorMaker
> ---
>
> Key: KAFKA-1497
> URL: https://issues.apache.org/jira/browse/KAFKA-1497
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Ivan Kunz
>
> Currently the MirrorMaker uses the following way of spreading the load into 
> configured producers :
> val producerId = 
> Utils.abs(java.util.Arrays.hashCode(msgAndMetadata.key)) % producers.size()
> This way if the producer side of MM uses different than the default 
> "partitioner.class" messages within the same partition can get re-ordered. 
> Also hashCode does not produce the same results on different machines 
> (verified by testing) so cannot be safely used for partitioning between 
> distributed systems connected via MM (for us message order preservation 
> within a partition is a critical feature).
> It would be great if the code above is changed to utilize the configured 
> "partitioner.class". 
> Something along the lines of  :
> At the initialization:
>   mmpartitioner = 
> Utils.createObject[Partitioner](config.partitionerClass, config.props)  
> During the processing:
> val producerId = 
> mmpartitioner.partition(msgAndMetadata.key,producers.size())
> This way the messages consumed and produced by MM can remain in the same 
> order.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3728: Add group-id to the metrics tags

[EtaCassiopeia]

GitHub user EtaCassiopeia opened a pull request:

https://github.com/apache/kafka/pull/3728

Add group-id to the metrics tags

It is better to have group-id in the JMX metrics. It would improve 
debuggability of systems:

```kafka_consumer_consumer_fetch_manager_metrics_test_0_records_lag{client_id="consumer-1",group_id="group-1",instance="service:8080",job="prometheus"}```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EtaCassiopeia/kafka 0.11.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3728.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3728


commit bd70fc8645b6f5f308633571ae7523cfa662c4e1
Author: Mohsen Zainalpour 
Date:   2017-08-24T13:59:47Z

Add group-id to the metrics tags




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3711: MINOR: Stateless transformation documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3711


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-0.11.0-jdk7 #283

[Apache Jenkins Server]

See 


Changes:

[damian.guy] KAFKA-5603; Don't abort TX for zombie tasks

--
[...truncated 862.56 KB...]
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:419)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:627)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:592)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:391)
at hudson.scm.SCM.poll(SCM.java:408)
at hudson.model.AbstractProject._poll(AbstractProject.java:1458)
at hudson.model.AbstractProject.poll(AbstractProject.java:1361)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:594)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:640)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.AdminClientWithPoliciesIntegrationTest > testInvalidAlterConfigs 
STARTED

kafka.api.AdminClientWithPoliciesIntegrationTest > testInvalidAlterConfigs 
PASSED

kafka.api.AdminClientWithPoliciesIntegrationTest > testValidAlterConfigs STARTED

kafka.api.AdminClientWithPoliciesIntegrationTest > testValidAlterConfigs PASSED

kafka.api.AdminClientWithPoliciesIntegrationTest > 
testInvalidAlterConfigsDueToPolicy STARTED

kafka.api.AdminClientWithPoliciesIntegrationTest > 
testInvalidAlterConfigsDueToPolicy PASSED

kafka.api.AdminClientIntegrationTest > testInvalidAlterConfigs STARTED

kafka.api.AdminClientIntegrationTest > testInvalidAlterConfigs PASSED

kafka.api.AdminClientIntegrationTest > testClose STARTED

kafka.api.AdminClientIntegrationTest > testClose PASSED

kafka.api.AdminClientIntegrationTest > testMinimumRequestTimeouts STARTED

kafka.api.AdminClientIntegrationTest > testMinimumRequestTimeouts PASSED

kafka.api.AdminClientIntegrationTest > testForceClose STARTED

kafka.api.AdminClientIntegrationTest > testForceClose PASSED

kafka.api.AdminClientIntegrationTest > testListNodes STARTED

kafka.api.AdminClientIntegrationTest > testListNodes PASSED

kafka.api.AdminClientIntegrationTest > testDelayedClose STARTED

kafka.api.AdminClientIntegrationTest > testDelayedClose PASSED

kafka.api.AdminClientIntegrationTest > testCreateDeleteTopics STARTED

kafka.api.AdminClientIntegrationTest > testCreateDeleteTopics PASSED

kafka.api.AdminClientIntegrationTest > testAclOperations STARTED

kafka.api.AdminClientIntegrationTest > testAclOperations PASSED

kafka.api.AdminClientIntegrationTest > testDescribeCluster STARTED

kafka.api.AdminClientIntegrationTest > testDescribeCluster PASSED

kafka.api.AdminClientIntegrationTest > testDescribeNonExistingTopic STARTED

kafka.api.AdminClientIntegrationTest > testDescribeNonExistingTopic PASSED

kafka.api.AdminClientIntegrationTest > testDescribeAndAlterConfigs STARTED

kafka.api.AdminClientIntegrationTest > testDescribeAndAlterConfigs PASSED

kafka.api.AdminClientIntegrationTest > testCallInFlightTimeouts STARTED

kafka.api.AdminClientIntegrationTest > testCallInFlightTimeouts PASSED

kafka.api.GroupCoordinatorIntegrationTest > 
testGroupCoordinatorPropagatesOfffsetsTopicCompressionCodec STARTED

kafka.api.GroupCoordinatorIntegrationTest > 
testGroupCoordinatorPropagatesOfffsetsTopicCompressionCodec PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testAcls STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testAcls PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

[DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

[Attila Kreiner]

Hi All,

I created KIP-191:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-191%3A+KafkaConsumer.subscribe%28%29+overload+that+takes+just+Pattern

Jira: https://issues.apache.org/jira/browse/KAFKA-5726
PR: https://github.com/apache/kafka/pull/3669

Please check it.

Thanks,
Attila

[GitHub] kafka pull request #3724: KAFKA-5769: Transient test failure org.apache.kafk...

[dguy]

Github user dguy closed the pull request at:

https://github.com/apache/kafka/pull/3724


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4425) Topic created with CreateTopic command, does not list partitions in metadata

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4425.
--
Resolution: Not A Problem

Fixed as per [~Fristi] comments

> Topic created with CreateTopic command, does not list partitions in metadata
> 
>
> Key: KAFKA-4425
> URL: https://issues.apache.org/jira/browse/KAFKA-4425
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Mark de Jong
>
> With the release of Kafka 0.10.10 there are now commands to delete and create 
> topics via the TCP protocol see (KIP 4 and KAFKA-2945)
> I've implemented this myself in my own driver. 
> When I send the following command to the current controller 
> "TopicDescriptor(topic = topic, nrPartitions = Some(3), replicationFactor = 
> Some(1), replicaAssignment = Seq.empty, config = Map())"
> I'll get back a response with NoError
> The server says this on the command line:
> [2016-11-20 17:54:19,599] INFO Topic creation 
> {"version":1,"partitions":{"2":[1003],"1":[1001],"0":[1002]}} 
> (kafka.admin.AdminUtils$)
> [2016-11-20 17:54:19,765] INFO [ReplicaFetcherManager on broker 1001] Removed 
> fetcher for partitions test212880282004727-1 
> (kafka.server.ReplicaFetcherManager)
> [2016-11-20 17:54:19,789] INFO Completed load of log test212880282004727-1 
> with 1 log segments and log end offset 0 in 17 ms (kafka.log.Log)
> [2016-11-20 17:54:19,791] INFO Created log for partition 
> [test212880282004727,1] in /kafka/kafka-logs-842dcf19f587 with properties 
> {compression.type -> producer, message.format.version -> 0.10.1-IV2, 
> file.delete.delay.ms -> 6, max.message.bytes -> 112, 
> min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, 
> min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, 
> min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, 
> unclean.leader.election.enable -> true, retention.bytes -> -1, 
> delete.retention.ms -> 8640, cleanup.policy -> [delete], flush.ms -> 
> 9223372036854775807, segment.ms -> 60480, segment.bytes -> 1073741824, 
> retention.ms -> 60480, message.timestamp.difference.max.ms -> 
> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 
> 9223372036854775807}. (kafka.log.LogManager)
> However when I immediately fetch metadata (v2 call). I either get;
> - The topic entry, but with no partitions data in it.
> - The topic entry, but with the status NotLeaderForPartition
> Is this an bug or I am missing something in my client?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka-site issue #71: KAFKA-4869: Update 0.10.2.0 upgrade notes

[omkreddy]

Github user omkreddy commented on the issue:

https://github.com/apache/kafka-site/pull/71
  
@ijuma minor doc cleanup


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site pull request #71: KAFKA-4869: Update 0.10.2.0 upgrade notes

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka-site/pull/71

KAFKA-4869: Update 0.10.2.0 upgrade notes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka-site cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/71.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #71


commit ea5b695df46ca9801875e6a023c0ab6f5a20729e
Author: Manikumar Reddy 
Date:   2017-08-24T11:46:31Z

KAFKA-4869: Update 0.10.2.0 upgrade notes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] 0.11.0.1 bug fix release

[Damian Guy]

A quick update. There are 2 remaining issues, that both have patches
available. Hopefully they will be merged soon and we can begin:
https://issues.apache.org/jira/projects/KAFKA/versions/12340632

Thanks,
Damian

On Tue, 22 Aug 2017 at 10:29 Damian Guy  wrote:

> An update on the 0.11.0.1 release. We currently have 5 outstanding issues:
> https://issues.apache.org/jira/projects/KAFKA/versions/12340632
> 3 with patches available that we can hopefully get merged pretty soon (1
> is actually already on 0.11.0.1)
> 2 issues that are Open, 1 is unassigned.
> Hopefully we can get this cleaned up in the next day or two and then i can
> go about building an RC.
>
> Thanks,
> Damian
>
> On Thu, 17 Aug 2017 at 17:45 Damian Guy  wrote:
>
>> Just a quick update.
>>
>> The list has reduced to 6 remaining issues:
>> https://issues.apache.org/jira/projects/KAFKA/versions/12340632
>>
>> Thanks to everyone for completing and/or moving tickets to future
>> releases.
>>
>> Damian
>>
>>
>> On Thu, 17 Aug 2017 at 09:50 Damian Guy  wrote:
>>
>>> Hi Srikanth,
>>> Optimistically i'm aiming for end of next week. Though it depends on how
>>> quickly the outstanding issues are closed and any other blockers that arise.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Thu, 17 Aug 2017 at 07:59 Srikanth Sampath 
>>> wrote:
>>>
 Thanks Damian.  What's the ballpark when 0.11.0.1 will be available?
 -Srikanth

 On Wed, Aug 16, 2017 at 5:59 PM, Damian Guy 
 wrote:

 > Hi,
 >
 > It seems like it must be time for 0.11.0.1 bug fix release!
 >
 > Since the 0.11.0.0 release we've fixed 30 JIRAs that
 > are targeted for 0.11.0.1:
 >
 > https://issues.apache.org/jira/browse/KAFKA-5700?jql=
 > project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Fixed%
 > 20AND%20fixVersion%20%3D%200.11.0.1%20ORDER%20BY%
 > 20priority%20DESC%2C%20key%20DESC
 >
 > We have 15 outstanding issues that are targeted at 0. <
 http://0.10.2.1/>
 > 11.0.1:
 >
 > https://issues.apache.org/jira/browse/KAFKA-5567?jql=
 > project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%
 > 20fixVersion%20%3D%200.11.0.1%20ORDER%20BY%20priority%
 > 20DESC%2C%20key%20DESC
 >
 > Can the owners of the remaining issues please resolve them or move
 them to
 > a future release.
 >
 > As soon as the remaining tasks for 0.11.0.1 reaches zero i'll create
 the
 > first RC.
 >
 > Thanks,
 > Damian
 >

>>>

[GitHub] kafka pull request #3727: KAFKA-5754 : Refactor Streams to use LogContext

[umesh9794]

GitHub user umesh9794 opened a pull request:

https://github.com/apache/kafka/pull/3727

KAFKA-5754 : Refactor Streams to use LogContext

This PR utilizes `org.apache.kafka.common.utils.LogContext` for logging in 
`KafkaStreams`. @hachikuji, @ijuma please review this and let me know your 
thoughts. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/umesh9794/kafka KAFKA-5754

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3727.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3727


commit 7560e473cd17c522c2e3b2aba6d83ffdadeeffab
Author: umesh chaudhary 
Date:   2017-08-24T08:53:03Z

KAFKA-5754 : Refactor Streams to use LogContext




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Damian Guy]

I've updated the kip to reflect Bill's comment and also to make
StreamBuilder methods have topic as the first param, i.e.,
StreamBuilder#stream no longer accepts varargs.

On Thu, 24 Aug 2017 at 09:12 Damian Guy  wrote:

> On Thu, 24 Aug 2017 at 02:49 Guozhang Wang  wrote:
>
>> I have a couple of comments but otherwise it LGTM:
>>
>> 1. For these two functions in StreamsBuilder, the topic String is set as
>> the second parameter in between of two options. Would that be better to be
>> set as the first or the last one instead?
>>
>> It would be better as the first, but then it is different to the
> #streams() methods due to varargs.
>
>
>> public synchronized  KTable table(final Consumed
>> consumed, final String topic, final Materialized materialized)
>>
>> public synchronized  GlobalKTable globalTable(final
>> Consumed> V> consumed, final String topic, final Materialized materialized)
>>
>> I understand that we cannot do it for the first parameter because of the
>> vararg type. So I'd suggest either
>>
>> a) set it as the last parameter, but then it is inconsistent with other
>> functions like these:
>>
>> void to(final String topic, final Produced options);
>>
>> KTable through(final String topic, final Materialized
>> options);
>>
>> b) only allow one single topic name parameter in StreamsBuilder.stream()
>> since in practice we do not see too many usages of multiple topics, plus
>> it
>> can be semi-supported with "merge" as we move it from StreamsBuilder to
>> KStream (KAFKA-5765),
>>
>> Perhaps this is the better approach
>
>
>> 2. KGroupedStream's function:
>>
>>  KTable aggregate(final Initializer initializer,
>>  final Aggregator
>> aggregator,
>>  final Serde aggValueSerde,
>>  final Materialized> VR>> materialized);
>>
>> The "aggValueSerde" seems not needed?
>>
>> 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think KGroupedStream
>> was
>> a bad name as a hind-sight. I personally feel we should just correct it
>> with a new class and deprecate / remove the old one before 1.0.0, but that
>> could be in its own KIP.
>>
>>
> The problem with this is that we'd need to add new `groupBy` and
> `groupByKey` methods that return `GroupedKStream`, we can't change the
> existing ones as that would break compatibility. So what would we name
> these methods?
>
>
>>
>> Guozhang
>>
>>
>>
>> On Wed, Aug 23, 2017 at 1:01 PM, Damian Guy  wrote:
>>
>> > We already have GlobalKTable and i can't rename KGroupedStream, which
>> > really should be GroupedKStream. So I think we should name new things
>> > correctly, i.e., WindowedKStream etc and fix the others when we can.
>> >
>> > On Wed, 23 Aug 2017 at 20:38 Matthias J. Sax 
>> > wrote:
>> >
>> > > About KGroupedStream vs GroupedKStream: shouldn't we keep the naming
>> > > convention consistent? And if we change the naming schema just change
>> > > all at once? I personally don't care which naming scheme is better,
>> but
>> > > I think consistency is super important!
>> > >
>> > > About Bill's comment: I agree, and had a similar thought.
>> > >
>> > >
>> > > -Matthias
>> > >
>> > > On 8/23/17 12:24 PM, Bill Bejeck wrote:
>> > > > Thanks for all the work on this KIP Damian.
>> > > >
>> > > > Both `Produced` and `Joined` have a `with` method accepting all
>> > > parameters,
>> > > > but `Consumed` doesn't. Should we add one for consistency?
>> > > >
>> > > > Thanks,
>> > > > Bill
>> > > >
>> > > > On Wed, Aug 23, 2017 at 4:15 AM, Damian Guy 
>> > > wrote:
>> > > >
>> > > >> KIP has been updated. thanks
>> > > >>
>> > > >> On Wed, 23 Aug 2017 at 09:10 Damian Guy 
>> wrote:
>> > > >>
>> > > >>> Hi Matthias,
>> > > >>>
>> > > >>>
>> > >  KStream:
>> > >  leftJoin and outerJoin for KStream/KTable join should not have
>> > >  `JoinWindows` parameter
>> > > 
>> > >  Thanks!
>> > > >>>
>> > > >>>
>> > > 
>> > >  Nit: TopologyBuilder -> Topology
>> > > 
>> > >  Ack
>> > > >>>
>> > > >>>
>> > >  Nit: new class Serialized list static method #with twice
>> > > 
>> > >  Ack
>> > > >>>
>> > > >>>
>> > >  WindowedKStream -> for consistency we should either have
>> > > GroupedKStream
>> > >  or KWindowedStream... (similar argument for
>> SessionWindowedKStream)
>> > > 
>> > >  We can't rename KGroupedStream -> GroupedKStream without breaking
>> > > >>> compatibility. So we are stuck with it for now. Hopefully in the
>> > future
>> > > >> we
>> > > >>> can rename KGroupedStream to GroupedKStream.
>> > > >>>
>> > > >>>
>> > > 
>> > >  KGroupedStream
>> > >  -> why do we use a different name for `sessionWindowedBy()` --
>> seems
>> > > to
>> > >  be cleaner to call both methods `windowedBy()`

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

[Damian Guy]

On Thu, 24 Aug 2017 at 02:49 Guozhang Wang  wrote:

> I have a couple of comments but otherwise it LGTM:
>
> 1. For these two functions in StreamsBuilder, the topic String is set as
> the second parameter in between of two options. Would that be better to be
> set as the first or the last one instead?
>
> It would be better as the first, but then it is different to the
#streams() methods due to varargs.


> public synchronized  KTable table(final Consumed
> consumed, final String topic, final Materialized materialized)
>
> public synchronized  GlobalKTable globalTable(final Consumed V> consumed, final String topic, final Materialized materialized)
>
> I understand that we cannot do it for the first parameter because of the
> vararg type. So I'd suggest either
>
> a) set it as the last parameter, but then it is inconsistent with other
> functions like these:
>
> void to(final String topic, final Produced options);
>
> KTable through(final String topic, final Materialized options);
>
> b) only allow one single topic name parameter in StreamsBuilder.stream()
> since in practice we do not see too many usages of multiple topics, plus it
> can be semi-supported with "merge" as we move it from StreamsBuilder to
> KStream (KAFKA-5765),
>
> Perhaps this is the better approach


> 2. KGroupedStream's function:
>
>  KTable aggregate(final Initializer initializer,
>  final Aggregator
> aggregator,
>  final Serde aggValueSerde,
>  final Materialized VR>> materialized);
>
> The "aggValueSerde" seems not needed?
>
> 3. +1 on `KGroupedStream` v.s. `GroupedKStream`. I think KGroupedStream was
> a bad name as a hind-sight. I personally feel we should just correct it
> with a new class and deprecate / remove the old one before 1.0.0, but that
> could be in its own KIP.
>
>
The problem with this is that we'd need to add new `groupBy` and
`groupByKey` methods that return `GroupedKStream`, we can't change the
existing ones as that would break compatibility. So what would we name
these methods?


>
> Guozhang
>
>
>
> On Wed, Aug 23, 2017 at 1:01 PM, Damian Guy  wrote:
>
> > We already have GlobalKTable and i can't rename KGroupedStream, which
> > really should be GroupedKStream. So I think we should name new things
> > correctly, i.e., WindowedKStream etc and fix the others when we can.
> >
> > On Wed, 23 Aug 2017 at 20:38 Matthias J. Sax 
> > wrote:
> >
> > > About KGroupedStream vs GroupedKStream: shouldn't we keep the naming
> > > convention consistent? And if we change the naming schema just change
> > > all at once? I personally don't care which naming scheme is better, but
> > > I think consistency is super important!
> > >
> > > About Bill's comment: I agree, and had a similar thought.
> > >
> > >
> > > -Matthias
> > >
> > > On 8/23/17 12:24 PM, Bill Bejeck wrote:
> > > > Thanks for all the work on this KIP Damian.
> > > >
> > > > Both `Produced` and `Joined` have a `with` method accepting all
> > > parameters,
> > > > but `Consumed` doesn't. Should we add one for consistency?
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > > > On Wed, Aug 23, 2017 at 4:15 AM, Damian Guy 
> > > wrote:
> > > >
> > > >> KIP has been updated. thanks
> > > >>
> > > >> On Wed, 23 Aug 2017 at 09:10 Damian Guy 
> wrote:
> > > >>
> > > >>> Hi Matthias,
> > > >>>
> > > >>>
> > >  KStream:
> > >  leftJoin and outerJoin for KStream/KTable join should not have
> > >  `JoinWindows` parameter
> > > 
> > >  Thanks!
> > > >>>
> > > >>>
> > > 
> > >  Nit: TopologyBuilder -> Topology
> > > 
> > >  Ack
> > > >>>
> > > >>>
> > >  Nit: new class Serialized list static method #with twice
> > > 
> > >  Ack
> > > >>>
> > > >>>
> > >  WindowedKStream -> for consistency we should either have
> > > GroupedKStream
> > >  or KWindowedStream... (similar argument for
> SessionWindowedKStream)
> > > 
> > >  We can't rename KGroupedStream -> GroupedKStream without breaking
> > > >>> compatibility. So we are stuck with it for now. Hopefully in the
> > future
> > > >> we
> > > >>> can rename KGroupedStream to GroupedKStream.
> > > >>>
> > > >>>
> > > 
> > >  KGroupedStream
> > >  -> why do we use a different name for `sessionWindowedBy()` --
> seems
> > > to
> > >  be cleaner to call both methods `windowedBy()`
> > > 
> > > 
> > > >>> I beg to differ that it is cleaner either way!
> > > >>>
> > > >>>
> > > 
> > >  StreamsBuilder#stream -> parameter order is confusing... We should
> > > have
> > >  Pattern as second parameter to align both methods.
> > > 
> > >  Ack
> > > >>>
> > > >>>
> > >  StreamsBuilder#table/globalTable -> move parameter `Consumed` as
> > first
> > >  

[jira] [Created] (KAFKA-5779) Single message may exploit application based on KStream

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-5779:
-

 Summary: Single message may exploit application based on KStream
 Key: KAFKA-5779
 URL: https://issues.apache.org/jira/browse/KAFKA-5779
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0, 0.10.2.1
Reporter: Seweryn Habdank-Wojewodzki
Priority: Critical


The context: in Kafka streamming I am *defining* simple KStream processing:

{code}
stringInput // line 54 of the SingleTopicStreamer class
.filter( streamFilter::passOrFilterMessages )
.map( normalizer )
.to( outTopicName );
{code}

For some reasons I got wrong message (I am still investigating what is the 
problem), 
but anyhow my services was exploited with FATAL error:

{code}
2017-08-22 17:08:44 FATAL SingleTopicStreamer:54 - Caught unhandled exception: 
Input record ConsumerRecord(topic = XXX_topic, partition = 8, offset = 15, 
CreateTime = -1, serialized key size = -1, serialized value size = 255, headers 
= RecordHeaders(headers = [], isReadOnly = false), key = null, value = 
{"recordTimestamp":"2017-08-22T17:07:40:619+02:00","logLevel":"INFO","sourceApplication":"WPT","message":"Kafka-Init","businessError":false,"normalizedStatus":"green","logger":"CoreLogger"})
 has invalid (negative) timestamp. Possibly because a pre-0.10 producer client 
was used to write this record to Kafka without embedding a timestamp, or 
because the input topic was created before upgrading the Kafka cluster to 
0.10+. Use a different TimestampExtractor to process this data.; 
[org.apache.kafka.streams.processor.FailOnInvalidTimestamp.onInvalidTimestamp(FailOnInvalidTimestamp.java:63),
 
org.apache.kafka.streams.processor.ExtractRecordMetadataTimestamp.extract(ExtractRecordMetadataTimestamp.java:61),
 
org.apache.kafka.streams.processor.FailOnInvalidTimestamp.extract(FailOnInvalidTimestamp.java:46),
 
org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:85),
 
org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:117),
 
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:464),
 
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:650),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:556),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)]
 in thread restreamer-d4e77d18-6e7b-4708-8436-7fea0d4b1cdf-StreamThread-3
{code}

The possible reason about using old producer in message is false, as we are 
using Kafka 0.10.2.1 and 0.11.0.0 and the topics had been created within this 
version of Kafka. 
The sender application is .NET client from Confluent.

All the matter is a bit problematic with this exception, as it was suggested it 
is thrown in scope of initialization of the stream, but effectively it happend 
in processing, so adding try{} catch {} around stringInput statement does not 
help, as stream was correctly defined, but only one message send later had 
exploited all the app.

In my opinion KStream shall be robust enough to catch all such a exception and 
shall protect application from death due to single corrupted message. 
Especially when timestamp is not embedded. In such a case one can patch message 
with current timestamp without loss of overall performance.

I would expect Kafka Stream will handle this.

I will continue to investigate, what is the problem with the message, but it is 
quite hard to me, as it happens internally in Kafka stream combined with .NET 
producer.

And I had already tested, that this problem does not occur when I got these 
concrete messages in old-fashioned Kafka Consumer :-).




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5778) Kafka cluster is not responding when one broker hangs and resulted in too many connections in close_wait in other brokers

[saichand (JIRA)]

saichand created KAFKA-5778:
---

 Summary: Kafka cluster is not responding when one broker hangs and 
resulted in too many connections in close_wait in other brokers
 Key: KAFKA-5778
 URL: https://issues.apache.org/jira/browse/KAFKA-5778
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.0.1
Reporter: saichand
Priority: Blocker


In a cluster of 3 brokers , one of the broker(Broker-1 ) is hanged and from 
then other two brokers has connections in close_wait for java client 
producer/consumer and also even some broker to broker connections are in close 
wait among those two brokers.
Kafka Version : 0.10.0.1
In logs I found replica fetcher thread connection refused exceptions:
In broker 0 : replica fetcher 0-1, replica fetcher 0-2
In broker 2 : replica fetcher 0-0, replica fetcher 0-1
In broker 1 : It was hung no logs were available at that time.

We tried restarting broker- 2 kafka and then it was not successful as it 
terminated saying zookeeper timeout 
then we tried restarting broker- 0 kafka and we got the same error
Broker -1 was hang so , we could not login even into it
so we restarted broker -1 machine
then we restarted all zookepers and then kafka brokers now everything is fine 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)