Build failed in Jenkins: kafka-trunk-jdk8 #2224

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: Log unexpected exceptions in Connect REST calls that generate

--
[...truncated 3.32 MB...]

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled STARTED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled PASSED

kafka.security.auth.ZkAuthorizationTest > testZkUtils STARTED

kafka.security.auth.ZkAuthorizationTest > testZkUtils PASSED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

[Hu Xi]

Hi Jun,


Seems the prefix that is used to be the unique Sensor name cannot be removed, 
so should we keep the prefix?



发件人: Jun Rao 
发送时间: 2017年11月21日 3:55
收件人: dev@kafka.apache.org
主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition 
lead metrics to KafkaConsumer

Hi, Hu,

For the new partition level metrics that you are adding, it seems it's
better to just add the topic/partition tag instead of using them in the
prefix. For the existing lag metrics, we can fix them in KIP-225.

Thanks,

Jun

On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi  wrote:

> Jun,
>
>
> Thanks for the comments. Do you think it'd better to add topic/partition
> tags for those metrics as well as keep the prefix? If those prefixes should
> really be removed, does this KIP need to do the same thing for `lag` ones?
>
> 
> 发件人: Jun Rao 
> 发送时间: 2017年11月18日 8:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Charly,
>
> Thanks for the input. It makes sense.
>
> Hi, Hu,
>
> Perhaps we can keep the per partition records-lead-min and records-lead-avg
> as you had before, but just add the topic and the partition as the tags
> instead of prefix of the metric name.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 15, 2017 at 4:58 AM, charly molter 
> wrote:
>
> > Hi Jun, Jiangle,
> >
> > I'd just like to clarify that KIP-225 seems to be using per partition
> > metric the same way as KIP-223 seems to be doing.
> >
> > I believe avg and max are still necessary because the MetricsReporter
> > doesn't work in a "push" manner and the "Value" measurableStat will only
> > keep the last recorded entry.
> > Therefore a MetricsReporter usually polls to grab a current view with
> Value
> > this view is incomplete so it becomes not possible to compute the
> > Max/Min/Avg.
> > Max/Min/Avg uses SampledStats which work with a rolling window of samples
> > and therefore periodic polling would work.
> >
> > This is why I believe it's necessary to keep Avg, Min and Max for these
> > metrics as otherwise we wouldn't be able to recompute it in an external
> > monitoring system.
> >
> > Am I wrong thinking this?
> >
> > Thanks,
> > Charly
> >
> >
> > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao  wrote:
> >
> > > Hi, Charly,
> > >
> > > Thanks for KIP-225. Your proposal looks reasonable.
> > >
> > > Hi, Jiangjie,
> > >
> > > Do you think the approach that KIP-225 proposes is better for exposing
> > the
> > > per partition metric? Also, do we really need the per partition
> > > record-lag-avg
> > > and record-lag-max? It seems that an external monitoring system can
> > always
> > > derive that from the per partition record-lag.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> charly.mol...@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Hu,
> > > >
> > > > I have KIP-225 open for adding tags to records-lag:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74686649
> > > >
> > > > I have a patch more or less ready so I could probably get the fix
> > checked
> > > > in (after the vote) and you could build on top of it. Otherwise we
> > could
> > > > merge both KIPs if you want but they do sound different to me.
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi  wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > >
> > > > > Let me double confirm with your comments:
> > > > >
> > > > > 1 remove partition-level records-lead-avg and records-lead-min
> since
> > > they
> > > > > both can be deduced by external monitoring system.
> > > > >
> > > > > 2 Tag partition-level records-lead with topic info
> > > > >
> > > > >
> > > > > If they are the case you expect, do we need to do the same thing
> for
> > > > those
> > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > tagged
> > > > > with topic information  which might deserve a bug.
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > > 
> > > > > 发件人: Jun Rao 
> > > > > 发送时间: 2017年11月14日 12:44
> > > > > 收件人: dev@kafka.apache.org
> > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}".
> > > > > So, it probably makes sense for records-lead-min to be an attribute
> > > under
> > > > > the same mbean.
> > > > >
> > > > > The partition level records-lead can probably be an attribute for
> the
> > > > mbean
> > > > > 

Jenkins build is back to normal : kafka-trunk-jdk7 #2985

[Apache Jenkins Server]

See 

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Jan Filipiak]

a remark of mine that got missed during migration:

There is this problem that even though we have source.table.filter.join
the state-fullness happens at the table step not a the join step. In a 
filter

we still going to present change.oldValue to the filter even though the
record context() is for change.newValue. I would go as far as applying
the filter before the table processor. Not to just get KIP-159, but because
I think its a side effect of a non ideal topology layout. If i can 
filter 99% of my

records. my state could be way smaller. Also widely escalates the context
of the KIP

I can only see upsides of executing the filter first.

Best Jan



On 20.11.2017 22:22, Matthias J. Sax wrote:

I am moving this back to the DISCUSS thread... Last 10 emails were sent
to VOTE thread.

Copying Guozhang's last summary below. Thanks for this summary. Very
comprehensive!

It seems, we all agree, that the current implementation of the context
at PAPI level is ok, but we should not leak it into DSL.

Thus, we can go with (2) or (3), were (3) is an extension to (2)
carrying the context to more operators than just sources. It also seems,
that we all agree, that many-to-one operations void the context.

I still think, that just going with plain (2) is too restrictive -- but
I am also fine if we don't go with the full proposal of (3).

Also note, that the two operators filter() and filterNot() don't modify
the record and thus for both, it would be absolutely valid to keep the
context.

I personally would keep the context for at least all one-to-one
operators. One-to-many is debatable and I am fine to not carry the
context further: at least the offset information is questionable for
this case -- note thought, that semantically, the timestamp is inherited
via one-to-many, and I also think this applies to "topic" and
"partition". Thus, I think it's still valuable information we can carry
downstreams.


-Matthias


Jan: which approach are you referring to as "the approach that is on the
table would be perfect"?

Note that in today's PAPI layer we are already effectively exposing the
record context which has the issues that we have been discussing right now,
and its semantics is always referring to the "processing record" at hand.
More specifically, we can think of processing a record a bit different:

1) the record traversed the topology from source to sink, it may be
transformed into new object or even generate multiple new objects (think:
branch) along the traversal. And the record context is referring to this
processing record. Here the "lifetime" of the record lasts for the entire
topology traversal and any new records of this traversal is treated as
different transformed values of this record (this applies to join and
aggregations as well).

2) the record being processed is wiped out in the first operator after the
source, and NEW records are forwarded to downstream operators. I.e. each
record only lives between two adjacent operators, once it reached the new
operator it's lifetime has ended and new records are generated.

I think in the past we have talked about Streams under both context, and we
do not have a clear agreement. I agree that 2) is logically more
understandable for users as it does not leak any internal implementation
details (e.g. for stream-table joins, table record's traversal ends at the
join operator as it is only be materialized, while stream record's
traversal goes through the join operator to further down until sinks).
However if we are going to interpret following 2) above then even for
non-stateful operators we would not inherit record context. What we're
discussing now, seems to infer a third semantics:

3) a record would traverse "through" one-to-one (non-stateful) operators,
will "replicate" at one-to-many (non-stateful) operators (think: "mapValues"
  ) and will "end" at many-to-one (stateful) operators where NEW records
will be generated and forwarded to the downstream operators.

Just wanted to lay the ground for discussions so we are all on the same
page before chatting more.


Guozhang



On 11/6/17 1:41 PM, Jeyhun Karimov wrote:

Hi Matthias,

Thanks a lot for correcting. It is a leftover from the past designs when
punctuate() was not deprecated.
I corrected.

Cheers,
Jeyhun

On Mon, Nov 6, 2017 at 5:30 PM Matthias J. Sax 
wrote:


I just re-read the KIP.

One minor comment: we don't need to introduce any deprecated methods.
Thus, RichValueTransformer#punctuate can be removed completely instead
of introducing it as deprecated.

Otherwise looks good to me.

Thanks for being so patient!


-Matthias

On 11/1/17 9:16 PM, Guozhang Wang wrote:

Jeyhun,

I think I'm convinced to not do KAFKA-3907 in this KIP. We should think
carefully if we should add this functionality to the DSL layer moving
forward since from what we discovered working on it the conclusion is

that

it would require revamping the public APIs quite a lot, and it's not

clear

if it is a good 

[GitHub] kafka pull request #4227: MINOR: Log unexpected exceptions in Connect REST c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4227


---

Build failed in Jenkins: kafka-0.11.0-jdk7 #338

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: Add HttpMetricsReporter for system tests

--
[...truncated 904.84 KB...]

kafka.producer.AsyncProducerTest > testSerializeEvents STARTED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize STARTED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner STARTED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration STARTED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition STARTED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker STARTED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed STARTED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer STARTED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder STARTED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.api.LogAppendTimeTest > testProduceConsume STARTED

kafka.api.LogAppendTimeTest > testProduceConsume PASSED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic STARTED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic PASSED

kafka.api.FetchRequestTest > testShuffle STARTED

kafka.api.FetchRequestTest > testShuffle PASSED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic STARTED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic PASSED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero STARTED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero PASSED

kafka.api.PlaintextProducerSendTest > testWrongSerializer STARTED

kafka.api.PlaintextProducerSendTest > testWrongSerializer PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testClose STARTED

kafka.api.PlaintextProducerSendTest > testClose PASSED

kafka.api.PlaintextProducerSendTest > testFlush STARTED

kafka.api.PlaintextProducerSendTest > testFlush PASSED

kafka.api.PlaintextProducerSendTest > testSendToPartition STARTED

kafka.api.PlaintextProducerSendTest > testSendToPartition PASSED

kafka.api.PlaintextProducerSendTest > testSendOffset STARTED

kafka.api.PlaintextProducerSendTest > testSendOffset PASSED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
STARTED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
PASSED

kafka.api.PlaintextProducerSendTest > testSendBeforeAndAfterPartitionExpansion 
STARTED

kafka.api.PlaintextProducerSendTest > testSendBeforeAndAfterPartitionExpansion 
PASSED

kafka.api.PlaintextConsumerTest > testEarliestOrLatestOffsets STARTED

kafka.api.PlaintextConsumerTest > testEarliestOrLatestOffsets PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate PASSED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMs STARTED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMs PASSED

kafka.api.PlaintextConsumerTest > testOffsetsForTimes STARTED

kafka.api.PlaintextConsumerTest > testOffsetsForTimes PASSED

kafka.api.PlaintextConsumerTest > testSubsequentPatternSubscription STARTED

kafka.api.PlaintextConsumerTest > testSubsequentPatternSubscription PASSED

kafka.api.PlaintextConsumerTest > testConsumeMessagesWithCreateTime STARTED

kafka.api.PlaintextConsumerTest > testConsumeMessagesWithCreateTime 

[GitHub] kafka pull request #4242: KAFKA-4857: [WIP] Replace StreamsKafkaClient with ...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/4242

KAFKA-4857: [WIP] Replace StreamsKafkaClient with AdminClient in Kafka 
Streams

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-4857-admit-client

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4242


commit aad938f9cb3b5dce2824be9c9b3aff00068a6bbf
Author: Matthias J. Sax 
Date:   2017-11-21T02:17:57Z

KAFKA-4857: Replace StreamsKafkaClient with AdminClient in Kafka Streams




---

[VOTE] KIP-219 - Improve Quota Communication

[Becket Qin]

Hi,

We would like to start the voting thread for KIP-219. The KIP proposes to
improve the quota communication between the brokers and clients, especially
for cases of long throttling time.

The KIP wiki is following:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-219+-+Improve+quota+
communication

The discussion thread is here:
http://markmail.org/search/?q=kafka+KIP-219#query:kafka%20KIP-219+page:1+mid:ooxabguy7nz7l7zy+state:results

Thanks,

Jiangjie (Becket) Qin

Please give me jira access rights

[Panuwat Anawatmongkhon]

Hello

I  want to try to contribute to the project.
Please give me rights to work with issues in JIRA

My Jira name is Panuwat Anawatmongkhon

Thank you

[GitHub] kafka pull request #4241: KAFKA-4893: Fix conflict between async topic delet...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/4241

KAFKA-4893: Fix conflict between async topic deletion and max topic length

With async topic deletion the topic partition folder name is affixed with a 
'.', a UUID, and '-delete'. If topic name length is close to its current limit 
(249) this could cause an issue because the folder name size goes over 255.
This PR implements the [suggestion 
solution](https://issues.apache.org/jira/browse/KAFKA-4893?focusedCommentId=15927155)
 by @onurkaraman in the JIRA.
This implementation is automatically backward compatible, and cleans up any 
folder marked for deletion using the old method (affixing the folder name).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4893

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4241


commit b6f7f2cad7f8484aa35f22988ab7e4685607966f
Author: Vahid Hashemian 
Date:   2017-11-21T00:28:44Z

KAFKA-4893: Fix conflict between async topic deletion and max topic lenth

With async topic deletion the topic partition folder name is affixed with a 
'.', a UUID, and '-delete'. If topic name length is close to its current limit 
(249) this could cause an issue because the folder name size goes over 255.
This PR implements the [suggestion 
solution](https://issues.apache.org/jira/browse/KAFKA-4893?focusedCommentId=15927155)
 by @onurkaraman in the JIRA.
This implementation is automatically backward compatible, and cleans up any 
folder marked for deletion using the old method (affixing the folder name).




---

[GitHub] kafka pull request #4207: MINOR: Add HttpMetricsReporter for system tests

[ewencp]

Github user ewencp closed the pull request at:

https://github.com/apache/kafka/pull/4207


---

Build failed in Jenkins: kafka-trunk-jdk8 #2223

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6170; KIP-220 Part 1: Add AdminClient to Streams

--
[...truncated 3.32 MB...]

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled STARTED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled PASSED

kafka.security.auth.ZkAuthorizationTest > testZkUtils STARTED

kafka.security.auth.ZkAuthorizationTest > testZkUtils PASSED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Matthias J. Sax]

Just list what each thing is:

K0: key type of first/this table
K1: key type of second/other table
KO: key type of result table (concatenation of both input keys )


something like this (not sure it the example above is correct---it's
just for illustration)


-Matthias


On 11/18/17 2:30 PM, Jan Filipiak wrote:
> -> it think the relationships between the different used types, K0,K1,KO
> should be explains explicitly (all information is there implicitly, but
> one need to think hard to figure it out)
> 
> 
> I'm probably blind for this. can you help me here? how would you
> formulate this?
> 
> Thanks,
> 
> Jan
> 
> 
> On 16.11.2017 23:18, Matthias J. Sax wrote:
>> Hi,
>>
>> I am just catching up on this discussion and did re-read the KIP and
>> discussion thread.
>>
>> In contrast to you, I prefer the second approach with CombinedKey as
>> return type for the following reasons:
>>
>>   1) the oneToManyJoin() method had less parameter
>>   2) those parameters are easy to understand
>>   3) we hide implementation details (joinPrefixFaker, leftKeyExtractor,
>> and the return type KO leaks internal implementation details from my
>> point of view)
>>   4) user can get their own KO type by extending CombinedKey interface
>> (this would also address the nesting issue Trevor pointed out)
>>
>> That's unclear to me is, why you care about JSON serdes? What is the
>> problem with regard to prefix? It seems I am missing something here.
>>
>> I also don't understand the argument about "the user can stick with his
>> default serde or his standard way of serializing"? If we have
>> `CombinedKey` as output, the use just provide the serdes for both input
>> combined-key types individually, and we can reuse both internally to do
>> the rest. This seems to be a way simpler API. With the KO output type
>> approach, users need to write an entirely new serde for KO in contrast.
>>
>> Finally, @Jan, there are still some open comments you did not address
>> and the KIP wiki page needs some updates. Would be great if you could do
>> this.
>>
>> Can you also explicitly describe the data layout of the store that is
>> used to do the range scans?
>>
>> Additionally:
>>
>> -> some arrows in the algorithm diagram are missing
>> -> was are those XXX in the diagram
>> -> can you finish the "Step by Step" example
>> -> it think the relationships between the different used types, K0,K1,KO
>> should be explains explicitly (all information is there implicitly, but
>> one need to think hard to figure it out)
>>
>>
>> Last but not least:
>>
>>> But noone is really interested.
>> Don't understand this statement...
>>
>>
>>
>> -Matthias
>>
>>
>> On 11/16/17 9:05 AM, Jan Filipiak wrote:
>>> We are running this perfectly fine. for us the smaller table changes
>>> rather infrequent say. only a few times per day. The performance of the
>>> flush is way lower than the computing power you need to bring to the
>>> table to account for all the records beeing emmited after the one single
>>> update.
>>>
>>> On 16.11.2017 18:02, Trevor Huey wrote:
 Ah, I think I see the problem now. Thanks for the explanation. That is
 tricky. As you said, it seems the easiest solution would just be to
 flush the cache. I wonder how big of a performance hit that'd be...

 On Thu, Nov 16, 2017 at 9:07 AM Jan Filipiak > wrote:

  Hi Trevor,

  I am leaning towards the less intrusive approach myself. Infact
  that is how we implemented our Internal API for this and how we
  run it in production.
  getting more voices towards this solution makes me really happy.
  The reason its a problem for Prefix and not for Range is the
  following. Imagine the intrusive approach. They key of the RockDB
  would be CombinedKey and the prefix scan would take an A, and
  the range scan would take an CombinedKey still. As you can
  see with the intrusive approach the keys are actually different
  types for different queries. With the less intrusive apporach we
  use the same type and rely on Serde Invariances. For us this works
  nice (protobuf) might bite some JSON users.

  Hope it makes it clear

  Best Jan


  On 16.11.2017 16:39, Trevor Huey wrote:
>  1. Going over KIP-213, I am leaning toward the "less intrusive"
>  approach. In my use case, I am planning on performing a sequence
>  of several oneToMany joins, From my understanding, the more
>  intrusive approach would result in several nested levels of
>  CombinedKey's. For example, consider Tables A, B, C, D with
>  corresponding keys KA, KB, KC. Joining A and B would produce
>  CombinedKey. Then joining that result on C would produce
>  CombinedKey>. My "keyOtherSerde" in this
>  case 

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Matthias J. Sax]

I am moving this back to the DISCUSS thread... Last 10 emails were sent
to VOTE thread.

Copying Guozhang's last summary below. Thanks for this summary. Very
comprehensive!

It seems, we all agree, that the current implementation of the context
at PAPI level is ok, but we should not leak it into DSL.

Thus, we can go with (2) or (3), were (3) is an extension to (2)
carrying the context to more operators than just sources. It also seems,
that we all agree, that many-to-one operations void the context.

I still think, that just going with plain (2) is too restrictive -- but
I am also fine if we don't go with the full proposal of (3).

Also note, that the two operators filter() and filterNot() don't modify
the record and thus for both, it would be absolutely valid to keep the
context.

I personally would keep the context for at least all one-to-one
operators. One-to-many is debatable and I am fine to not carry the
context further: at least the offset information is questionable for
this case -- note thought, that semantically, the timestamp is inherited
via one-to-many, and I also think this applies to "topic" and
"partition". Thus, I think it's still valuable information we can carry
downstreams.


-Matthias

> Jan: which approach are you referring to as "the approach that is on the
> table would be perfect"?
> 
> Note that in today's PAPI layer we are already effectively exposing the
> record context which has the issues that we have been discussing right now,
> and its semantics is always referring to the "processing record" at hand.
> More specifically, we can think of processing a record a bit different:
> 
> 1) the record traversed the topology from source to sink, it may be
> transformed into new object or even generate multiple new objects (think:
> branch) along the traversal. And the record context is referring to this
> processing record. Here the "lifetime" of the record lasts for the entire
> topology traversal and any new records of this traversal is treated as
> different transformed values of this record (this applies to join and
> aggregations as well).
> 
> 2) the record being processed is wiped out in the first operator after the
> source, and NEW records are forwarded to downstream operators. I.e. each
> record only lives between two adjacent operators, once it reached the new
> operator it's lifetime has ended and new records are generated.
> 
> I think in the past we have talked about Streams under both context, and we
> do not have a clear agreement. I agree that 2) is logically more
> understandable for users as it does not leak any internal implementation
> details (e.g. for stream-table joins, table record's traversal ends at the
> join operator as it is only be materialized, while stream record's
> traversal goes through the join operator to further down until sinks).
> However if we are going to interpret following 2) above then even for
> non-stateful operators we would not inherit record context. What we're
> discussing now, seems to infer a third semantics:
> 
> 3) a record would traverse "through" one-to-one (non-stateful) operators,
> will "replicate" at one-to-many (non-stateful) operators (think: "mapValues"
>  ) and will "end" at many-to-one (stateful) operators where NEW records
> will be generated and forwarded to the downstream operators.
> 
> Just wanted to lay the ground for discussions so we are all on the same
> page before chatting more.
> 
> 
> Guozhang



On 11/6/17 1:41 PM, Jeyhun Karimov wrote:
> Hi Matthias,
> 
> Thanks a lot for correcting. It is a leftover from the past designs when
> punctuate() was not deprecated.
> I corrected.
> 
> Cheers,
> Jeyhun
> 
> On Mon, Nov 6, 2017 at 5:30 PM Matthias J. Sax 
> wrote:
> 
>> I just re-read the KIP.
>>
>> One minor comment: we don't need to introduce any deprecated methods.
>> Thus, RichValueTransformer#punctuate can be removed completely instead
>> of introducing it as deprecated.
>>
>> Otherwise looks good to me.
>>
>> Thanks for being so patient!
>>
>>
>> -Matthias
>>
>> On 11/1/17 9:16 PM, Guozhang Wang wrote:
>>> Jeyhun,
>>>
>>> I think I'm convinced to not do KAFKA-3907 in this KIP. We should think
>>> carefully if we should add this functionality to the DSL layer moving
>>> forward since from what we discovered working on it the conclusion is
>> that
>>> it would require revamping the public APIs quite a lot, and it's not
>> clear
>>> if it is a good trade-off than asking users to call process() instead.
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Wed, Nov 1, 2017 at 4:50 AM, Damian Guy  wrote:
>>>
 Hi Jeyhun, thanks, looks good.
 Do we need to remove the line that says:

- on-demand commit() feature

 Cheers,
 Damian

 On Tue, 31 Oct 2017 at 23:07 Jeyhun Karimov 
>> wrote:

> Hi,
>
> I removed the 'commit()' feature, as we discussed. It simplified  the
> overall design of KIP a lot.
> 

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Ted Yu]

Thanks for the quick response.

It seems the config following --delete-config should be quoted.

Cheers

On Mon, Nov 20, 2017 at 12:02 PM, Rajini Sivaram 
wrote:

> Ted,
>
> Have added an example for --delete-config.
>
> On Mon, Nov 20, 2017 at 7:42 PM, Ted Yu  wrote:
>
> > bq. There is a --delete-config option
> >
> > Consider adding a sample with the above option to the KIP.
> >
> > Thanks
> >
> > On Mon, Nov 20, 2017 at 11:36 AM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> > wrote:
> >
> > > Hi Ted,
> > >
> > > Thank you for reviewing the KIP.
> > >
> > > *Would decreasing network/IO threads be supported ?*
> > > Yes, As described in the KIP, some connections will be closed if
> network
> > > thread count is reduced (and reconnections will be processed on
> remaining
> > > threads)
> > >
> > > *What if some keys in configs are not in the Set returned
> > > by reconfigurableConfigs()? Would exception be thrown ?*
> > > No, *reconfigurableConfigs() *will be used to decide which classes are
> > > notified when a configuration update is made*.
> **reconfigure(Map > ?>
> > > configs)* will be invoked with all of the configured configs of the
> > broker,
> > >  similar to  *configure(Map configs). *For example, when
> > > *SslChannelBuilder* is made reconfigurable, it could just create a new
> > > SslFactory with the latest configs, using the same code as
> *configure()*.
> > > We avoid reconfiguring *SslChannelBuilder *unnecessarily*, *for example
> > if
> > > a topic config has changed, since topic configs are not listed in the
> > > *SslChannelBuilder#**reconfigurableConfigs().*
> > >
> > > *The sample commands for bin/kafka-configs include '--add-config'.
> Would
> > > there be '--remove-config' ?*
> > > bin/kafka-configs.sh is an existing tool whose parameters will not be
> > > modified by this KIP. There is a --delete-config option.
> > >
> > > *ssl.keystore.password appears a few lines above. Would there be any
> > > issue with mixture of connections (with old and new password) ?*
> > > No, passwords (and the actual keystore) are only used during
> > > authentication. Any channel created using the old SslFactory will not
> be
> > > impacted.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Mon, Nov 20, 2017 at 4:39 PM, Ted Yu  wrote:
> > >
> > > > bq. (e.g. increase network/IO threads)
> > > >
> > > > Would decreasing network/IO threads be supported ?
> > > >
> > > > bq. void reconfigure(Map configs);
> > > >
> > > > What if some keys in configs are not in the Set returned by
> > > > reconfigurableConfigs()
> > > > ? Would exception be thrown ?
> > > > If so, please specify which exception would be thrown.
> > > >
> > > > The sample commands for bin/kafka-configs include '--add-config'.
> > > > Would there be '--remove-config' ?
> > > >
> > > > bq. Existing connections will not be affected, new connections will
> use
> > > the
> > > > new keystore.
> > > >
> > > > ssl.keystore.password appears a few lines above. Would there be any
> > issue
> > > > with mixture of connections (with old and new password) ?
> > > >
> > > >
> > > > Cheers
> > > >
> > > >
> > > >
> > > > On Mon, Nov 20, 2017 at 5:57 AM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have submitted KIP-226 to enable dynamic reconfiguration of
> brokers
> > > > > without restart:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 226+-+Dynamic+Broker+Configuration
> > > > >
> > > > > The KIP proposes to extend the current dynamic replication quota
> > > > > configuration for brokers to support dynamic reconfiguration of a
> > > limited
> > > > > set of configuration options that are typically updated during the
> > > > lifetime
> > > > > of a broker.
> > > > >
> > > > > Feedback and suggestions are welcome.
> > > > >
> > > > > Thank you...
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

[Matthias J. Sax]

Thanks for following up!

One thought about an older reply from you:

 I strongly disagree here. The purpose of this handler isn't *just* to
 make a decision for streams. There may also be desirable side effects that
 users wish to cause when production exceptions occur. There may be
 side-effects that they wish to cause when AuthenticationExceptions occur,
 as well. We should still give them the hooks to preform those side effects.

And your follow up:

>>- I think I would rather invoke it for all exceptions that could occur
>>from attempting to produce - even those exceptions were returning CONTINUE
>>may not be a good idea (e.g. Authorization exception). Until there is a
>>different type for exceptions that are totally fatal (for example a
>>FatalStreamsException or some sort), maintaining a list of exceptions that
>>you can intercept with this handler and exceptions you cannot would be
>>really error-prone and hard to keep correct.

I understand what you are saying, however, consider that Streams needs
to die for a fatal exception. Thus, if you call the handler for a fatal
exception, we would  need to ignore the return value and fail -- this
seems to be rather intuitive. Furthermore, users can register an
uncaught-exception-handler and side effects for fatal errors can be
triggered there.

Btw: an AuthorizationException is fatal -- not sure what you mean by an
"totally fatal" exception -- there is no superlative to fatal from my
understanding.

About maintaining a list of exceptions: I don't think this is too hard,
and users also don't need to worry about it IMHO. We would only exclude
exception Streams can handle itself (like ProducerFencedException) --
thus, if the handler has code to react to this, it would not be bad, as
this code is just never called. In case that there is an exception
Streams could actually handle but we still call the handler (ie, bug),
there should be no harm either -- the handler needs to return either
CONTINUE or FAIL and we would obey. It could only happen, that Streams
dies---as request by the user(!)---even if Streams could actually handle
the error and move on. But this seems to be not a issue from my point of
view.

Thus, I am still in favor of not calling the ProductionExceptionHandler
for fatal exception.



About the "always continue" case. Sounds good to me to remove it (not
sure why we need it in test package?) and to rename the "failing"
handler to "Default" (even if "default" is less descriptive and I would
still prefer "Fail" in the name).


Last question:

>>   - Continue to *only* invoke it on the first exception that we
>>   encounter (before sendException is set)


What is there reasoning for invoking the handler only for the first
exception?




-Matthias

On 11/20/17 10:43 AM, Matt Farmer wrote:
> Alright, here are some updates I'm planning to make after thinking on this
> for awhile:
> 
>- Given that the "always continue" handler isn't something I'd recommend
>for production use as is, I'm going to move it into the test namespace and
>remove it from mention in the public API.
>- I'm going to rename the "AlwaysFailProductionExceptionHandler" to
>"DefaultProductionExceptionHandler"
>- Given that the API for the exception handler involves being invoked by
>streams to make a decision about whether or not to continue — I think that
>we should:
>   - Continue to *only* invoke it on the first exception that we
>   encounter (before sendException is set)
>   - Stop invoking it for the self-healing fenced exceptions.
>- I think I would rather invoke it for all exceptions that could occur
>from attempting to produce - even those exceptions were returning CONTINUE
>may not be a good idea (e.g. Authorization exception). Until there is a
>different type for exceptions that are totally fatal (for example a
>FatalStreamsException or some sort), maintaining a list of exceptions that
>you can intercept with this handler and exceptions you cannot would be
>really error-prone and hard to keep correct.
>   - I'm happy to file a KIP for the creation of this new Exception type
>   if there is interest.
> 
> @ Matthias — What do you think about the above?
> 
> On Tue, Nov 14, 2017 at 9:44 AM Matt Farmer  wrote:
> 
>> I responded before reading your code review and didn't see the bit about
>> how ProducerFencedException is self-healing. This error handling logic is
>> *quite* confusing to reason about... I think I'm going to sit down with
>> the code a bit more today, but I'm inclined to think that if the fenced
>> exceptions are, indeed, self healing that we still invoke the handler but
>> ignore its result for those exceptions.
>>
>> On Tue, Nov 14, 2017 at 9:37 AM Matt Farmer  wrote:
>>
>>> Hi there,
>>>
>>> Following up here...
>>>
 One tiny comment: I would prefer to remove the "Always" from the
>>> handler 

Build failed in Jenkins: kafka-trunk-jdk7 #2984

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6170; KIP-220 Part 1: Add AdminClient to Streams

--
[...truncated 383.84 KB...]
kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED
ERROR: Could not install GRADLE_3_4_RC_2_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:887)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:421)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:629)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:594)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:391)
at hudson.scm.SCM.poll(SCM.java:408)
at hudson.model.AbstractProject._poll(AbstractProject.java:1394)
at hudson.model.AbstractProject.poll(AbstractProject.java:1297)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:594)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:640)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing PASSED

kafka.log.ProducerStateManagerTest > testTruncate STARTED

kafka.log.ProducerStateManagerTest > testTruncate PASSED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload STARTED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload PASSED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump STARTED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Rajini Sivaram]

Ted,

Have added an example for --delete-config.

On Mon, Nov 20, 2017 at 7:42 PM, Ted Yu  wrote:

> bq. There is a --delete-config option
>
> Consider adding a sample with the above option to the KIP.
>
> Thanks
>
> On Mon, Nov 20, 2017 at 11:36 AM, Rajini Sivaram 
> wrote:
>
> > Hi Ted,
> >
> > Thank you for reviewing the KIP.
> >
> > *Would decreasing network/IO threads be supported ?*
> > Yes, As described in the KIP, some connections will be closed if network
> > thread count is reduced (and reconnections will be processed on remaining
> > threads)
> >
> > *What if some keys in configs are not in the Set returned
> > by reconfigurableConfigs()? Would exception be thrown ?*
> > No, *reconfigurableConfigs() *will be used to decide which classes are
> > notified when a configuration update is made*. **reconfigure(Map ?>
> > configs)* will be invoked with all of the configured configs of the
> broker,
> >  similar to  *configure(Map configs). *For example, when
> > *SslChannelBuilder* is made reconfigurable, it could just create a new
> > SslFactory with the latest configs, using the same code as *configure()*.
> > We avoid reconfiguring *SslChannelBuilder *unnecessarily*, *for example
> if
> > a topic config has changed, since topic configs are not listed in the
> > *SslChannelBuilder#**reconfigurableConfigs().*
> >
> > *The sample commands for bin/kafka-configs include '--add-config'. Would
> > there be '--remove-config' ?*
> > bin/kafka-configs.sh is an existing tool whose parameters will not be
> > modified by this KIP. There is a --delete-config option.
> >
> > *ssl.keystore.password appears a few lines above. Would there be any
> > issue with mixture of connections (with old and new password) ?*
> > No, passwords (and the actual keystore) are only used during
> > authentication. Any channel created using the old SslFactory will not be
> > impacted.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Mon, Nov 20, 2017 at 4:39 PM, Ted Yu  wrote:
> >
> > > bq. (e.g. increase network/IO threads)
> > >
> > > Would decreasing network/IO threads be supported ?
> > >
> > > bq. void reconfigure(Map configs);
> > >
> > > What if some keys in configs are not in the Set returned by
> > > reconfigurableConfigs()
> > > ? Would exception be thrown ?
> > > If so, please specify which exception would be thrown.
> > >
> > > The sample commands for bin/kafka-configs include '--add-config'.
> > > Would there be '--remove-config' ?
> > >
> > > bq. Existing connections will not be affected, new connections will use
> > the
> > > new keystore.
> > >
> > > ssl.keystore.password appears a few lines above. Would there be any
> issue
> > > with mixture of connections (with old and new password) ?
> > >
> > >
> > > Cheers
> > >
> > >
> > >
> > > On Mon, Nov 20, 2017 at 5:57 AM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have submitted KIP-226 to enable dynamic reconfiguration of brokers
> > > > without restart:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 226+-+Dynamic+Broker+Configuration
> > > >
> > > > The KIP proposes to extend the current dynamic replication quota
> > > > configuration for brokers to support dynamic reconfiguration of a
> > limited
> > > > set of configuration options that are typically updated during the
> > > lifetime
> > > > of a broker.
> > > >
> > > > Feedback and suggestions are welcome.
> > > >
> > > > Thank you...
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
>

Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

[Jun Rao]

Hi, Hu,

For the new partition level metrics that you are adding, it seems it's
better to just add the topic/partition tag instead of using them in the
prefix. For the existing lag metrics, we can fix them in KIP-225.

Thanks,

Jun

On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi  wrote:

> Jun,
>
>
> Thanks for the comments. Do you think it'd better to add topic/partition
> tags for those metrics as well as keep the prefix? If those prefixes should
> really be removed, does this KIP need to do the same thing for `lag` ones?
>
> 
> 发件人: Jun Rao 
> 发送时间: 2017年11月18日 8:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Charly,
>
> Thanks for the input. It makes sense.
>
> Hi, Hu,
>
> Perhaps we can keep the per partition records-lead-min and records-lead-avg
> as you had before, but just add the topic and the partition as the tags
> instead of prefix of the metric name.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 15, 2017 at 4:58 AM, charly molter 
> wrote:
>
> > Hi Jun, Jiangle,
> >
> > I'd just like to clarify that KIP-225 seems to be using per partition
> > metric the same way as KIP-223 seems to be doing.
> >
> > I believe avg and max are still necessary because the MetricsReporter
> > doesn't work in a "push" manner and the "Value" measurableStat will only
> > keep the last recorded entry.
> > Therefore a MetricsReporter usually polls to grab a current view with
> Value
> > this view is incomplete so it becomes not possible to compute the
> > Max/Min/Avg.
> > Max/Min/Avg uses SampledStats which work with a rolling window of samples
> > and therefore periodic polling would work.
> >
> > This is why I believe it's necessary to keep Avg, Min and Max for these
> > metrics as otherwise we wouldn't be able to recompute it in an external
> > monitoring system.
> >
> > Am I wrong thinking this?
> >
> > Thanks,
> > Charly
> >
> >
> > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao  wrote:
> >
> > > Hi, Charly,
> > >
> > > Thanks for KIP-225. Your proposal looks reasonable.
> > >
> > > Hi, Jiangjie,
> > >
> > > Do you think the approach that KIP-225 proposes is better for exposing
> > the
> > > per partition metric? Also, do we really need the per partition
> > > record-lag-avg
> > > and record-lag-max? It seems that an external monitoring system can
> > always
> > > derive that from the per partition record-lag.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> charly.mol...@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Hu,
> > > >
> > > > I have KIP-225 open for adding tags to records-lag:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74686649
> > > >
> > > > I have a patch more or less ready so I could probably get the fix
> > checked
> > > > in (after the vote) and you could build on top of it. Otherwise we
> > could
> > > > merge both KIPs if you want but they do sound different to me.
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi  wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > >
> > > > > Let me double confirm with your comments:
> > > > >
> > > > > 1 remove partition-level records-lead-avg and records-lead-min
> since
> > > they
> > > > > both can be deduced by external monitoring system.
> > > > >
> > > > > 2 Tag partition-level records-lead with topic info
> > > > >
> > > > >
> > > > > If they are the case you expect, do we need to do the same thing
> for
> > > > those
> > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > tagged
> > > > > with topic information  which might deserve a bug.
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > > 
> > > > > 发件人: Jun Rao 
> > > > > 发送时间: 2017年11月14日 12:44
> > > > > 收件人: dev@kafka.apache.org
> > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}".
> > > > > So, it probably makes sense for records-lead-min to be an attribute
> > > under
> > > > > the same mbean.
> > > > >
> > > > > The partition level records-lead can probably be an attribute for
> the
> > > > mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > where topic and partition are the tags. This matches the topic
> level
> > > > mbeans
> > > > > that we have in the consumer. I am not sure what the per partition
> > > level
> > > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> > the
> > > > lead
> 

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

[Jan Filipiak]

On 19.11.2017 21:12, Guozhang Wang wrote:

Jan: which approach are you referring to as "the approach that is on the
table would be perfect"?

The SourcesKStream/Table suggestion.


Note that in today's PAPI layer we are already effectively exposing the
record context which has the issues that we have been discussing right now,
and its semantics is always referring to the "processing record" at hand.
More specifically, we can think of processing a record a bit different:

1) the record traversed the topology from source to sink, it may be
transformed into new object or even generate multiple new objects (think:
branch) along the traversal. And the record context is referring to this
processing record. Here the "lifetime" of the record lasts for the entire
topology traversal and any new records of this traversal is treated as
different transformed values of this record (this applies to join and
aggregations as well).

2) the record being processed is wiped out in the first operator after the
source, and NEW records are forwarded to downstream operators. I.e. each
record only lives between two adjacent operators, once it reached the new
operator it's lifetime has ended and new records are generated.

I think in the past we have talked about Streams under both context, and we
do not have a clear agreement. I agree that 2) is logically more
understandable for users as it does not leak any internal implementation
details (e.g. for stream-table joins, table record's traversal ends at the
join operator as it is only be materialized, while stream record's
traversal goes through the join operator to further down until sinks).
However if we are going to interpret following 2) above then even for
non-stateful operators we would not inherit record context. What we're
discussing now, seems to infer a third semantics:

3) a record would traverse "through" one-to-one (non-stateful) operators,
will "replicate" at one-to-many (non-stateful) operators (think: "mapValues"
  ) and will "end" at many-to-one (stateful) operators where NEW records
will be generated and forwarded to the downstream operators.

There is this problem that even though we have source.table.filter.join
the state-fullness happens at the table step not a the join step. In a 
filter

we still going to present change.oldValue to the filter even though the
record context() is for change.newValue. I would go as far as applying
the filter before the table processor. Not to just get KIP-159, but because
I think its a side effect of a non ideal topology layout. If i can 
filter 99% of my

records. my state could be way smaller. Also widely escalates the context
of the KIP


Just wanted to lay the ground for discussions so we are all on the same
page before chatting more.


Guozhang


On Sat, Nov 18, 2017 at 3:10 AM, Jan Filipiak 
wrote:


Hi,

  not an issue at all. IMO
the approach that is on the table would be perfect


On 18.11.2017 10:58, Jeyhun Karimov wrote:


Hi,

I did not expected that Context will be this much an issue. Instead of
applying different semantics for different operators, I think we should
remove this feature completely.


Cheers,
Jeyhun
On Sat 18. Nov 2017 at 07:49, Jan Filipiak 
wrote:

Yes, the mail said only join so I wanted to clarify.



On 17.11.2017 19:05, Matthias J. Sax wrote:


Yes. But I think an aggregation is an many-to-one operation, too.

For the stripping off part: internally, we can just keep some record
context, but just do not allow users to access it (because the context
context does not make sense for them) by hiding the corresponding APIs.


-Matthias

On 11/16/17 10:05 PM, Guozhang Wang wrote:


Matthias,

For this idea, are your proposing that for any many-to-one mapping
operations (for now only Join operators), we will strip off the record
context in the resulted records and claim "we cannot infer its traced
context anymore"?


Guozhang


On Thu, Nov 16, 2017 at 1:03 PM, Matthias J. Sax <
matth...@confluent.io
wrote:

Any thoughts about my latest proposal?

-Matthias

On 11/10/17 10:02 PM, Jan Filipiak wrote:


Hi,

i think this is the better way. Naming is always tricky Source is


kinda

taken

I had TopicBackedK[Source|Table] in mind
but for the user its way better already IMHO

Thank you for reconsideration

Best Jan


On 10.11.2017 22:48, Matthias J. Sax wrote:


I was thinking about the source stream/table idea once more and it


seems

it would not be too hard to implement:

We add two new classes

  SourceKStream extends KStream

and

  SourceKTable extend KTable

and return both from StreamsBuilder#stream and StreamsBuilder#table

As both are sub-classes, this change is backward compatible. We


change

the return type for any single-record transform to this new types,

too,

and use KStream/KTable as return type for any multi-record operation.

The new RecordContext API is added to both new classes. For old


classes,

we only implement 

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Ted Yu]

bq. There is a --delete-config option

Consider adding a sample with the above option to the KIP.

Thanks

On Mon, Nov 20, 2017 at 11:36 AM, Rajini Sivaram 
wrote:

> Hi Ted,
>
> Thank you for reviewing the KIP.
>
> *Would decreasing network/IO threads be supported ?*
> Yes, As described in the KIP, some connections will be closed if network
> thread count is reduced (and reconnections will be processed on remaining
> threads)
>
> *What if some keys in configs are not in the Set returned
> by reconfigurableConfigs()? Would exception be thrown ?*
> No, *reconfigurableConfigs() *will be used to decide which classes are
> notified when a configuration update is made*. **reconfigure(Map
> configs)* will be invoked with all of the configured configs of the broker,
>  similar to  *configure(Map configs). *For example, when
> *SslChannelBuilder* is made reconfigurable, it could just create a new
> SslFactory with the latest configs, using the same code as *configure()*.
> We avoid reconfiguring *SslChannelBuilder *unnecessarily*, *for example if
> a topic config has changed, since topic configs are not listed in the
> *SslChannelBuilder#**reconfigurableConfigs().*
>
> *The sample commands for bin/kafka-configs include '--add-config'. Would
> there be '--remove-config' ?*
> bin/kafka-configs.sh is an existing tool whose parameters will not be
> modified by this KIP. There is a --delete-config option.
>
> *ssl.keystore.password appears a few lines above. Would there be any
> issue with mixture of connections (with old and new password) ?*
> No, passwords (and the actual keystore) are only used during
> authentication. Any channel created using the old SslFactory will not be
> impacted.
>
> Regards,
>
> Rajini
>
>
> On Mon, Nov 20, 2017 at 4:39 PM, Ted Yu  wrote:
>
> > bq. (e.g. increase network/IO threads)
> >
> > Would decreasing network/IO threads be supported ?
> >
> > bq. void reconfigure(Map configs);
> >
> > What if some keys in configs are not in the Set returned by
> > reconfigurableConfigs()
> > ? Would exception be thrown ?
> > If so, please specify which exception would be thrown.
> >
> > The sample commands for bin/kafka-configs include '--add-config'.
> > Would there be '--remove-config' ?
> >
> > bq. Existing connections will not be affected, new connections will use
> the
> > new keystore.
> >
> > ssl.keystore.password appears a few lines above. Would there be any issue
> > with mixture of connections (with old and new password) ?
> >
> >
> > Cheers
> >
> >
> >
> > On Mon, Nov 20, 2017 at 5:57 AM, Rajini Sivaram  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I have submitted KIP-226 to enable dynamic reconfiguration of brokers
> > > without restart:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 226+-+Dynamic+Broker+Configuration
> > >
> > > The KIP proposes to extend the current dynamic replication quota
> > > configuration for brokers to support dynamic reconfiguration of a
> limited
> > > set of configuration options that are typically updated during the
> > lifetime
> > > of a broker.
> > >
> > > Feedback and suggestions are welcome.
> > >
> > > Thank you...
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> >
>

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Rajini Sivaram]

Hi Ted,

Thank you for reviewing the KIP.

*Would decreasing network/IO threads be supported ?*
Yes, As described in the KIP, some connections will be closed if network
thread count is reduced (and reconnections will be processed on remaining
threads)

*What if some keys in configs are not in the Set returned
by reconfigurableConfigs()? Would exception be thrown ?*
No, *reconfigurableConfigs() *will be used to decide which classes are
notified when a configuration update is made*. **reconfigure(Map
configs)* will be invoked with all of the configured configs of the broker,
 similar to  *configure(Map configs). *For example, when
*SslChannelBuilder* is made reconfigurable, it could just create a new
SslFactory with the latest configs, using the same code as *configure()*.
We avoid reconfiguring *SslChannelBuilder *unnecessarily*, *for example if
a topic config has changed, since topic configs are not listed in the
*SslChannelBuilder#**reconfigurableConfigs().*

*The sample commands for bin/kafka-configs include '--add-config'. Would
there be '--remove-config' ?*
bin/kafka-configs.sh is an existing tool whose parameters will not be
modified by this KIP. There is a --delete-config option.

*ssl.keystore.password appears a few lines above. Would there be any
issue with mixture of connections (with old and new password) ?*
No, passwords (and the actual keystore) are only used during
authentication. Any channel created using the old SslFactory will not be
impacted.

Regards,

Rajini


On Mon, Nov 20, 2017 at 4:39 PM, Ted Yu  wrote:

> bq. (e.g. increase network/IO threads)
>
> Would decreasing network/IO threads be supported ?
>
> bq. void reconfigure(Map configs);
>
> What if some keys in configs are not in the Set returned by
> reconfigurableConfigs()
> ? Would exception be thrown ?
> If so, please specify which exception would be thrown.
>
> The sample commands for bin/kafka-configs include '--add-config'.
> Would there be '--remove-config' ?
>
> bq. Existing connections will not be affected, new connections will use the
> new keystore.
>
> ssl.keystore.password appears a few lines above. Would there be any issue
> with mixture of connections (with old and new password) ?
>
>
> Cheers
>
>
>
> On Mon, Nov 20, 2017 at 5:57 AM, Rajini Sivaram 
> wrote:
>
> > Hi all,
> >
> > I have submitted KIP-226 to enable dynamic reconfiguration of brokers
> > without restart:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 226+-+Dynamic+Broker+Configuration
> >
> > The KIP proposes to extend the current dynamic replication quota
> > configuration for brokers to support dynamic reconfiguration of a limited
> > set of configuration options that are typically updated during the
> lifetime
> > of a broker.
> >
> > Feedback and suggestions are welcome.
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
> >
>

[GitHub] kafka pull request #4211: KAFKA-6170; KIP-220 Part 1: Add AdminClient to Str...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4211


---

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

[Guozhang Wang]

+1 from myself as well.

I'm closing this KIP as accepted with 3 binding votes (Gwen, Damian, me)
and 3 non-binding votes (Colin, Ted, Matthias).


Guozhang

On Mon, Nov 20, 2017 at 9:56 AM, Damian Guy  wrote:

> +1
>
> On Mon, 20 Nov 2017 at 17:52 Gwen Shapira  wrote:
>
> > +1
> >
> > Make sense. We have a supplier for every other client type :)
> >
> > On Fri, Nov 17, 2017 at 1:33 PM Matthias J. Sax 
> > wrote:
> >
> > > +1
> > >
> > > On 11/17/17 9:35 AM, Ted Yu wrote:
> > > > +1
> > > >
> > > > On Fri, Nov 17, 2017 at 9:34 AM, Bill Bejeck 
> > wrote:
> > > >
> > > >> +1
> > > >>
> > > >> Thanks,
> > > >> Bill
> > > >>
> > > >> On Fri, Nov 17, 2017 at 12:13 PM, Colin McCabe 
> > > wrote:
> > > >>
> > > >>> +1 (non-binding)
> > > >>>
> > > >>> Colin
> > > >>>
> > > >>> On Tue, Nov 14, 2017, at 10:02, Guozhang Wang wrote:
> > >  Hello folks,
> > > 
> > >  I have filed a new KIP on adding AdminClient into Streams for
> > internal
> > >  topic management.
> > > 
> > >  Please review and cast your vote on this thread.
> > > 
> > >  *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
> > >   > > >>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier>*
> > > 
> > >  The discussion is in another thread so if you have detailed
> > questions
> > >  please chime in there.
> > > 
> > > 
> > >  -- Guozhang
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

[Guozhang Wang]

@Jeyhun,

I understand that the discussion about KIP-159 is dragging long, so while
we are moving on discussion for whether / hows of KIP-159, maybe you can
start implementing the non overlapping part of the APIs of KIP-149 to get
you unblocked?

Guozhang

On Sun, Nov 19, 2017 at 12:12 PM, Guozhang Wang  wrote:

> Jan: which approach are you referring to as "the approach that is on the
> table would be perfect"?
>
> Note that in today's PAPI layer we are already effectively exposing the
> record context which has the issues that we have been discussing right now,
> and its semantics is always referring to the "processing record" at hand.
> More specifically, we can think of processing a record a bit different:
>
> 1) the record traversed the topology from source to sink, it may be
> transformed into new object or even generate multiple new objects (think:
> branch) along the traversal. And the record context is referring to this
> processing record. Here the "lifetime" of the record lasts for the entire
> topology traversal and any new records of this traversal is treated as
> different transformed values of this record (this applies to join and
> aggregations as well).
>
> 2) the record being processed is wiped out in the first operator after the
> source, and NEW records are forwarded to downstream operators. I.e. each
> record only lives between two adjacent operators, once it reached the new
> operator it's lifetime has ended and new records are generated.
>
> I think in the past we have talked about Streams under both context, and
> we do not have a clear agreement. I agree that 2) is logically more
> understandable for users as it does not leak any internal implementation
> details (e.g. for stream-table joins, table record's traversal ends at the
> join operator as it is only be materialized, while stream record's
> traversal goes through the join operator to further down until sinks).
> However if we are going to interpret following 2) above then even for
> non-stateful operators we would not inherit record context. What we're
> discussing now, seems to infer a third semantics:
>
> 3) a record would traverse "through" one-to-one (non-stateful) operators,
> will "replicate" at one-to-many (non-stateful) operators (think:
> "mapValues" ) and will "end" at many-to-one (stateful) operators where
> NEW records will be generated and forwarded to the downstream operators.
>
> Just wanted to lay the ground for discussions so we are all on the same
> page before chatting more.
>
>
> Guozhang
>
>
> On Sat, Nov 18, 2017 at 3:10 AM, Jan Filipiak 
> wrote:
>
>> Hi,
>>
>>  not an issue at all. IMO
>> the approach that is on the table would be perfect
>>
>>
>> On 18.11.2017 10:58, Jeyhun Karimov wrote:
>>
>>> Hi,
>>>
>>> I did not expected that Context will be this much an issue. Instead of
>>> applying different semantics for different operators, I think we should
>>> remove this feature completely.
>>>
>>>
>>> Cheers,
>>> Jeyhun
>>> On Sat 18. Nov 2017 at 07:49, Jan Filipiak 
>>> wrote:
>>>
>>> Yes, the mail said only join so I wanted to clarify.



 On 17.11.2017 19:05, Matthias J. Sax wrote:

> Yes. But I think an aggregation is an many-to-one operation, too.
>
> For the stripping off part: internally, we can just keep some record
> context, but just do not allow users to access it (because the context
> context does not make sense for them) by hiding the corresponding APIs.
>
>
> -Matthias
>
> On 11/16/17 10:05 PM, Guozhang Wang wrote:
>
>> Matthias,
>>
>> For this idea, are your proposing that for any many-to-one mapping
>> operations (for now only Join operators), we will strip off the record
>> context in the resulted records and claim "we cannot infer its traced
>> context anymore"?
>>
>>
>> Guozhang
>>
>>
>> On Thu, Nov 16, 2017 at 1:03 PM, Matthias J. Sax <
>> matth...@confluent.io
>> wrote:
>>
>> Any thoughts about my latest proposal?
>>>
>>> -Matthias
>>>
>>> On 11/10/17 10:02 PM, Jan Filipiak wrote:
>>>
 Hi,

 i think this is the better way. Naming is always tricky Source is

>>> kinda

> taken
 I had TopicBackedK[Source|Table] in mind
 but for the user its way better already IMHO

 Thank you for reconsideration

 Best Jan


 On 10.11.2017 22:48, Matthias J. Sax wrote:

> I was thinking about the source stream/table idea once more and it
>
 seems

> it would not be too hard to implement:
>
> We add two new classes
>
>  SourceKStream extends KStream
>
> and
>
>  SourceKTable extend KTable
>
> and return both from StreamsBuilder#stream and 

[GitHub] kafka pull request #4240: KAFKA-6247. Fix system test dependency issues

[cmccabe]

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/4240

KAFKA-6247. Fix system test dependency issues

Fix an omission where Kibosh was not getting installed on Vagrant
instances running in AWS.

Centralize test dependency versions in tests/versions.sh.

Fix an issue where the Dockerfile was unable to download old Apache
Kafka releases.  See the discussion on KAFKA-6233.

*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*

*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-6247

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4240.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4240


commit 233c5ab30dbbf855e2451f2cbe6a8808e12cbcb7
Author: Colin P. Mccabe 
Date:   2017-11-20T18:54:12Z

KAFKA-6247. Fix system test dependency issues

Fix an omission where Kibosh was not getting installed on Vagrant
instances running in AWS.

Centralize test dependency versions in tests/versions.sh.

Fix an issue where the Dockerfile was unable to download old Apache
Kafka releases.  See the discussion on KAFKA-6233.




---

[jira] [Created] (KAFKA-6247) Fix system test dependency issues

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-6247:
--

 Summary: Fix system test dependency issues
 Key: KAFKA-6247
 URL: https://issues.apache.org/jira/browse/KAFKA-6247
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Reporter: Colin P. McCabe


Kibosh needs to be installed on Vagrant instances as well as in Docker 
environments.  And we need to download old Apache Kafka releases from a stable 
mirror that will not purge old releases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-224: Add configuration parameters `retries` to Streams API

[Matthias J. Sax]

Your understanding is correct and thanks for voting.

I am also +1 :)


I am closing this vote as accepted with 3 binding (Gwen, Guozhang,
Damian) and 3 non-binding (Ted, Bill, Matthias) votes.


-Matthias


On 11/20/17 9:58 AM, Gwen Shapira wrote:
> +1
> 
> My understanding is that the KIP is going to take two existing parameters
> (retries and retries-backoff) and apply them to GlobalKTable implementation
> where previously the settings were ignored and behavior was hard-coded.
> 
> If my understanding is incorrect, please disregard my vote.
> 
> Gwen
> 
> On Tue, Nov 14, 2017 at 11:44 AM Guozhang Wang  wrote:
> 
>> +1
>>
>> On Tue, Nov 14, 2017 at 3:02 AM, Damian Guy  wrote:
>>
>>> +1
>>>
>>> On Tue, 14 Nov 2017 at 02:40 Bill Bejeck  wrote:
>>>
 Thanks for the KIP, +1

 -Bill

 On Mon, Nov 13, 2017 at 7:25 PM, Ted Yu  wrote:

> +1
>
> On Mon, Nov 13, 2017 at 4:20 PM, Matthias J. Sax <
>>> matth...@confluent.io>
> wrote:
>
>> Hi @all,
>>
>> I would like to start the vote for KIP-224:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 224%3A+Add+configuration+parameter+%60retries%60+to+Streams+API
>>
>>
>> Thanks a lot!
>>
>>
>> -Matthias
>>
>>
>

>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

[Matt Farmer]

Alright, here are some updates I'm planning to make after thinking on this
for awhile:

   - Given that the "always continue" handler isn't something I'd recommend
   for production use as is, I'm going to move it into the test namespace and
   remove it from mention in the public API.
   - I'm going to rename the "AlwaysFailProductionExceptionHandler" to
   "DefaultProductionExceptionHandler"
   - Given that the API for the exception handler involves being invoked by
   streams to make a decision about whether or not to continue — I think that
   we should:
  - Continue to *only* invoke it on the first exception that we
  encounter (before sendException is set)
  - Stop invoking it for the self-healing fenced exceptions.
   - I think I would rather invoke it for all exceptions that could occur
   from attempting to produce - even those exceptions were returning CONTINUE
   may not be a good idea (e.g. Authorization exception). Until there is a
   different type for exceptions that are totally fatal (for example a
   FatalStreamsException or some sort), maintaining a list of exceptions that
   you can intercept with this handler and exceptions you cannot would be
   really error-prone and hard to keep correct.
  - I'm happy to file a KIP for the creation of this new Exception type
  if there is interest.

@ Matthias — What do you think about the above?

On Tue, Nov 14, 2017 at 9:44 AM Matt Farmer  wrote:

> I responded before reading your code review and didn't see the bit about
> how ProducerFencedException is self-healing. This error handling logic is
> *quite* confusing to reason about... I think I'm going to sit down with
> the code a bit more today, but I'm inclined to think that if the fenced
> exceptions are, indeed, self healing that we still invoke the handler but
> ignore its result for those exceptions.
>
> On Tue, Nov 14, 2017 at 9:37 AM Matt Farmer  wrote:
>
>> Hi there,
>>
>> Following up here...
>>
>> > One tiny comment: I would prefer to remove the "Always" from the
>> handler implementation names -- it sounds "cleaner" to me without it.
>>
>> Let me think on this. I generally prefer expressiveness to clean-ness,
>> and I think that comes out in the names I chose for things. The straw man
>> in my head is the person that tried to substitute in the "AlwaysContinue"
>> variant and the "Always" is a trigger to really consider whether or not
>> they *always* want to try to continue.
>>
>> To be truthful, after some thought, using the "AlwaysContinue" variant
>> isn't something I'd recommend generally in a production system. Ideally
>> these handlers are targeted at handling a specific series of exceptions
>> that a user wants to ignore, and not ignoring all exceptions. More on this
>> in a minute.
>>
>> > For the first category, it seems to not make sense to call the handle but
>> Streams should always fail. If we follow this design, the KIP should list
>> all exceptions for which the handler is not called.
>>
>> I strongly disagree here. The purpose of this handler isn't *just* to
>> make a decision for streams. There may also be desirable side effects that
>> users wish to cause when production exceptions occur. There may be
>> side-effects that they wish to cause when AuthenticationExceptions occur,
>> as well. We should still give them the hooks to preform those side effects.
>>
>> In light of the above, I'm thinking that the
>> "AlwaysContinueProductionExceptionHandler" variant should probably be
>> removed from the public API and moved into tests since that's where it's
>> most useful. The more I think on it, the more I feel that having that in
>> the public API is a landmine. If you agree, then, we can rename the
>> "AlwaysFailProductionExceptionHandler" to
>> "DefaultProductionExceptionHandler".
>>
>> Thoughts?
>>
>> On Fri, Nov 10, 2017 at 6:13 PM Matthias J. Sax 
>> wrote:
>>
>>> I just review the PR, and there is one thing we should discuss.
>>>
>>> There are different types of exceptions that could occur. Some apply to
>>> all records (like Authorization exception) while others are for single
>>> records only (like record too large).
>>>
>>> For the first category, it seems to not make sense to call the handle
>>> but Streams should always fail. If we follow this design, the KIP should
>>> list all exceptions for which the handler is not called.
>>>
>>> WDYT?
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 11/10/17 2:56 PM, Matthias J. Sax wrote:
>>> > Just catching up on this KIP.
>>> >
>>> > One tiny comment: I would prefer to remove the "Always" from the
>>> handler
>>> > implementation names -- it sounds "cleaner" to me without it.
>>> >
>>> >
>>> > -Matthias
>>> >
>>> > On 11/5/17 12:57 PM, Matt Farmer wrote:
>>> >> It is agreed, then. I've updated the pull request. I'm trying to also
>>> >> update the KIP accordingly, but cwiki is being slow and dropping
>>> >> connections. I'll try again a bit later but please 

[jira] [Reopened] (KAFKA-6237) stream stopped working after exception: Cannot execute transactional method because we are in an error state

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-6237:


> stream stopped working after exception: Cannot execute transactional method 
> because we are in an error state
> 
>
> Key: KAFKA-6237
> URL: https://issues.apache.org/jira/browse/KAFKA-6237
> Project: Kafka
>  Issue Type: Bug
>  Components: core, streams
>Reporter: DHRUV BANSAL
>Priority: Critical
> Attachments: nohup.out
>
>
> 017-11-19 07:52:44,673 
> [project_logs_stream-a30ea242-3c9f-46a9-a01c-51903bd40ca5-StreamThread-1] 
> ERROR: org.apache.kafka.streams.processor.internals.StreamThread - 
> stream-thread 
> [orion_logs_stream-a30ea242-3c9f-46a9-a01c-51903bd40ca5-StreamThread-1] 
> Failed while closing StreamTask 0_1:
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:524)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.beginAbort(TransactionManager.java:198)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.abortTransaction(KafkaProducer.java:598)
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:434)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:1086)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:1041)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:538)
> Caused by: org.apache.kafka.common.KafkaException: Unexpected error in 
> AddOffsetsToTxnResponse: The server experienced an unexpected error when 
> processing the request
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager$AddOffsetsToTxnHandler.handleResponse(TransactionManager.java:978)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:648)
>   at 
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
>   at 
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:454)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:446)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:206)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>   at java.lang.Thread.run(Thread.java:745)
> Also when I see the state of the corresponding consumer group it is saying:
> +Warning: Consumer group  is rebalancing.+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-6237) stream stopped working after exception: Cannot execute transactional method because we are in an error state

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6237.

Resolution: Duplicate

Thanks for reporting this. It is a duplicate and the issue is fixed in 
{{0.11.0.1}}.

> stream stopped working after exception: Cannot execute transactional method 
> because we are in an error state
> 
>
> Key: KAFKA-6237
> URL: https://issues.apache.org/jira/browse/KAFKA-6237
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: DHRUV BANSAL
>Priority: Critical
> Attachments: nohup.out
>
>
> 017-11-19 07:52:44,673 
> [project_logs_stream-a30ea242-3c9f-46a9-a01c-51903bd40ca5-StreamThread-1] 
> ERROR: org.apache.kafka.streams.processor.internals.StreamThread - 
> stream-thread 
> [orion_logs_stream-a30ea242-3c9f-46a9-a01c-51903bd40ca5-StreamThread-1] 
> Failed while closing StreamTask 0_1:
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:524)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.beginAbort(TransactionManager.java:198)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.abortTransaction(KafkaProducer.java:598)
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:434)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:1086)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:1041)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:538)
> Caused by: org.apache.kafka.common.KafkaException: Unexpected error in 
> AddOffsetsToTxnResponse: The server experienced an unexpected error when 
> processing the request
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager$AddOffsetsToTxnHandler.handleResponse(TransactionManager.java:978)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:648)
>   at 
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
>   at 
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:454)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:446)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:206)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>   at java.lang.Thread.run(Thread.java:745)
> Also when I see the state of the corresponding consumer group it is saying:
> +Warning: Consumer group  is rebalancing.+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-224: Add configuration parameters `retries` to Streams API

[Gwen Shapira]

+1

My understanding is that the KIP is going to take two existing parameters
(retries and retries-backoff) and apply them to GlobalKTable implementation
where previously the settings were ignored and behavior was hard-coded.

If my understanding is incorrect, please disregard my vote.

Gwen

On Tue, Nov 14, 2017 at 11:44 AM Guozhang Wang  wrote:

> +1
>
> On Tue, Nov 14, 2017 at 3:02 AM, Damian Guy  wrote:
>
> > +1
> >
> > On Tue, 14 Nov 2017 at 02:40 Bill Bejeck  wrote:
> >
> > > Thanks for the KIP, +1
> > >
> > > -Bill
> > >
> > > On Mon, Nov 13, 2017 at 7:25 PM, Ted Yu  wrote:
> > >
> > > > +1
> > > >
> > > > On Mon, Nov 13, 2017 at 4:20 PM, Matthias J. Sax <
> > matth...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hi @all,
> > > > >
> > > > > I would like to start the vote for KIP-224:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 224%3A+Add+configuration+parameter+%60retries%60+to+Streams+API
> > > > >
> > > > >
> > > > > Thanks a lot!
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

[Damian Guy]

+1

On Mon, 20 Nov 2017 at 17:52 Gwen Shapira  wrote:

> +1
>
> Make sense. We have a supplier for every other client type :)
>
> On Fri, Nov 17, 2017 at 1:33 PM Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 11/17/17 9:35 AM, Ted Yu wrote:
> > > +1
> > >
> > > On Fri, Nov 17, 2017 at 9:34 AM, Bill Bejeck 
> wrote:
> > >
> > >> +1
> > >>
> > >> Thanks,
> > >> Bill
> > >>
> > >> On Fri, Nov 17, 2017 at 12:13 PM, Colin McCabe 
> > wrote:
> > >>
> > >>> +1 (non-binding)
> > >>>
> > >>> Colin
> > >>>
> > >>> On Tue, Nov 14, 2017, at 10:02, Guozhang Wang wrote:
> >  Hello folks,
> > 
> >  I have filed a new KIP on adding AdminClient into Streams for
> internal
> >  topic management.
> > 
> >  Please review and cast your vote on this thread.
> > 
> >  *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
> >   > >>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier>*
> > 
> >  The discussion is in another thread so if you have detailed
> questions
> >  please chime in there.
> > 
> > 
> >  -- Guozhang
> > >>>
> > >>
> > >
> >
> >
>

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

[Gwen Shapira]

+1

Make sense. We have a supplier for every other client type :)

On Fri, Nov 17, 2017 at 1:33 PM Matthias J. Sax 
wrote:

> +1
>
> On 11/17/17 9:35 AM, Ted Yu wrote:
> > +1
> >
> > On Fri, Nov 17, 2017 at 9:34 AM, Bill Bejeck  wrote:
> >
> >> +1
> >>
> >> Thanks,
> >> Bill
> >>
> >> On Fri, Nov 17, 2017 at 12:13 PM, Colin McCabe 
> wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> Colin
> >>>
> >>> On Tue, Nov 14, 2017, at 10:02, Guozhang Wang wrote:
>  Hello folks,
> 
>  I have filed a new KIP on adding AdminClient into Streams for internal
>  topic management.
> 
>  Please review and cast your vote on this thread.
> 
>  *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
>   >>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier>*
> 
>  The discussion is in another thread so if you have detailed questions
>  please chime in there.
> 
> 
>  -- Guozhang
> >>>
> >>
> >
>
>

Kafka schema registry problem

[Luca Del Grosso]

Hi
i have a problem, i would like to create an avro schema on schema registry
of kafka that references a type that is declared in a different avro
schema, the places under an example:

First avro schema with reference UserAction

{"namespace": "com.myorg.other",
 "type": "record",
 "name": "SearchSuggest",
 "fields": [
 {"name": "name", "type": "string"},
 {"name": "userAction", "type": "UserAction"}
 ]
}

second avro schema with enum:

{"namespace": "com.myorg.other",
 "type": "enum",
 "name": "UserAction",
 "symbols": ["S", "V", "C"]
}

This work in my maven project, but when i try create this on schema
registry, it's invalid.

[GitHub] kafka pull request #4239: KAFKA-6214: enable use of in-memory store for stan...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/4239

KAFKA-6214: enable use of in-memory store for standby tasks

Remove the flag in `ProcessorStateManager` that checks if a store is 
persistent when registering it as a standby task.
Updated the smoke test to use an in-memory store.

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-6214

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4239


commit a9cc7b3bbf2bc42462d0876c6c13fdc8173f47ec
Author: Damian Guy 
Date:   2017-11-20T13:30:35Z

enable use of in-memory store for standby tasks




---

Re: Request to add to contributor's list...

[Guozhang Wang]

Done.

Cheers,
Guozhang

On Sun, Nov 19, 2017 at 11:12 AM, Alex Ott  wrote:

> Hello
>
> I have some free time & want to try to contribute to Kafka via resolving
> issues, and contributing to documentation.
> Please give me access rights for work with issues in JIRA & wiki edit
> rights
>
> My JIRA name is alexott
>
> Thank you
>
> --
> With best wishes, Alex Ott
> http://alexott.blogspot.com/http://alexott.net/
> http://alexott-ru.blogspot.com/
> Skype: alex.ott
>



-- 
-- Guozhang

Re: [DISCUSS] Kafka 2.0.0 in June 2018

[Ismael Juma]

Yes, makes sense.

Ismael

On Mon, Nov 20, 2017 at 4:49 PM, Gwen Shapira  wrote:

> Agree. I don't know that it actually matters. They can keep using whatever
> they are using now since we don't plan on breaking the protocol.
>
> But since the issue does keep coming up, I figured we'll need a clear
> message around what the removal means and what users need to do.
>
> Gwen
>
>
> On Mon, Nov 20, 2017 at 8:21 AM Ismael Juma  wrote:
>
> > It's worth emphasizing that the impact to such users is independent of
> > whether we remove the old high-level consumer in 2.0.0 or not. They are
> > unable to use the message format introduced in 0.11.0 or security
> features
> > today.
> >
> > Ismael
> >
> > On Mon, Nov 20, 2017 at 4:11 PM, Gwen Shapira  wrote:
> >
> > > >
> > > >
> > > > Personally, I suspect that those who absolutely need a rolling
> > migration
> > > > and cannot handle a short period of downtime while doing a migration
> > > > probably have in-house experts on Kafka who are familiar with the
> > issues
> > > > and willing to figure out a solution. The rest of the world can
> > generally
> > > > handle a short maintenance window.
> > > >
> > >
> > > I really wish that was true :)
> > > I know at least a few companies who are stuck with "no downtime" policy
> > and
> > > not enough expertise to do with kind of migration (which is really
> > > non-trivial).
> > >
> > > We can say "not our problem", but as we know, lack of good migration
> path
> > > really slows down adoption (Python 3.0, for instance).
> > >
> > > I'd love to at least get a feel of how many in the community will be
> > > impacted.
> > >
> > > Gwen
> > >
> > >
> > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Nov 10, 2017 at 10:46 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > Hi Gwen,
> > > > >
> > > > > A KIP has been proposed, but it is stalled:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+
> > > > > ZookeeperConsumerConnector+to+KafkaConsumer+Migration+and+Rollback
> > > > >
> > > > > Unless the interested parties pick that up, we would drop support
> > > > without a
> > > > > rolling upgrade path. Users would be able to use the old consumers
> > from
> > > > > 1.1.x for a long time. The old Scala clients don't support the
> > message
> > > > > format introduced in 0.11.0, so the feature set is pretty much
> frozen
> > > and
> > > > > there's little benefit in upgrading. But there is a cost in keeping
> > > them
> > > > in
> > > > > the codebase.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Nov 10, 2017 at 6:02 PM, Gwen Shapira 
> > > wrote:
> > > > >
> > > > > > Last time we tried deprecating the Scala consumer, there were
> > > concerns
> > > > > > about a lack of upgrade path. There is no rolling upgrade, and
> > > > migrating
> > > > > > offsets is not trivial (and not documented).
> > > > > >
> > > > > > Did anything change in that regard? Or are we planning on
> dropping
> > > > > support
> > > > > > without an upgrade path?
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 10, 2017 at 5:37 PM Guozhang Wang <
> wangg...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Ismael, the proposal looks good to me.
> > > > > > >
> > > > > > > A side note regarding: https://issues.apache.org/
> > > > > jira/browse/KAFKA-5637,
> > > > > > > could we resolve this ticket sooner than later to make clear
> > about
> > > > the
> > > > > > code
> > > > > > > deprecation and support duration when moving from 1.0.x to
> 2.0.x?
> > > > > > >
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma <
> ism...@juma.me.uk>
> > > > > wrote:
> > > > > > >
> > > > > > > > Features for 2.0.0 will be known after 1.1.0 is released in
> > > > February
> > > > > > > 2018.
> > > > > > > > We are still doing the usual time-based release process[1].
> > > > > > > >
> > > > > > > > I am raising this well ahead of time because of the potential
> > > > impact
> > > > > of
> > > > > > > > removing the old Scala clients (particularly the old
> high-level
> > > > > > consumer)
> > > > > > > > and dropping support for Java 7. Hopefully users can then
> plan
> > > > > > > accordingly.
> > > > > > > > We would do these changes in trunk soon after 1.1.0 is
> released
> > > > > (around
> > > > > > > > February).
> > > > > > > >
> > > > > > > > I think it makes sense to complete some of the work that was
> > not
> > > > > ready
> > > > > > in
> > > > > > > > time for 1.0.0 (Controller improvements and JBOD are two that
> > > come
> > > > to
> > > > > > > mind)
> > > > > > > > in 1.1.0 (January 2018) and combined with the desire to give
> > > > advance
> > > > > > > > notice, June 2018 was the logical choice.
> > > > > > > >
> > > > > > > > There is no plan to support a particular release for longer.
> > 1.x
> > > > > versus
> > > > > > > 2.x
> > > > > > > 

Re: [DISCUSS] Kafka 2.0.0 in June 2018

[Gwen Shapira]

Agree. I don't know that it actually matters. They can keep using whatever
they are using now since we don't plan on breaking the protocol.

But since the issue does keep coming up, I figured we'll need a clear
message around what the removal means and what users need to do.

Gwen


On Mon, Nov 20, 2017 at 8:21 AM Ismael Juma  wrote:

> It's worth emphasizing that the impact to such users is independent of
> whether we remove the old high-level consumer in 2.0.0 or not. They are
> unable to use the message format introduced in 0.11.0 or security features
> today.
>
> Ismael
>
> On Mon, Nov 20, 2017 at 4:11 PM, Gwen Shapira  wrote:
>
> > >
> > >
> > > Personally, I suspect that those who absolutely need a rolling
> migration
> > > and cannot handle a short period of downtime while doing a migration
> > > probably have in-house experts on Kafka who are familiar with the
> issues
> > > and willing to figure out a solution. The rest of the world can
> generally
> > > handle a short maintenance window.
> > >
> >
> > I really wish that was true :)
> > I know at least a few companies who are stuck with "no downtime" policy
> and
> > not enough expertise to do with kind of migration (which is really
> > non-trivial).
> >
> > We can say "not our problem", but as we know, lack of good migration path
> > really slows down adoption (Python 3.0, for instance).
> >
> > I'd love to at least get a feel of how many in the community will be
> > impacted.
> >
> > Gwen
> >
> >
> >
> > >
> > >
> > >
> > >
> > > On Fri, Nov 10, 2017 at 10:46 AM, Ismael Juma 
> wrote:
> > >
> > > > Hi Gwen,
> > > >
> > > > A KIP has been proposed, but it is stalled:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+
> > > > ZookeeperConsumerConnector+to+KafkaConsumer+Migration+and+Rollback
> > > >
> > > > Unless the interested parties pick that up, we would drop support
> > > without a
> > > > rolling upgrade path. Users would be able to use the old consumers
> from
> > > > 1.1.x for a long time. The old Scala clients don't support the
> message
> > > > format introduced in 0.11.0, so the feature set is pretty much frozen
> > and
> > > > there's little benefit in upgrading. But there is a cost in keeping
> > them
> > > in
> > > > the codebase.
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Nov 10, 2017 at 6:02 PM, Gwen Shapira 
> > wrote:
> > > >
> > > > > Last time we tried deprecating the Scala consumer, there were
> > concerns
> > > > > about a lack of upgrade path. There is no rolling upgrade, and
> > > migrating
> > > > > offsets is not trivial (and not documented).
> > > > >
> > > > > Did anything change in that regard? Or are we planning on dropping
> > > > support
> > > > > without an upgrade path?
> > > > >
> > > > >
> > > > > On Fri, Nov 10, 2017 at 5:37 PM Guozhang Wang 
> > > > wrote:
> > > > >
> > > > > > Thanks Ismael, the proposal looks good to me.
> > > > > >
> > > > > > A side note regarding: https://issues.apache.org/
> > > > jira/browse/KAFKA-5637,
> > > > > > could we resolve this ticket sooner than later to make clear
> about
> > > the
> > > > > code
> > > > > > deprecation and support duration when moving from 1.0.x to 2.0.x?
> > > > > >
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma 
> > > > wrote:
> > > > > >
> > > > > > > Features for 2.0.0 will be known after 1.1.0 is released in
> > > February
> > > > > > 2018.
> > > > > > > We are still doing the usual time-based release process[1].
> > > > > > >
> > > > > > > I am raising this well ahead of time because of the potential
> > > impact
> > > > of
> > > > > > > removing the old Scala clients (particularly the old high-level
> > > > > consumer)
> > > > > > > and dropping support for Java 7. Hopefully users can then plan
> > > > > > accordingly.
> > > > > > > We would do these changes in trunk soon after 1.1.0 is released
> > > > (around
> > > > > > > February).
> > > > > > >
> > > > > > > I think it makes sense to complete some of the work that was
> not
> > > > ready
> > > > > in
> > > > > > > time for 1.0.0 (Controller improvements and JBOD are two that
> > come
> > > to
> > > > > > mind)
> > > > > > > in 1.1.0 (January 2018) and combined with the desire to give
> > > advance
> > > > > > > notice, June 2018 was the logical choice.
> > > > > > >
> > > > > > > There is no plan to support a particular release for longer.
> 1.x
> > > > versus
> > > > > > 2.x
> > > > > > > is no different than 0.10.x versus 0.11.x from the perspective
> of
> > > > > > > supporting older releases.
> > > > > > >
> > > > > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+
> > > > > > > Based+Release+Plan
> > > > > > >
> > > > > > > On Fri, Nov 10, 2017 at 11:21 AM, Jaikiran Pai <
> > > > > jai.forums2...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > 

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Ted Yu]

bq. (e.g. increase network/IO threads)

Would decreasing network/IO threads be supported ?

bq. void reconfigure(Map configs);

What if some keys in configs are not in the Set returned by
reconfigurableConfigs()
? Would exception be thrown ?
If so, please specify which exception would be thrown.

The sample commands for bin/kafka-configs include '--add-config'.
Would there be '--remove-config' ?

bq. Existing connections will not be affected, new connections will use the
new keystore.

ssl.keystore.password appears a few lines above. Would there be any issue
with mixture of connections (with old and new password) ?


Cheers



On Mon, Nov 20, 2017 at 5:57 AM, Rajini Sivaram 
wrote:

> Hi all,
>
> I have submitted KIP-226 to enable dynamic reconfiguration of brokers
> without restart:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 226+-+Dynamic+Broker+Configuration
>
> The KIP proposes to extend the current dynamic replication quota
> configuration for brokers to support dynamic reconfiguration of a limited
> set of configuration options that are typically updated during the lifetime
> of a broker.
>
> Feedback and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
>

Re: [DISCUSS] Kafka 2.0.0 in June 2018

[Ismael Juma]

It's worth emphasizing that the impact to such users is independent of
whether we remove the old high-level consumer in 2.0.0 or not. They are
unable to use the message format introduced in 0.11.0 or security features
today.

Ismael

On Mon, Nov 20, 2017 at 4:11 PM, Gwen Shapira  wrote:

> >
> >
> > Personally, I suspect that those who absolutely need a rolling migration
> > and cannot handle a short period of downtime while doing a migration
> > probably have in-house experts on Kafka who are familiar with the issues
> > and willing to figure out a solution. The rest of the world can generally
> > handle a short maintenance window.
> >
>
> I really wish that was true :)
> I know at least a few companies who are stuck with "no downtime" policy and
> not enough expertise to do with kind of migration (which is really
> non-trivial).
>
> We can say "not our problem", but as we know, lack of good migration path
> really slows down adoption (Python 3.0, for instance).
>
> I'd love to at least get a feel of how many in the community will be
> impacted.
>
> Gwen
>
>
>
> >
> >
> >
> >
> > On Fri, Nov 10, 2017 at 10:46 AM, Ismael Juma  wrote:
> >
> > > Hi Gwen,
> > >
> > > A KIP has been proposed, but it is stalled:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+
> > > ZookeeperConsumerConnector+to+KafkaConsumer+Migration+and+Rollback
> > >
> > > Unless the interested parties pick that up, we would drop support
> > without a
> > > rolling upgrade path. Users would be able to use the old consumers from
> > > 1.1.x for a long time. The old Scala clients don't support the message
> > > format introduced in 0.11.0, so the feature set is pretty much frozen
> and
> > > there's little benefit in upgrading. But there is a cost in keeping
> them
> > in
> > > the codebase.
> > >
> > > Ismael
> > >
> > > On Fri, Nov 10, 2017 at 6:02 PM, Gwen Shapira 
> wrote:
> > >
> > > > Last time we tried deprecating the Scala consumer, there were
> concerns
> > > > about a lack of upgrade path. There is no rolling upgrade, and
> > migrating
> > > > offsets is not trivial (and not documented).
> > > >
> > > > Did anything change in that regard? Or are we planning on dropping
> > > support
> > > > without an upgrade path?
> > > >
> > > >
> > > > On Fri, Nov 10, 2017 at 5:37 PM Guozhang Wang 
> > > wrote:
> > > >
> > > > > Thanks Ismael, the proposal looks good to me.
> > > > >
> > > > > A side note regarding: https://issues.apache.org/
> > > jira/browse/KAFKA-5637,
> > > > > could we resolve this ticket sooner than later to make clear about
> > the
> > > > code
> > > > > deprecation and support duration when moving from 1.0.x to 2.0.x?
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Features for 2.0.0 will be known after 1.1.0 is released in
> > February
> > > > > 2018.
> > > > > > We are still doing the usual time-based release process[1].
> > > > > >
> > > > > > I am raising this well ahead of time because of the potential
> > impact
> > > of
> > > > > > removing the old Scala clients (particularly the old high-level
> > > > consumer)
> > > > > > and dropping support for Java 7. Hopefully users can then plan
> > > > > accordingly.
> > > > > > We would do these changes in trunk soon after 1.1.0 is released
> > > (around
> > > > > > February).
> > > > > >
> > > > > > I think it makes sense to complete some of the work that was not
> > > ready
> > > > in
> > > > > > time for 1.0.0 (Controller improvements and JBOD are two that
> come
> > to
> > > > > mind)
> > > > > > in 1.1.0 (January 2018) and combined with the desire to give
> > advance
> > > > > > notice, June 2018 was the logical choice.
> > > > > >
> > > > > > There is no plan to support a particular release for longer. 1.x
> > > versus
> > > > > 2.x
> > > > > > is no different than 0.10.x versus 0.11.x from the perspective of
> > > > > > supporting older releases.
> > > > > >
> > > > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+
> > > > > > Based+Release+Plan
> > > > > >
> > > > > > On Fri, Nov 10, 2017 at 11:21 AM, Jaikiran Pai <
> > > > jai.forums2...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Ismael,
> > > > > > >
> > > > > > > Are there any new features other than the language specific
> > changes
> > > > > that
> > > > > > > are being planned for 2.0.0? Also, when 2.x gets released, will
> > the
> > > > 1.x
> > > > > > > series see continued bug fixes and releases in the community or
> > is
> > > > the
> > > > > > plan
> > > > > > > to have one single main version that gets continuous updates
> and
> > > > > > releases?
> > > > > > >
> > > > > > > By the way, why June 2018? :)
> > > > > > >
> > > > > > > -Jaikiran
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 09/11/17 3:14 PM, Ismael Juma wrote:
> > > 

Re: [DISCUSS] Kafka 2.0.0 in June 2018

[Gwen Shapira]

>
>
> Personally, I suspect that those who absolutely need a rolling migration
> and cannot handle a short period of downtime while doing a migration
> probably have in-house experts on Kafka who are familiar with the issues
> and willing to figure out a solution. The rest of the world can generally
> handle a short maintenance window.
>

I really wish that was true :)
I know at least a few companies who are stuck with "no downtime" policy and
not enough expertise to do with kind of migration (which is really
non-trivial).

We can say "not our problem", but as we know, lack of good migration path
really slows down adoption (Python 3.0, for instance).

I'd love to at least get a feel of how many in the community will be
impacted.

Gwen



>
>
>
>
> On Fri, Nov 10, 2017 at 10:46 AM, Ismael Juma  wrote:
>
> > Hi Gwen,
> >
> > A KIP has been proposed, but it is stalled:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+
> > ZookeeperConsumerConnector+to+KafkaConsumer+Migration+and+Rollback
> >
> > Unless the interested parties pick that up, we would drop support
> without a
> > rolling upgrade path. Users would be able to use the old consumers from
> > 1.1.x for a long time. The old Scala clients don't support the message
> > format introduced in 0.11.0, so the feature set is pretty much frozen and
> > there's little benefit in upgrading. But there is a cost in keeping them
> in
> > the codebase.
> >
> > Ismael
> >
> > On Fri, Nov 10, 2017 at 6:02 PM, Gwen Shapira  wrote:
> >
> > > Last time we tried deprecating the Scala consumer, there were concerns
> > > about a lack of upgrade path. There is no rolling upgrade, and
> migrating
> > > offsets is not trivial (and not documented).
> > >
> > > Did anything change in that regard? Or are we planning on dropping
> > support
> > > without an upgrade path?
> > >
> > >
> > > On Fri, Nov 10, 2017 at 5:37 PM Guozhang Wang 
> > wrote:
> > >
> > > > Thanks Ismael, the proposal looks good to me.
> > > >
> > > > A side note regarding: https://issues.apache.org/
> > jira/browse/KAFKA-5637,
> > > > could we resolve this ticket sooner than later to make clear about
> the
> > > code
> > > > deprecation and support duration when moving from 1.0.x to 2.0.x?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > Features for 2.0.0 will be known after 1.1.0 is released in
> February
> > > > 2018.
> > > > > We are still doing the usual time-based release process[1].
> > > > >
> > > > > I am raising this well ahead of time because of the potential
> impact
> > of
> > > > > removing the old Scala clients (particularly the old high-level
> > > consumer)
> > > > > and dropping support for Java 7. Hopefully users can then plan
> > > > accordingly.
> > > > > We would do these changes in trunk soon after 1.1.0 is released
> > (around
> > > > > February).
> > > > >
> > > > > I think it makes sense to complete some of the work that was not
> > ready
> > > in
> > > > > time for 1.0.0 (Controller improvements and JBOD are two that come
> to
> > > > mind)
> > > > > in 1.1.0 (January 2018) and combined with the desire to give
> advance
> > > > > notice, June 2018 was the logical choice.
> > > > >
> > > > > There is no plan to support a particular release for longer. 1.x
> > versus
> > > > 2.x
> > > > > is no different than 0.10.x versus 0.11.x from the perspective of
> > > > > supporting older releases.
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+
> > > > > Based+Release+Plan
> > > > >
> > > > > On Fri, Nov 10, 2017 at 11:21 AM, Jaikiran Pai <
> > > jai.forums2...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Ismael,
> > > > > >
> > > > > > Are there any new features other than the language specific
> changes
> > > > that
> > > > > > are being planned for 2.0.0? Also, when 2.x gets released, will
> the
> > > 1.x
> > > > > > series see continued bug fixes and releases in the community or
> is
> > > the
> > > > > plan
> > > > > > to have one single main version that gets continuous updates and
> > > > > releases?
> > > > > >
> > > > > > By the way, why June 2018? :)
> > > > > >
> > > > > > -Jaikiran
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 09/11/17 3:14 PM, Ismael Juma wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I'm starting this discussion early because of the potential
> > impact.
> > > > > >>
> > > > > >> Kafka 1.0.0 was just released and the focus was on achieving the
> > > > > original
> > > > > >> project vision in terms of features provided while maintaining
> > > > > >> compatibility for the most part (i.e. we did not remove
> deprecated
> > > > > >> components like the Scala clients).
> > > > > >>
> > > > > >> This was the right decision, in my opinion, but it's time to
> start
> > > > > >> thinking
> > > > > >> about 2.0.0, which is 

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Luca Del Grosso]

Hi, can you help me?
i have a problem, i would like to create an avro schema on schema registry
of kafka that references a type that is declared in a different avro
schema, the places under an example:

First avro schema with reference UserAction

{"namespace": "com.myorg.other",
 "type": "record",
 "name": "SearchSuggest",
 "fields": [
 {"name": "name", "type": "string"},
 {"name": "userAction", "type": "UserAction"}
 ]
}

second avro schema with enum:

{"namespace": "com.myorg.other",
 "type": "enum",
 "name": "UserAction",
 "symbols": ["S", "V", "C"]
}

This work in my maven project, but when i try create this on schema
registry, it's invalid.


2017-11-20 14:57 GMT+01:00 Rajini Sivaram :

> Hi all,
>
> I have submitted KIP-226 to enable dynamic reconfiguration of brokers
> without restart:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 226+-+Dynamic+Broker+Configuration
>
> The KIP proposes to extend the current dynamic replication quota
> configuration for brokers to support dynamic reconfiguration of a limited
> set of configuration options that are typically updated during the lifetime
> of a broker.
>
> Feedback and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
>

[DISCUSS] KIP 226 - Dynamic Broker Configuration

[Rajini Sivaram]

Hi all,

I have submitted KIP-226 to enable dynamic reconfiguration of brokers
without restart:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration

The KIP proposes to extend the current dynamic replication quota
configuration for brokers to support dynamic reconfiguration of a limited
set of configuration options that are typically updated during the lifetime
of a broker.

Feedback and suggestions are welcome.

Thank you...

Regards,

Rajini

[jira] [Created] (KAFKA-6246) Enable reconfiguration of listeners and security configs

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6246:
-

 Summary: Enable  reconfiguration of listeners and security configs
 Key: KAFKA-6246
 URL: https://issues.apache.org/jira/browse/KAFKA-6246
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.1.0


See 
[KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration]
 for details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6244) Enable dynamic reconfiguration of log cleaners

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6244:
-

 Summary: Enable dynamic reconfiguration of log cleaners
 Key: KAFKA-6244
 URL: https://issues.apache.org/jira/browse/KAFKA-6244
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.1.0


See 
[KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration]
 for details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6245) Enable reconfiguration of default topic configs used by brokers

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6245:
-

 Summary: Enable reconfiguration of default topic configs used by 
brokers
 Key: KAFKA-6245
 URL: https://issues.apache.org/jira/browse/KAFKA-6245
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.1.0


See 
[KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration]
 for details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6243) Enable reconfiguration of metrics reporters and their custom configs

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6243:
-

 Summary: Enable reconfiguration of metrics reporters and their 
custom configs
 Key: KAFKA-6243
 URL: https://issues.apache.org/jira/browse/KAFKA-6243
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.1.0


See 
[KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration]
 for details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6242) Enable resizing various broker thread pools

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6242:
-

 Summary: Enable resizing various broker thread pools
 Key: KAFKA-6242
 URL: https://issues.apache.org/jira/browse/KAFKA-6242
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.1.0


See 
[KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration]
 for details



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6241) Enable dynamic reconfiguration of SSL keystores

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6241:
-

 Summary: Enable dynamic reconfiguration of SSL keystores
 Key: KAFKA-6241
 URL: https://issues.apache.org/jira/browse/KAFKA-6241
 Project: Kafka
  Issue Type: Sub-task
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.1.0


This will include the base implementation to enable dynamic updates of broker 
configs.
SSL keystore update will be implemented as part of this task to enable testing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6240) Support dynamic updates of frequently updated broker configs

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6240:
-

 Summary: Support dynamic updates of frequently updated broker 
configs
 Key: KAFKA-6240
 URL: https://issues.apache.org/jira/browse/KAFKA-6240
 Project: Kafka
  Issue Type: New Feature
  Components: core
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 1.1.0


See 
[KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration]
 for details.

Implementation will be done under sub-tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6239) Consume group hung into rebalancing state, now stream not able to poll data

[DHRUV BANSAL (JIRA)]

DHRUV BANSAL created KAFKA-6239:
---

 Summary: Consume group hung into rebalancing state, now stream not 
able to poll data
 Key: KAFKA-6239
 URL: https://issues.apache.org/jira/browse/KAFKA-6239
 Project: Kafka
  Issue Type: Bug
Reporter: DHRUV BANSAL
Priority: Critical


./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group 
mitra-log-parser --describe

Note: This will only show information about consumers that use the Java 
consumer API (non-ZooKeeper-based consumers).

Warning: Consumer group 'mitra-log-parser' is rebalancing.

How to restore the consumer group state?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Kafka 2.0.0 in June 2018

[Ismael Juma]

Thanks for the update Onur. Are you and the other committers and
contributors from LinkedIn planning to push this over the line?

Ismael

On Fri, Nov 10, 2017 at 9:53 PM, Onur Karaman 
wrote:

> Hey everyone. Regarding the status of KIP-125, just a heads up: I have an
> implementation of KIP-125 (KAFKA-4513) here:
> https://github.com/onurkaraman/kafka/commit/3b5448006ab70ba2b0b5e177853d19
> 1d0f777452
>
> The code might need to be rebased. The steps described in the KIP are a bit
> involved. Other than that, the implementation might have a bug with respect
> to converting arbitrary blacklist regexes to whitelist regexes since the
> new consumer only accepts whitelists.
>
> On Fri, Nov 10, 2017 at 11:36 AM, Jeff Widman  wrote:
>
> > Re: migrating offsets for old Scala consumers.
> >
> > I work in the python world, so haven't directly used the old high level
> > consumer, but from what I understand the underlying problem remains the
> > migration of zookeeper offsets to the __consumer_offsets topic.
> >
> > We've used a slightly modified version of Grant Henke's script for
> > migrating offsets here: https://github.com/apache/kafka/pull/2615
> > It doesn't support rolling upgrades, but other than that it's great...
> I've
> > used it for multiple migrations, and very thankful for the time Grant put
> > into it.
> >
> > I don't know that it's worth pulling this into core, it might be, it
> might
> > not be. But it probably is worth documenting the procedure at least
> > somewhere.
> >
> > Personally, I suspect that those who absolutely need a rolling migration
> > and cannot handle a short period of downtime while doing a migration
> > probably have in-house experts on Kafka who are familiar with the issues
> > and willing to figure out a solution. The rest of the world can generally
> > handle a short maintenance window.
> >
> >
> >
> >
> > On Fri, Nov 10, 2017 at 10:46 AM, Ismael Juma  wrote:
> >
> > > Hi Gwen,
> > >
> > > A KIP has been proposed, but it is stalled:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+
> > > ZookeeperConsumerConnector+to+KafkaConsumer+Migration+and+Rollback
> > >
> > > Unless the interested parties pick that up, we would drop support
> > without a
> > > rolling upgrade path. Users would be able to use the old consumers from
> > > 1.1.x for a long time. The old Scala clients don't support the message
> > > format introduced in 0.11.0, so the feature set is pretty much frozen
> and
> > > there's little benefit in upgrading. But there is a cost in keeping
> them
> > in
> > > the codebase.
> > >
> > > Ismael
> > >
> > > On Fri, Nov 10, 2017 at 6:02 PM, Gwen Shapira 
> wrote:
> > >
> > > > Last time we tried deprecating the Scala consumer, there were
> concerns
> > > > about a lack of upgrade path. There is no rolling upgrade, and
> > migrating
> > > > offsets is not trivial (and not documented).
> > > >
> > > > Did anything change in that regard? Or are we planning on dropping
> > > support
> > > > without an upgrade path?
> > > >
> > > >
> > > > On Fri, Nov 10, 2017 at 5:37 PM Guozhang Wang 
> > > wrote:
> > > >
> > > > > Thanks Ismael, the proposal looks good to me.
> > > > >
> > > > > A side note regarding: https://issues.apache.org/
> > > jira/browse/KAFKA-5637,
> > > > > could we resolve this ticket sooner than later to make clear about
> > the
> > > > code
> > > > > deprecation and support duration when moving from 1.0.x to 2.0.x?
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Features for 2.0.0 will be known after 1.1.0 is released in
> > February
> > > > > 2018.
> > > > > > We are still doing the usual time-based release process[1].
> > > > > >
> > > > > > I am raising this well ahead of time because of the potential
> > impact
> > > of
> > > > > > removing the old Scala clients (particularly the old high-level
> > > > consumer)
> > > > > > and dropping support for Java 7. Hopefully users can then plan
> > > > > accordingly.
> > > > > > We would do these changes in trunk soon after 1.1.0 is released
> > > (around
> > > > > > February).
> > > > > >
> > > > > > I think it makes sense to complete some of the work that was not
> > > ready
> > > > in
> > > > > > time for 1.0.0 (Controller improvements and JBOD are two that
> come
> > to
> > > > > mind)
> > > > > > in 1.1.0 (January 2018) and combined with the desire to give
> > advance
> > > > > > notice, June 2018 was the logical choice.
> > > > > >
> > > > > > There is no plan to support a particular release for longer. 1.x
> > > versus
> > > > > 2.x
> > > > > > is no different than 0.10.x versus 0.11.x from the perspective of
> > > > > > supporting older releases.
> > > > > >
> > > > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+
> > > > > > 

Re: [DISCUSS] KIP-225 - Use tags for consumer “records.lag” metrics

[charly molter]

Hi,

Any dev could comment on this? I'd quite like to launch the vote for this
soon.

Thanks!

On Fri, Nov 17, 2017 at 6:40 AM, James Cheng  wrote:

> Ah, that's a great point. KIP-153 didn't *rename* the metric but changed
> its meaning, yet we didn't seem to discuss compatibility much when we made
> that change.
>
> If the Kafka devs can comment on the backwards-compatibility-ness of
> metrics and how we treat that, that would be helpful.
>
> -James
>
> > On Nov 16, 2017, at 2:06 AM, charly molter 
> wrote:
> >
> > Yes James you are right.
> > I wasn't sure what to do about it and followed what happened with
> BytesOut
> > in KIP-153 which completely changed meaning without any deprecation
> window.
> > I'm happy to adapt my KIP if the community thinks we should duplicate the
> > metric for a while.
> >
> > Thanks!
> >
> > On Thu, Nov 16, 2017 at 8:13 AM, James Cheng 
> wrote:
> >
> >> This KIP will break backwards compatibility for anyone who is using the
> >> existing attribute names.
> >>
> >> Kafka devs, I believe that metrics are a supported interface, and so
> this
> >> would be a breaking change. In order to do this, we would need a
> >> deprecation timeframe for the old metric, and a transition plan to the
> new
> >> name. Is that right? I'm not sure how we deprecate metrics...
> >>
> >> During the deprecation timeframe, we could duplicate the metric to the
> new
> >> name.
> >>
> >> -James
> >>
> >> On Nov 13, 2017, at 6:09 AM, charly molter 
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> There doesn't seem to be much opposition to this KIP, I'll leave a
> couple
> >>> more days before starting the vote.
> >>>
> >>> Thanks!
> >>>
> >>> On Thu, Nov 9, 2017 at 1:59 PM, charly molter  >
> >>> wrote:
> >>>
>  Hi,
> 
>  I'd like to start the discussion on KIP-225.
> 
>  This KIP tries to correct the way the consumer lag metrics are
> reported
> >> to
>  use built in tags from MetricName.
> 
>  Here's the link:
>  https://cwiki.apache.org/confluence/pages/viewpage.
> >> action?pageId=74686649
> 
>  Thanks!
>  --
>  Charly Molter
> 
> >>>
> >>>
> >>>
> >>> --
> >>> Charly Molter
> >>
> >>
> >
> >
> > --
> > Charly Molter
>
>


-- 
Charly Molter

Re: [jira] [Created] (KAFKA-6238) Issues with protocol version when applying a rolling upgrade to 1.0.0

[Luca Del Grosso]

what should I do?


2017-11-20 12:06 GMT+01:00 Diego Louzán (JIRA) :

> Diego Louzán created KAFKA-6238:
> ---
>
>  Summary: Issues with protocol version when applying a rolling
> upgrade to 1.0.0
>  Key: KAFKA-6238
>  URL: https://issues.apache.org/jira/browse/KAFKA-6238
>  Project: Kafka
>   Issue Type: Bug
>   Components: documentation
> Affects Versions: 1.0.0
> Reporter: Diego Louzán
>
>
> Hello,
>
> I am trying to perform a rolling upgrade from 0.10.0.1 to 1.0.0, and
> according to the instructions in the documentation, I should only have to
> upgrade the "inter.broker.protocol.version" parameter in the first step.
> But after setting the value to "0.10.0" or "0.10.0.1" (tried both), the
> broker refuses to start with the following error:
>
> {code}
> [2017-11-20 08:28:46,620] FATAL  (kafka.Kafka$)
> java.lang.IllegalArgumentException: requirement failed:
> log.message.format.version 1.0-IV0 cannot be used when
> inter.broker.protocol.version is set to 0.10.0.1
> at scala.Predef$.require(Predef.scala:224)
> at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1205)
> at kafka.server.KafkaConfig.(KafkaConfig.scala:1170)
> at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:881)
> at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:878)
> at kafka.server.KafkaServerStartable$.fromProps(
> KafkaServerStartable.scala:28)
> at kafka.Kafka$.main(Kafka.scala:82)
> at kafka.Kafka.main(Kafka.scala)
> {code}
>
> I checked the instructions for rolling upgrades to previous versions
> (namely 0.11.0.0), and in here it's stated that is also needed to upgrade
> the "log.message.format.version" parameter in two stages. I have tried that
> and the upgrade worked. It seems it still applies to version 1.0.0, so I'm
> not sure if this is wrong documentation, or an actual issue with kafka
> since it should work as stated in the docs.
>
> Regards,
> Diego Louzán
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
>

[jira] [Created] (KAFKA-6238) Issues with protocol version when applying a rolling upgrade to 1.0.0

[JIRA]

Diego Louzán created KAFKA-6238:
---

 Summary: Issues with protocol version when applying a rolling 
upgrade to 1.0.0
 Key: KAFKA-6238
 URL: https://issues.apache.org/jira/browse/KAFKA-6238
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Affects Versions: 1.0.0
Reporter: Diego Louzán


Hello,

I am trying to perform a rolling upgrade from 0.10.0.1 to 1.0.0, and according 
to the instructions in the documentation, I should only have to upgrade the 
"inter.broker.protocol.version" parameter in the first step. But after setting 
the value to "0.10.0" or "0.10.0.1" (tried both), the broker refuses to start 
with the following error:

{code}
[2017-11-20 08:28:46,620] FATAL  (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: 
log.message.format.version 1.0-IV0 cannot be used when 
inter.broker.protocol.version is set to 0.10.0.1
at scala.Predef$.require(Predef.scala:224)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1205)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1170)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:881)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:878)
at 
kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
{code}

I checked the instructions for rolling upgrades to previous versions (namely 
0.11.0.0), and in here it's stated that is also needed to upgrade the 
"log.message.format.version" parameter in two stages. I have tried that and the 
upgrade worked. It seems it still applies to version 1.0.0, so I'm not sure if 
this is wrong documentation, or an actual issue with kafka since it should work 
as stated in the docs.

Regards,
Diego Louzán



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

kafka schema registry problem

[Luca Del Grosso]

Hi
i have a problem, i would like to create an avro schema on schema registry
of kafka that references a type that is declared in a different avro
schema, the places under an example:

First avro schema with reference UserAction

{"namespace": "com.myorg.other",
 "type": "record",
 "name": "SearchSuggest",
 "fields": [
 {"name": "name", "type": "string"},
 {"name": "userAction", "type": "UserAction"}
 ]
}

second avro schema with enum:

{"namespace": "com.myorg.other",
 "type": "enum",
 "name": "UserAction",
 "symbols": ["S", "V", "C"]
}

This work in my maven project, but when i try create this on schema
registry, it's invalid.

Re: SessionKeySchema#segmentsToSearch()

[Damian Guy]

w.r.t `SessionKeySchema` MAX_VALUE is correct. This is because we use the
end time (rather than the start time) of the session to define which
segment the session is in. So it could be in any segment starting from
`from` time.

On Sun, 19 Nov 2017 at 20:27 Ted Yu  wrote:

> For `getMinSegmentGreaterThanEqualToTimestamp` , the email was indeed
> meant for #4162.
>
> Pardon.
>
> On Sun, Nov 19, 2017 at 11:55 AM, Guozhang Wang 
> wrote:
>
>> For `SessionKeySchema#segmentsToSearch`: for session store, multiple
>> sessions may merge together when receiving late arrived records. When I
>> looked at the code, it seems that we have merged the sessions during
>> aggregations to effectively move the sessions between segments. So I'm not
>> 100% certain why we still need to enforce MAX_VALUE. @Damian?
>>
>> For `getMinSegmentGreaterThanEqualToTimestamp` and `
>> getMaxSegmentLessThanEqualToTimestamp`: I think you meant to leave it as a
>> comment on https://github.com/apache/kafka/pull/4162? This is only added
>> in
>> that PR.
>>
>>
>> Guozhang
>>
>>
>> On Sat, Nov 18, 2017 at 11:16 AM, Ted Yu  wrote:
>> >
>> > This code:
>> >
>> > final Segment minSegment = segments
>> > .getMinSegmentGreaterThanEqualToTimestamp(timeFrom);
>> >
>> > final Segment maxSegment = segments
>> > .getMaxSegmentLessThanEqualToTimestamp(timeTo);
>> >
>> > Can be replaced with:
>> >
>> > final List searchSpace = keySchema.segmentsToSearch(
>> > segments, from, to);
>> >
>> > The minSegment would be first in List and maxSegment would be last in
>> List.
>> >
>> > On Sat, Nov 18, 2017 at 11:09 AM, Ted Yu  wrote:
>> >
>> > > Hi,
>> > > I was reading code for SessionKeySchema#segmentsToSearch() where:
>> > >
>> > > public List segmentsToSearch(final Segments segments,
>> final
>> > > long from, final long to) {
>> > > return segments.segments(from, Long.MAX_VALUE);
>> > >
>> > > I wonder why the parameter to is ignored.
>> > > WindowKeySchema#segmentsToSearch() passes parameter to
>> > > to segments.segments().
>> > >
>> > > Cheers
>> > >
>>
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>

[jira] [Created] (KAFKA-6237) stream stopped working after exception: Cannot execute transactional method because we are in an error state

[DHRUV BANSAL (JIRA)]

DHRUV BANSAL created KAFKA-6237:
---

 Summary: stream stopped working after exception: Cannot execute 
transactional method because we are in an error state
 Key: KAFKA-6237
 URL: https://issues.apache.org/jira/browse/KAFKA-6237
 Project: Kafka
  Issue Type: Bug
Reporter: DHRUV BANSAL
Priority: Critical


017-11-19 07:52:44,673 
[project_logs_stream-a30ea242-3c9f-46a9-a01c-51903bd40ca5-StreamThread-1] 
ERROR: org.apache.kafka.streams.processor.internals.StreamThread - 
stream-thread 
[orion_logs_stream-a30ea242-3c9f-46a9-a01c-51903bd40ca5-StreamThread-1] Failed 
while closing StreamTask 0_1:
org.apache.kafka.common.KafkaException: Cannot execute transactional method 
because we are in an error state
at 
org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:524)
at 
org.apache.kafka.clients.producer.internals.TransactionManager.beginAbort(TransactionManager.java:198)
at 
org.apache.kafka.clients.producer.KafkaProducer.abortTransaction(KafkaProducer.java:598)
at 
org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:434)
at 
org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:1086)
at 
org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:1041)
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:538)
Caused by: org.apache.kafka.common.KafkaException: Unexpected error in 
AddOffsetsToTxnResponse: The server experienced an unexpected error when 
processing the request
at 
org.apache.kafka.clients.producer.internals.TransactionManager$AddOffsetsToTxnHandler.handleResponse(TransactionManager.java:978)
at 
org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:648)
at 
org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
at 
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:454)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:446)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:206)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

[Sönke Liebau]

@Randall: are you happy with the KIP as it stands so I can call for a vote,
or are there any outstanding items still to discuss?

Same question to anybody else who'd like to participate of course :)

On Thu, Nov 16, 2017 at 5:35 PM, Sönke Liebau 
wrote:

> Sounds good. I've added a few sentences to this effect to the KIP.
>
> On Thu, Nov 16, 2017 at 5:02 PM, Randall Hauch  wrote:
>
>> Nice job updating the KIP. The PR (
>> https://github.com/apache/kafka/pull/2755/files) for the proposed
>> implementation does prevent names from being empty, and it trims
>> whitespace
>> from the name only when creating a new connector. However, the KIP's
>> "Proposed Change" section should probably be very clear about this, and
>> the
>> migration section should address how a connector that was created with
>> leading and/or trailing whitespace characters will still be able to be
>> updated and deleted. I think that decreases the likelihood of this change
>> negatively impacting existing users. Basically, going forward, the names
>> of
>> new connectors will be trimmed.
>>
>> WDYT?
>>
>> On Thu, Nov 16, 2017 at 9:32 AM, Sönke Liebau <
>> soenke.lie...@opencore.com.invalid> wrote:
>>
>> > I've added some more detail to the KIP [1] around current scenarios that
>> > might break in the future. I actually came up with a second limitation
>> that
>> > we'd impose on users and also documented this.
>> >
>> > Let me know what you think.
>> >
>> > Kind regards,
>> > Sönke
>> >
>> > [1]
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 212%3A+Enforce+set+of+legal+characters+for+connector+names
>> >
>> >
>> > On Thu, Nov 16, 2017 at 2:59 PM, Sönke Liebau <
>> soenke.lie...@opencore.com>
>> > wrote:
>> >
>> > > Hi Randall,
>> > >
>> > > I had mentioned this edge case in the KIP, but will add some further
>> > > detail to further clarify all changing scenarios post pull request.
>> > >
>> > > Kind regards,
>> > > Sönke
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Nov 16, 2017 at 2:06 PM, Randall Hauch 
>> wrote:
>> > >
>> > >> No, we need to keep the KIP since we want to change/correct the
>> existing
>> > >> behavior. But we do need to clarify in the KIP these edge cases that
>> > will
>> > >> change.
>> > >>
>> > >> Thanks for the continued work on this, Sönke.
>> > >>
>> > >> Regards,
>> > >>
>> > >> Randall
>> > >>
>> > >> > On Nov 16, 2017, at 1:56 AM, Sönke Liebau <
>> soenke.lie...@opencore.com
>> > >> .INVALID> wrote:
>> > >> >
>> > >> > Hi Randall,
>> > >> >
>> > >> > zero length definitely works, that's what sent me down this hole in
>> > the
>> > >> > first place. I had a customer accidentally create a connector
>> without
>> > a
>> > >> > name in his environment and then be unable to delete it. No
>> connector
>> > >> name
>> > >> > doesn't work, as this throws a null pointer exception due to
>> > KAFKA-4938
>> > >> ,
>> > >> > but once that is fixed would create a connector named "null" I
>> think.
>> > >> Have
>> > >> > not retested this, but seen it in the past.
>> > >> >
>> > >> > Also, it is possible to create connectors with trailing and leading
>> > >> > whitespaces, this errors out on the create request (which will be
>> > fixed
>> > >> > when KAFKA-4827 is merged), but correctly creates the connector and
>> > you
>> > >> can
>> > >> > access it if you percent-escape the curl call. This for me is the
>> main
>> > >> > reason why a KIP might be needed, as we are changing public facing
>> > >> behavior
>> > >> > here. I agree with you, that this will probably not affect anyone
>> or
>> > >> hardly
>> > >> > anyone, but in principle it is a change that should need a KIP I
>> > think.
>> > >> >
>> > >> > I've retested and documented this for Confluent 3.3.0:
>> > >> > https://gist.github.com/soenkeliebau/9363745cff23560fcc234d9
>> b64ac14c4
>> > >> >
>> > >> > I am of course happy to withdraw the KIP if you think it is
>> > unnecessary,
>> > >> > I've also updated the pull request for KAFKA-4930 to reflect the
>> > changes
>> > >> > stated in the KIP and tested the code with Arjuns pull request for
>> > >> > KAFKA-4827 to ensure they don't interfere with each other.
>> > >> >
>> > >> > Let me know what you think.
>> > >> >
>> > >> > Kind regards,
>> > >> > Sönke
>> > >> >
>> > >> > ᐧ
>> > >> >
>> > >> >> On Tue, Nov 14, 2017 at 7:03 PM, Randall Hauch 
>> > >> wrote:
>> > >> >>
>> > >> >> Thanks for updating the KIP to reflect the current process.
>> However,
>> > I
>> > >> >> still question whether it is necessary to have a KIP - it depends
>> on
>> > >> >> whether it was possible with prior versions to have connectors
>> with
>> > >> >> zero-length or blank names. Have you tried both of these cases?
>> > >> >>
>> > >> >> On Fri, Nov 10, 2017 at 3:52 AM, Sönke Liebau <
>> > >> >> soenke.lie...@opencore.com.invalid> wrote:
>> > >> >>
>> > >> >>> Hi Randall,
>> > >> >>>
>> > >> >>> I have set aside some time to work on 

[jira] [Resolved] (KAFKA-1130) "log.dirs" is a confusing property name

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-1130.
-
Resolution: Won't Fix

Due to the age of the last comment on this issue and the fact that there has 
been not a lot of discussion around the naming of this parameter in the recent 
past I believe we can close this issue.

> "log.dirs" is a confusing property name
> ---
>
> Key: KAFKA-1130
> URL: https://issues.apache.org/jira/browse/KAFKA-1130
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.0
>Reporter: David Arthur
>Priority: Minor
> Attachments: KAFKA-1130.diff
>
>
> "log.dirs" is a somewhat misleading config name. The term "log" comes from an 
> internal Kafka class name, and shouldn't leak out into the public API (in 
> this case, the config).
> Something like "data.dirs" would be less confusing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)