Build failed in Jenkins: kafka-trunk-jdk8 #2922

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Make JAAS configurable via template variables in system tests

[wangguoz] MINOR: Fix streams Scala foreach recursive call (#5539)

[mjsax] MINOR: restructure Windows to favor immutable implementation (#5536)

--
[...truncated 2.47 MB...]
org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldWriteCheckpointForPersistentLogEnabledStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 

Jenkins build is back to normal : kafka-trunk-jdk10 #432

[Apache Jenkins Server]

See 

Request for contributor permissions

[王又田]

JIRA ID: tony80720

Cwiki ID:  Yu Tien Wang

Thanks in advance!

Tony



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Request for contributor permissions

[王又田]

JIRA ID: tony80720



Cwiki ID: 王又田




Thanks in advance!

Tony



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

hi

[王又田]

Tony



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Build failed in Jenkins: kafka-trunk-jdk10 #431

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-7225; Corrected system tests by generating external properties

[jason] MINOR: Make JAAS configurable via template variables in system tests

--
[...truncated 2.16 MB...]
org.apache.kafka.connect.transforms.FlattenTest > topLevelMapRequired STARTED

org.apache.kafka.connect.transforms.FlattenTest > topLevelMapRequired PASSED

org.apache.kafka.connect.transforms.FlattenTest > topLevelStructRequired STARTED

org.apache.kafka.connect.transforms.FlattenTest > topLevelStructRequired PASSED

org.apache.kafka.connect.transforms.FlattenTest > testOptionalFieldStruct 
STARTED

org.apache.kafka.connect.transforms.FlattenTest > testOptionalFieldStruct PASSED

org.apache.kafka.connect.transforms.FlattenTest > testNestedMapWithDelimiter 
STARTED

org.apache.kafka.connect.transforms.FlattenTest > testNestedMapWithDelimiter 
PASSED

org.apache.kafka.connect.transforms.FlattenTest > testOptionalFieldMap STARTED

org.apache.kafka.connect.transforms.FlattenTest > testOptionalFieldMap PASSED

org.apache.kafka.connect.transforms.FlattenTest > testUnsupportedTypeInMap 
STARTED

org.apache.kafka.connect.transforms.FlattenTest > testUnsupportedTypeInMap 
PASSED

org.apache.kafka.connect.transforms.FlattenTest > testNestedStruct STARTED

org.apache.kafka.connect.transforms.FlattenTest > testNestedStruct PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList PASSED

org.apache.kafka.connect.transforms.HoistFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.HoistFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.HoistFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.HoistFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullWithSchema 
STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullWithSchema PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless 
STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless PASSED

org.apache.kafka.connect.transforms.ValueToKeyTest > withSchema STARTED

org.apache.kafka.connect.transforms.ValueToKeyTest > withSchema PASSED

org.apache.kafka.connect.transforms.ValueToKeyTest > schemaless STARTED

org.apache.kafka.connect.transforms.ValueToKeyTest > schemaless PASSED

org.apache.kafka.connect.transforms.ReplaceFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.ReplaceFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.ReplaceFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.ReplaceFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > 
schemalessInsertConfiguredFields STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > 
schemalessInsertConfiguredFields PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > topLevelStructRequired 
STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > topLevelStructRequired 
PASSED

org.apache.kafka.connect.transforms.InsertFieldTest > 
copySchemaAndInsertConfiguredFields STARTED

org.apache.kafka.connect.transforms.InsertFieldTest > 
copySchemaAndInsertConfiguredFields PASSED

> Task :streams:examples:compileJava
> Task :streams:examples:processResources NO-SOURCE
> Task :streams:examples:classes
> Task :streams:examples:checkstyleMain
> Task :streams:examples:compileTestJava
> Task :streams:examples:processTestResources NO-SOURCE
> Task :streams:examples:testClasses
> Task :streams:examples:checkstyleTest

> Task :streams:examples:test

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test 
STARTED

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test PASSED

> Task :spotlessScala
> Task :spotlessScalaCheck
> Task :streams:streams-scala:compileJava NO-SOURCE

> Task :streams:streams-scala:compileScala
Pruning sources from previous analysis, due to incompatible CompileSetup.
:25:
 Unused 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Hello Boyang,

Thanks for the updated proposal, a few questions:

1. Where will "change-group-timeout" be communicated to the broker? Will
that be a new field in the JoinGroupRequest, or are we going to piggy-back
on the existing session-timeout field (assuming that the original value
will not be used anywhere in the static membership any more)?

2. "However, if the consumer takes longer than session timeout to return,
we shall still trigger rebalance but it could still try to catch
`change-group-timeout`.": what does this mean? I thought your proposal is
that for static memberships, the broker will NOT trigger rebalance even
after session-timeout has been detected, but only that after
change-group-timeout
which is supposed to be longer than session-timeout to be defined?

3. "A join group request with member.name set will be treated as
`static-membership` strategy", in this case, how would the switch from
dynamic to static happen, since whoever changed the member.name to not-null
will be rejected, right?

4. "just erase the cached mapping, and wait for session timeout to trigger
rebalance should be sufficient." this is also a bit unclear to me: who will
erase the cached mapping? Since it is on the broker-side I assume that
broker has to do it. Are you suggesting to use a new request for it?

5. "Halfway switch": following 3) above, if your proposal is basically to
let "first join-request wins", and the strategy will stay as is until all
members are gone, then this will also not happen since whoever used
different strategy as the first guy who sends join-group request will be
rejected right?


Guozhang


On Tue, Aug 21, 2018 at 9:28 AM, John Roesler  wrote:

> This sounds good to me!
>
> Thanks for the time you've spent on it,
> -John
>
> On Tue, Aug 21, 2018 at 12:13 AM Boyang Chen  wrote:
>
> > Thanks Matthias for the input. Sorry I was busy recently and haven't got
> > time to update this thread. To summarize what we come up so far, here is
> a
> > draft updated plan:
> >
> >
> > Introduce a new config called `member.name` which is supposed to be
> > provided uniquely by the consumer client. The broker will maintain a
> cache
> > with [key:member.name, value:member.id]. A join group request with
> > member.name set will be treated as `static-membership` strategy, and
> will
> > reject any join group request without member.name. So this coordination
> > change will be differentiated from the `dynamic-membership` protocol we
> > currently have.
> >
> >
> > When handling static join group request:
> >
> >   1.   The broker will check the membership to see whether this is a new
> > member. If new, broker allocate a unique member id, cache the mapping and
> > move to rebalance stage.
> >   2.   Following 1, if this is an existing member, broker will not change
> > group state, and return its cached member.id and current assignment.
> > (unless this is leader, we shall trigger rebalance)
> >   3.   Although Guozhang has mentioned we could rejoin with pair member
> > name and id, I think for join group request it is ok to leave member id
> > blank as member name is the unique identifier. In commit offset request
> we
> > *must* have both.
> >
> >
> > When handling commit offset request, if enabled with static membership,
> > each time the commit request must have both member.name and member.id to
> > be identified as a `certificated member`. If not, this means there are
> > duplicate consumer members with same member name and the request will be
> > rejected to guarantee consumption uniqueness.
> >
> >
> > When rolling restart/shutting down gracefully, the client will send a
> > leave group request (static membership mode). In static membership, we
> will
> > also define `change-group-timeout` to hold on rebalance provided by
> leader.
> > So we will wait for all the members to rejoin the group and do exactly
> one
> > rebalance since all members are expected to rejoin within timeout. If
> > consumer crashes, the join group request from the restarted consumer will
> > be recognized as an existing member and be handled as above condition 1;
> > However, if the consumer takes longer than session timeout to return, we
> > shall still trigger rebalance but it could still try to catch
> > `change-group-timeout`. If it failed to catch second timeout, its cached
> > state on broker will be garbage collected and trigger a new rebalance
> when
> > it finally joins.
> >
> >
> > And consider the switch between dynamic to static membership.
> >
> >   1.  Dynamic to static: the first joiner shall revise the membership to
> > static and wait for all the current members to restart, since their
> > membership is still dynamic. Here our assumption is that the restart
> > process shouldn't take a long time, as long restart is breaking the
> > `rebalance timeout` in whatever membership protocol we are using. Before
> > restart, all dynamic member join requests will be rejected.
> >   2.  Static to dynamic: this is more like a 

Re: [DISCUSS] KIP-363: Make FunctionConversions private

[Guozhang Wang]

+1.

On Thu, Aug 23, 2018 at 12:47 PM, Ted Yu  wrote:

> +1
>
> In the Motivation section, you can quote the comment from pull request so
> that reader doesn't have to click through.
>
> Cheers
>
> On Thu, Aug 23, 2018 at 12:13 PM Joan Goyeau  wrote:
>
> > Hi,
> >
> > As pointed out in this comment #5539 (comment)
> >  the
> > object FunctionConversions is only of internal use and therefore should
> be
> > private to the lib only so that we can do changes without going through
> KIP
> > like this one.
> >
> > KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+
> FunctionConversions+private
> >
> > Thanks
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-280: Enhanced log compaction

[Jun Rao]

Hi, Luis,

Thanks for the reply. A few more comments below.

1. About the topic level configuration. It seems that it's useful for the
new configs to be at the topic level. Currently, the following configs
related to compaction are already at the topic level.

min.cleanable.dirty.ratio
min.compaction.lag.ms
cleanup.policy

2. Have you documented the memory impact in the KIP?

3. Could you document how we deal with the last message in the log, which
is potentially cleanable now?

4. Could you document how key deletion will be handled?

10. As for Jason's proposal on CompactionStrategy, it does make the feature
more general. On the other hand, it will be useful not to require
user-level code if the compaction value only comes from the header.

20. "If compaction.strategy.header is chosen and compaction.strategy.header
is not set, the KIP falls back to offset." I am wondering if it's better to
just fail the configuration in the case.

Jun



On Thu, Aug 16, 2018 at 1:33 PM, Guozhang Wang  wrote:

> Regarding "broker-agnostic of headers": there are some KIPs from Streams to
> use headers for internal purposes as well, e.g. KIP-258 and KIP-213 (I
> admit there may be a conflict with user space, but practically I think it
> is very rare). So I think we are very likely going to make Kafka internals
> to be "headers-aware" anyways.
>
> Regarding the general API: I think it is a good idea in general, but it may
> still have limits: note that right now our KIP enforce a header type to be
> long, and we have a very careful discussion about the fall-back policy if
> header does not have the specified key or if the value is not long-typed;
> but if we enforce long type version in the interface, it would require
> users trying to customizing their compaction logic (think: based on some
> value payload field) to transform their fields to long as well. So I feel
> we can still go with the current proposed approach, and only consider this
> general API if we observe it does have a general usage requirement. By that
> time we can still extend the config values of "log.cleaner.compaction.
> strategy" to "offset, timestamp, header, myFuncName".
>
> @Bertus
>
> Thanks for your feedback, I believe the proposed config is indeed for both
> global (for the whole broker) and per-topic, Luís can confirm if this is
> the case, and update the wiki page to make it clear.
>
>
> Guozhang
>
>
> On Thu, Aug 16, 2018 at 9:09 AM, Bertus Greeff <
> bgre...@microsoft.com.invalid> wrote:
>
> > I'm interested to know the status of this KIP.  I see that the status is
> > "Voting".  How long does this normally take?
> >
> > We want to use Kafka and this KIP provides exactly the log compaction
> > logic that we want for many of our projects.
> >
> > One piece of feedback that I have is that log.cleaner.compaction.
> strategy
> > and log.cleaner.compaction.strategy.header needs to be per topic.  The
> > text of the KIP makes it sound that the config is only available for all
> > topics but this makes no sense.  Different topics will need different
> > strategies and/or headers.
> >
> > From the KIP:
> > Provide the configuration for the individual topics
> > None of the configurations for log compaction are available at topic
> > level, so adding it there is not a part of this KIP
> >
> >
> >
> > On 2018/04/05 08:44:00, Luís Cabral  wrote:
> > > Hello all,>
> > > Starting a discussion for this feature.>
> > > KIP-280   :  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 280%
> > 3A+Enhanced+log+compactionPull-4822 :  https://github.com/apache/kafk
> > a/pull/4822>
> >
> > > Kind Regards,Luís>
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-346 - Improve LogCleaner behavior on error

[Jun Rao]

Hi, Stan,

Thanks for the KIP. Looks good to me overall. Just one comment below.

uncleanable-partitions-count is per logDir, but uncleanable-bytes is not.
Should we make them consistent?

Jun


On Wed, Aug 22, 2018 at 4:15 AM, Stanislav Kozlovski  wrote:

> Hi everybody,
>
> @Jason - I've updated the section. Thanks for the reminder
>
> I'm glad to say that the vote *has passed* with 3 binding votes (Jason,
> Gwen, Harsha) and 6 non-binding votes (Dhruvil, Colin, Mickael, Manikumar,
> Ray, Ted, Thomas).
>
> The PR is ready for review at https://github.com/apache/kafka/pull/5439
>
> On Tue, Aug 21, 2018 at 4:55 PM Jason Gustafson 
> wrote:
>
> > +1 Thanks for the KIP! I'd suggest mentioning the configurations that
> were
> > previously proposed in the rejected alternatives section. We may
> reconsider
> > them in the future.
> >
> > On Mon, Aug 13, 2018 at 9:48 AM, Dhruvil Shah 
> > wrote:
> >
> > > Thanks for the KIP, Stanislav! +1 (non-binding)
> > >
> > > - Dhruvil
> > >
> > > On Mon, Aug 13, 2018 at 9:39 AM Colin McCabe 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Tue, Aug 7, 2018, at 04:19, Stanislav Kozlovski wrote:
> > > > > Hey everybody,
> > > > > I'm starting a vote on KIP-346
> > > > > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 346+-+Improve+LogCleaner+behavior+on+error
> > > > >
> > > > >
> > > > > --
> > > > > Best,
> > > > > Stanislav
> > > >
> > >
> >
>
>
> --
> Best,
> Stanislav
>

Build failed in Jenkins: kafka-trunk-jdk10 #430

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Return correct instance of SessionWindowSerde (#5546)

--
[...truncated 1.53 MB...]

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendNewMetadataToLogOnAddPartitionsWhenPartitionsAdded PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsNull STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsNull PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithCoordinatorLoadInProgressOnEndTxnWhenCoordinatorIsLoading 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithCoordinatorLoadInProgressOnEndTxnWhenCoordinatorIsLoading 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithSuccessOnAddPartitionsWhenStateIsCompleteAbort STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithSuccessOnAddPartitionsWhenStateIsCompleteAbort PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareAbortState
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareAbortState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

[Jun Rao]

Hi, Lucas,

Sorry for the delay. The new proposal looks good to me overall. A few minor
comments below.

1. It's possible that listener.name.for.controller is set, but set to the
same value as inter.broker.listener.name. In that case, should we have a
single network thread and the request handling thread for that listener?

2. Currently, the controller always picks the listener specified by
inter.broker.listener.name even if the listener name is not present in the
receiving broker. This KIP proposes a slightly different approach for
picking listener.name.for.controller only when the receiving end has the
listener and switches listener.name.for.controller otherwise. There are
some tradeoffs between the two approaches. To change the inter broker
listener, the former requires 2 steps: (1) adding the new listener to
listener list in every broker and (2) changing listener.name.for.controller.
The latter can do both changes in 1 step. On the hand, if
listener.name.for.controller
is mis-configured, the former will report an error and the latter will hide
it (so the user may not know the misconfiguration). It seems that we should
pick one approach to handle both listener.name.for.controller and
inter.broker.listener.name consistently. To me, the former seems slightly
better.

3. To be consistent with the existing naming, should
listener.name.for.controller
be controller.listener.name?

Thanks,

Jun


On Thu, Aug 9, 2018 at 3:21 PM, Lucas Wang  wrote:

> Hi Jun and Joel,
>
> The KIP writeup has changed by quite a bit since your +1.
> Can you please take another review? Thanks a lot for your time!
>
> Lucas
>
> On Tue, Jul 17, 2018 at 10:33 AM, Joel Koshy  wrote:
>
> > +1 on the KIP.
> >
> > (I'm not sure we actually necessary to introduce the condition variables
> > for the concern that Jun raised, but it's an implementation detail that
> we
> > can defer to a discussion in the PR.)
> >
> > On Sat, Jul 14, 2018 at 10:45 PM, Lucas Wang 
> > wrote:
> >
> > > Hi Jun,
> > >
> > > I agree by using the conditional variables, there is no need to add
> such
> > a
> > > new config.
> > > Also thanks for approving this KIP.
> > >
> > > Lucas
> > >
> >
>

Metadata Update Corner Case

[James Lent]

While updating a small Kafka client application to use Kafka 1.1.1 and the 
KafkaProducer (rather than the deprecated Producer) a maintenance related unit 
test started failing.  The scenario is out of the ordinary, but, the resulting 
problem may still be of interest:

 *   Two Kafka nodes
 *   One KafkaProducer
*   "retry.backoff.ms" = 3000
   *   Unchanged from the existing code.
   *   Perhaps a bad setting.
   *   Plays an interesting role in the issue.
 *   Send two messages
 *   Shutdown Kafka Node 0
 *   Send two messages
 *   Restart Kafka Node 0
 *   Shutdown Kafka Node 1
 *   Send two messages
*   This is the step that fails.

Detailed investigation shows that:

 *   Setting "retry.backoff.ms" seems to also increase the minimum metadata 
refresh interval.
 *   When Node 0 goes down a metadata refresh does not occur immediately - it 
is delayed about 3 seconds.
 *   When it does occur Node 0 is gone and the Cluster is recreated with only 
Node 1.
 *   When Kafka Node 0 is restarted nothing seems to trigger a refresh right 
away.
 *   When Kafka Node 1 goes down a metadata refresh is attempted, but, the only 
node in the Cluster current instance is Node 1 which is now not available.
*   All metadata refresh attempts fail (forever?)
*   NetworkClient.maybeUpdate (after call to leastLoadedNode)
*   log.debug("Give up sending metadata request since no node is 
available");

I noted the following:

 *   If the "retry.backoff.ms" is not overridden the problem goes away, but, 
for a strange reason:
*   The metadata refreshes triggered by the shutdowns happen so fast that 
it returns both Nodes (even though one is in the process of going down) and the 
Cluster is recreated with both nodes still available.
 *   If a 3 node configuration is used the problem is much harder to create.
*   I have done so with a Unit Test added to the Kafka source code based on 
the code in DynamicBrokerReconfigurationTest
*   Mostly to better understand the issue.
*   It requires:
   *   Shutting down two nodes at the same time
   *   Waiting for a metadata update that catches this state (or overriding the 
"retry.backoff.ms" value)
  *   Cluster now recreated with just one node
   *   Returning them both to service
   *   Shutting down the third node.
   *   Before a metadata update triggers a Cluster rebuild, send a message.

I realize the Unit Test could be rewritten to be more realistic, but, I still 
think the error state it triggers may be of interest.  Getting in a state where 
you can't update the metadata is pretty serious and perhaps there are more 
realistic ways to trigger this state.

I looked at the old Producer code and it does not seem to have an issue like 
this.  It seems to have a fixed list of metadata nodes that it will randomly 
poll.  Therefore as long as at least one is reachable it will eventually get a 
metadata reply.  In one run of the unit test with the old code I observed that 
it tried the out of service node 5 times in a row before it randomly selected 
the in service one. Perhaps not elegant, but, the unit test passed.




This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

[jira] [Created] (KAFKA-7335) Store clusterId locally to ensure broker joins the right cluster

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-7335:
--

 Summary: Store clusterId locally to ensure broker joins the right 
cluster
 Key: KAFKA-7335
 URL: https://issues.apache.org/jira/browse/KAFKA-7335
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson


We have seen situations where a broker somehow got the wrong configuration and 
joined a different cluster than the one it was previously registered in. This 
can create various kinds of metadata inconsistencies in the cluster and can be 
difficult to debug.  It was suggested in 
[KIP-78|https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id] 
that we could store the clusterId locally after initial registration and verify 
upon startup that the locally stored value matches what is in zookeeper.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #2920

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Jason Gustafson]

To clarify, what I am suggesting is to only remove the default
implementation for these methods. So users would be required to implement
serialize(topic, data) and deserialize(topic, data).

-Jason

On Thu, Aug 23, 2018 at 1:48 PM, Jason Gustafson  wrote:

> Hey Viktor,
>
> Thinking about it a little more, I wonder if we should just not provide a
> default method for serialize(topic, data) and deserialize(topic, data).
> Implementing these methods is a trivial burden for users and it feels like
> there's no good solution which allows both methods to have default
> implementations.
>
> Also, ack on KIP-331. Thanks for the pointer.
>
> -Jason
>
> On Thu, Aug 23, 2018 at 12:30 PM, Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
>
>> Hi Ismael,
>>
>> Regarding the deprecation of the 2 parameter method: should we do this
>> with
>> the Serializer interface as well?
>>
>> I've updated the "Rejected Alternatives" with a few.
>> I've added this circular reference one too but actually there's a way
>> (pretty heavyweight) by adding a guard class that prevents recursive
>> invocation of either methods. I've tried this out but it seems to me an
>> overshoot. So just for the sake of completeness I'll copy it here. :)
>>
>> public interface Deserializer extends Closeable {
>>
>> class Guard {
>>
>> private Set objects = Collections.synchronizedSet(new
>> HashSet<>()); // might as well use concurrent hashmap
>>
>> private void methodCallInProgress(Object x) {
>> objects.add(x);
>> }
>>
>> private boolean isMethodCallInProgress(Object x) {
>> return objects.contains(x);
>> }
>>
>> private void clearMethodCallInProgress(Object x) {
>> objects.remove(x);
>> }
>>
>> private  T guard(Supplier supplier) {
>> if (GUARD.isMethodCallInProgress(this)) {
>> throw new IllegalStateException("You must implement one of
>> the deserialize methods");
>> } else {
>> try {
>> GUARD.methodCallInProgress(this);
>> return supplier.get();
>> } finally {
>> GUARD.clearMethodCallInProgress(this);
>> }
>> }
>> }
>> }
>>
>> Guard GUARD = new Guard();
>>
>> void configure(Map configs, boolean isKey);
>>
>> default T deserialize(String topic, byte[] data) {
>> return GUARD.guard(() -> deserialize(topic, null, data));
>> }
>>
>> default T deserialize(String topic, Headers headers, byte[] data) {
>> return GUARD.guard(() -> deserialize(topic, data));
>> }
>>
>> @Override
>> void close();
>> }
>>
>>
>> Cheers,
>> Viktor
>>
>> On Thu, Aug 23, 2018 at 3:50 PM Ismael Juma  wrote:
>>
>> > Also, we may consider deprecating the deserialize method that does not
>> take
>> > headers. Yes, it's a convenience, but it also adds confusion.
>> >
>> > Ismael
>> >
>> > On Thu, Aug 23, 2018 at 6:48 AM Ismael Juma  wrote:
>> >
>> > > I think the KIP needs the rejected alternatives section to have more
>> > > detail. For example, another option would be something like the
>> > following,
>> > > which works great as long as one overrides one of the methods, but
>> pretty
>> > > bad if one doesn't. :)
>> > >
>> > > default T deserialize(String topic, byte[] data) {
>> > > return deserialize(topic, null, data);
>> > > }
>> > >
>> > > default T deserialize(String topic, Headers headers, byte[] data) { //
>> > > This is the new method
>> > > return deserialize(topic, data);
>> > > }
>> > >
>> > >
>> > > On Thu, Aug 23, 2018 at 3:57 AM Viktor Somogyi-Vass <
>> > > viktorsomo...@gmail.com> wrote:
>> > >
>> > >> Hi Jason,
>> > >>
>> > >> Thanks for the feedback.
>> > >> 1. I chose to return null here because according to the
>> documentation it
>> > >> may return null data, therefore the users of this methods are
>> perpared
>> > for
>> > >> getting a null. Thinking of it though it may be better to throw an
>> > >> exception by default because it'd indicate a programming error.
>> However,
>> > >> would that be a backward incompatible change? I'm simply thinking of
>> > this
>> > >> because this is a new behavior that we'd introduce but I'm not sure
>> yet
>> > if
>> > >> it'd cause problems.
>> > >> Do you think it'd make sense to do the same in `serialize`?
>> > >> 2. Yes, I believe that is covered in KIP-331:
>> > >>
>> > >>
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+
>> Add+default+implementation+to+close%28%29+and+configure%28%
>> 29+for+Serializer%2C+Deserializer+and+Serde
>> > >>
>> > >> Cheers,
>> > >> Viktor
>> > >>
>> > >> On Wed, Aug 22, 2018 at 6:11 PM Jason Gustafson 
>> > >> wrote:
>> > >>
>> > >> > Hey Viktor,
>> > >> >
>> > >> > This is a nice cleanup. Just a couple quick questions:
>> > >> >
>> > >> > 1. Rather than returning null for the default `deserialize(topic,
>> > >> data)`,
>> > >> > would it be 

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Jason Gustafson]

Hey Viktor,

Thinking about it a little more, I wonder if we should just not provide a
default method for serialize(topic, data) and deserialize(topic, data).
Implementing these methods is a trivial burden for users and it feels like
there's no good solution which allows both methods to have default
implementations.

Also, ack on KIP-331. Thanks for the pointer.

-Jason

On Thu, Aug 23, 2018 at 12:30 PM, Viktor Somogyi-Vass <
viktorsomo...@gmail.com> wrote:

> Hi Ismael,
>
> Regarding the deprecation of the 2 parameter method: should we do this with
> the Serializer interface as well?
>
> I've updated the "Rejected Alternatives" with a few.
> I've added this circular reference one too but actually there's a way
> (pretty heavyweight) by adding a guard class that prevents recursive
> invocation of either methods. I've tried this out but it seems to me an
> overshoot. So just for the sake of completeness I'll copy it here. :)
>
> public interface Deserializer extends Closeable {
>
> class Guard {
>
> private Set objects = Collections.synchronizedSet(new
> HashSet<>()); // might as well use concurrent hashmap
>
> private void methodCallInProgress(Object x) {
> objects.add(x);
> }
>
> private boolean isMethodCallInProgress(Object x) {
> return objects.contains(x);
> }
>
> private void clearMethodCallInProgress(Object x) {
> objects.remove(x);
> }
>
> private  T guard(Supplier supplier) {
> if (GUARD.isMethodCallInProgress(this)) {
> throw new IllegalStateException("You must implement one of
> the deserialize methods");
> } else {
> try {
> GUARD.methodCallInProgress(this);
> return supplier.get();
> } finally {
> GUARD.clearMethodCallInProgress(this);
> }
> }
> }
> }
>
> Guard GUARD = new Guard();
>
> void configure(Map configs, boolean isKey);
>
> default T deserialize(String topic, byte[] data) {
> return GUARD.guard(() -> deserialize(topic, null, data));
> }
>
> default T deserialize(String topic, Headers headers, byte[] data) {
> return GUARD.guard(() -> deserialize(topic, data));
> }
>
> @Override
> void close();
> }
>
>
> Cheers,
> Viktor
>
> On Thu, Aug 23, 2018 at 3:50 PM Ismael Juma  wrote:
>
> > Also, we may consider deprecating the deserialize method that does not
> take
> > headers. Yes, it's a convenience, but it also adds confusion.
> >
> > Ismael
> >
> > On Thu, Aug 23, 2018 at 6:48 AM Ismael Juma  wrote:
> >
> > > I think the KIP needs the rejected alternatives section to have more
> > > detail. For example, another option would be something like the
> > following,
> > > which works great as long as one overrides one of the methods, but
> pretty
> > > bad if one doesn't. :)
> > >
> > > default T deserialize(String topic, byte[] data) {
> > > return deserialize(topic, null, data);
> > > }
> > >
> > > default T deserialize(String topic, Headers headers, byte[] data) { //
> > > This is the new method
> > > return deserialize(topic, data);
> > > }
> > >
> > >
> > > On Thu, Aug 23, 2018 at 3:57 AM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com> wrote:
> > >
> > >> Hi Jason,
> > >>
> > >> Thanks for the feedback.
> > >> 1. I chose to return null here because according to the documentation
> it
> > >> may return null data, therefore the users of this methods are perpared
> > for
> > >> getting a null. Thinking of it though it may be better to throw an
> > >> exception by default because it'd indicate a programming error.
> However,
> > >> would that be a backward incompatible change? I'm simply thinking of
> > this
> > >> because this is a new behavior that we'd introduce but I'm not sure
> yet
> > if
> > >> it'd cause problems.
> > >> Do you think it'd make sense to do the same in `serialize`?
> > >> 2. Yes, I believe that is covered in KIP-331:
> > >>
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+
> implementation+to+close%28%29+and+configure%28%29+for+
> Serializer%2C+Deserializer+and+Serde
> > >>
> > >> Cheers,
> > >> Viktor
> > >>
> > >> On Wed, Aug 22, 2018 at 6:11 PM Jason Gustafson 
> > >> wrote:
> > >>
> > >> > Hey Viktor,
> > >> >
> > >> > This is a nice cleanup. Just a couple quick questions:
> > >> >
> > >> > 1. Rather than returning null for the default `deserialize(topic,
> > >> data)`,
> > >> > would it be better to throw UnsupportedOperationException? I assume
> > that
> > >> > internally we'll always invoke the api which takes headers.
> Similarly
> > >> for
> > >> > `serialize(topic, data)`.
> > >> > 2. Would it make sense to have default no-op implementations for
> > >> > `configure` and `close`?
> > >> >
> > >> > Thanks,
> > >> > Jason
> > >> >
> > >> > On Wed, Aug 22, 2018 at 5:27 AM, Satish Duggana <
> > >> 

Re: Build failed in Jenkins: kafka-trunk-jdk10 #429

[Ted Yu]

I ran streams unit tests as of
commit 4156ea0a9bcca67d209fd3b43d2268c9abd5a0b5 .

All tests passed locally.

FYI

On Thu, Aug 23, 2018 at 12:23 PM Joan Goyeau  wrote:

> I'm looking into this one.
>
> On Thu, 23 Aug 2018 at 20:19 Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
> > See <
> >
> https://builds.apache.org/job/kafka-trunk-jdk10/429/display/redirect?page=changes
> > >
> >
> > Changes:
> >
> > [wangguoz] KAFKA-7316: Fix Streams Scala filter recursive call #5538
> >
> > --
> > [...truncated 1.98 MB...]
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldNotHaveSameAssignmentOnAnyTwoHosts PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldRebalanceTasksToClientsBasedOnCapacity STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldRebalanceTasksToClientsBasedOnCapacity PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > >
> shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousStandbyTasks
> > STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > >
> shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousStandbyTasks
> > PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > >
> >
> shouldNotAssignStandbyTaskReplicasWhenNoClientAvailableWithoutHavingTheTaskAssigned
> > STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > >
> >
> shouldNotAssignStandbyTaskReplicasWhenNoClientAvailableWithoutHavingTheTaskAssigned
> > PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignEachActiveTaskToOneClientWhenMoreClientsThanTasks STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignEachActiveTaskToOneClientWhenMoreClientsThanTasks PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignTopicGroupIdEvenlyAcrossClientsWithStandByTasks STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignTopicGroupIdEvenlyAcrossClientsWithStandByTasks PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignTasksNotPreviouslyActiveToNewClient STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignTasksNotPreviouslyActiveToNewClient PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignOneActiveTaskToEachProcessWhenTaskCountSameAsProcessCount
> > STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignOneActiveTaskToEachProcessWhenTaskCountSameAsProcessCount
> > PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldReBalanceTasksAcrossClientsWhenCapacityLessThanTaskCount STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldReBalanceTasksAcrossClientsWhenCapacityLessThanTaskCount PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldBalanceActiveAndStandbyTasksAcrossAvailableClients STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldBalanceActiveAndStandbyTasksAcrossAvailableClients PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignActiveAndStandbyTasks STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignActiveAndStandbyTasks PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousActiveTasks
> > STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousActiveTasks
> > PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignTasksToClientWithPreviousStandbyTasks STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldAssignTasksToClientWithPreviousStandbyTasks PASSED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldReBalanceTasksAcrossAllClientsWhenCapacityAndTaskCountTheSame
> > STARTED
> >
> >
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > > shouldReBalanceTasksAcrossAllClientsWhenCapacityAndTaskCountTheSame
> 

Re: [DISCUSS] KIP-363: Make FunctionConversions private

[Ted Yu]

+1

In the Motivation section, you can quote the comment from pull request so
that reader doesn't have to click through.

Cheers

On Thu, Aug 23, 2018 at 12:13 PM Joan Goyeau  wrote:

> Hi,
>
> As pointed out in this comment #5539 (comment)
>  the
> object FunctionConversions is only of internal use and therefore should be
> private to the lib only so that we can do changes without going through KIP
> like this one.
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+FunctionConversions+private
>
> Thanks
>

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

[Harsha]

+1 (binding)

Thanks,
Harsha

On Wed, Aug 22, 2018, at 9:15 AM, Manikumar wrote:
> Hi Viktor,
> We already have a method in Authorizer interface to get acls for a given
> principal.
> We will use this method to fetch acls and filter the results for requested
> Resources.
> Authorizer {
>def getAcls(principal: KafkaPrincipal): Map[Resource, Set[Acl]]
> }
> Currently AdminClient API doesn't have a API to fetch acls for a given
> principal.
> So while using AclCommand with AdminClient API (KIP-332), we just filter
> the results returned
> from describeAcls API. We can add new AdminClient API/new
> DescribeAclsRequest if required in future.
> 
> Updated the KIP. Thanks for the review.
> 
> Thanks,
> 
> On Wed, Aug 22, 2018 at 5:53 PM Viktor Somogyi-Vass 
> wrote:
> 
> > Hi Manikumar,
> >
> > Implementation-wise is it just a filter over the returned ACL listing or do
> > you plan to add new methods to the Authorizer as well?
> >
> > Thanks,
> > Viktor
> >
> > On Fri, Aug 17, 2018 at 9:18 PM Priyank Shah 
> > wrote:
> >
> > > +1(non-binding)
> > >
> > > Thanks.
> > > Priyank
> > >
> > > On 8/16/18, 6:01 AM, "Manikumar"  wrote:
> > >
> > > Hi all,
> > >
> > > I have created a minor KIP to add support to list ACLs per principal
> > > using
> > > AclCommand (kafka-acls.sh)
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-357%3A++Add+support+to+list+ACLs+per+principal
> > >
> > > Please take a look.
> > >
> > > Thanks,
> > >
> > >
> > >
> >

[jira] [Created] (KAFKA-7334) Suggest changing config for state.dir in case of FileNotFoundException

[Ted Yu (JIRA)]

Ted Yu created KAFKA-7334:
-

 Summary: Suggest changing config for state.dir in case of 
FileNotFoundException
 Key: KAFKA-7334
 URL: https://issues.apache.org/jira/browse/KAFKA-7334
 Project: Kafka
  Issue Type: Improvement
Reporter: Ted Yu


Quoting stack trace from KAFKA-5998 :
{code}
WARN [2018-08-22 03:17:03,745] 
org.apache.kafka.streams.processor.internals.ProcessorStateManager: task [0_45] 
Failed to write offset checkpoint file to /tmp/kafka-streams/
{{ /0_45/.checkpoint: {}}}
{{ ! java.nio.file.NoSuchFileException: 
/tmp/kafka-streams//0_45/.checkpoint.tmp}}
{{ ! at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)}}
{{ ! at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)}}
{code}
When state.dir is left at default configuration, there is a chance that certain 
files under the state directory are cleaned by OS.

[~mjsax] and I proposed to suggest user, through exception message, to change 
the location for state.dir .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Viktor Somogyi-Vass]

Hi Ismael,

Regarding the deprecation of the 2 parameter method: should we do this with
the Serializer interface as well?

I've updated the "Rejected Alternatives" with a few.
I've added this circular reference one too but actually there's a way
(pretty heavyweight) by adding a guard class that prevents recursive
invocation of either methods. I've tried this out but it seems to me an
overshoot. So just for the sake of completeness I'll copy it here. :)

public interface Deserializer extends Closeable {

class Guard {

private Set objects = Collections.synchronizedSet(new
HashSet<>()); // might as well use concurrent hashmap

private void methodCallInProgress(Object x) {
objects.add(x);
}

private boolean isMethodCallInProgress(Object x) {
return objects.contains(x);
}

private void clearMethodCallInProgress(Object x) {
objects.remove(x);
}

private  T guard(Supplier supplier) {
if (GUARD.isMethodCallInProgress(this)) {
throw new IllegalStateException("You must implement one of
the deserialize methods");
} else {
try {
GUARD.methodCallInProgress(this);
return supplier.get();
} finally {
GUARD.clearMethodCallInProgress(this);
}
}
}
}

Guard GUARD = new Guard();

void configure(Map configs, boolean isKey);

default T deserialize(String topic, byte[] data) {
return GUARD.guard(() -> deserialize(topic, null, data));
}

default T deserialize(String topic, Headers headers, byte[] data) {
return GUARD.guard(() -> deserialize(topic, data));
}

@Override
void close();
}


Cheers,
Viktor

On Thu, Aug 23, 2018 at 3:50 PM Ismael Juma  wrote:

> Also, we may consider deprecating the deserialize method that does not take
> headers. Yes, it's a convenience, but it also adds confusion.
>
> Ismael
>
> On Thu, Aug 23, 2018 at 6:48 AM Ismael Juma  wrote:
>
> > I think the KIP needs the rejected alternatives section to have more
> > detail. For example, another option would be something like the
> following,
> > which works great as long as one overrides one of the methods, but pretty
> > bad if one doesn't. :)
> >
> > default T deserialize(String topic, byte[] data) {
> > return deserialize(topic, null, data);
> > }
> >
> > default T deserialize(String topic, Headers headers, byte[] data) { //
> > This is the new method
> > return deserialize(topic, data);
> > }
> >
> >
> > On Thu, Aug 23, 2018 at 3:57 AM Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com> wrote:
> >
> >> Hi Jason,
> >>
> >> Thanks for the feedback.
> >> 1. I chose to return null here because according to the documentation it
> >> may return null data, therefore the users of this methods are perpared
> for
> >> getting a null. Thinking of it though it may be better to throw an
> >> exception by default because it'd indicate a programming error. However,
> >> would that be a backward incompatible change? I'm simply thinking of
> this
> >> because this is a new behavior that we'd introduce but I'm not sure yet
> if
> >> it'd cause problems.
> >> Do you think it'd make sense to do the same in `serialize`?
> >> 2. Yes, I believe that is covered in KIP-331:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> >>
> >> Cheers,
> >> Viktor
> >>
> >> On Wed, Aug 22, 2018 at 6:11 PM Jason Gustafson 
> >> wrote:
> >>
> >> > Hey Viktor,
> >> >
> >> > This is a nice cleanup. Just a couple quick questions:
> >> >
> >> > 1. Rather than returning null for the default `deserialize(topic,
> >> data)`,
> >> > would it be better to throw UnsupportedOperationException? I assume
> that
> >> > internally we'll always invoke the api which takes headers. Similarly
> >> for
> >> > `serialize(topic, data)`.
> >> > 2. Would it make sense to have default no-op implementations for
> >> > `configure` and `close`?
> >> >
> >> > Thanks,
> >> > Jason
> >> >
> >> > On Wed, Aug 22, 2018 at 5:27 AM, Satish Duggana <
> >> satish.dugg...@gmail.com>
> >> > wrote:
> >> >
> >> > > +1
> >> > >
> >> > > On Wed, Aug 22, 2018 at 4:45 PM, Ted Yu 
> wrote:
> >> > >
> >> > > > +1
> >> > > >  Original message From: Kamal Chandraprakash <
> >> > > > kamal.chandraprak...@gmail.com> Date: 8/22/18  3:19 AM
> (GMT-08:00)
> >> > To:
> >> > > > dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate
> >> > > > ExtendedSerializer/Serializer and
> ExtendedDeserializer/Deserializer
> >> > > > +1
> >> > > >
> >> > > > Thanks for the KIP!
> >> > > >
> >> > > > On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass <
> >> > > > viktorsomo...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi All,
> >> > > > >
> >> > > > > I'd like to start a vote on 

Re: Build failed in Jenkins: kafka-trunk-jdk10 #429

[Joan Goyeau]

I'm looking into this one.

On Thu, 23 Aug 2018 at 20:19 Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See <
> https://builds.apache.org/job/kafka-trunk-jdk10/429/display/redirect?page=changes
> >
>
> Changes:
>
> [wangguoz] KAFKA-7316: Fix Streams Scala filter recursive call #5538
>
> --
> [...truncated 1.98 MB...]
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldNotHaveSameAssignmentOnAnyTwoHosts PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldRebalanceTasksToClientsBasedOnCapacity STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldRebalanceTasksToClientsBasedOnCapacity PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousStandbyTasks
> STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousStandbyTasks
> PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> >
> shouldNotAssignStandbyTaskReplicasWhenNoClientAvailableWithoutHavingTheTaskAssigned
> STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> >
> shouldNotAssignStandbyTaskReplicasWhenNoClientAvailableWithoutHavingTheTaskAssigned
> PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignEachActiveTaskToOneClientWhenMoreClientsThanTasks STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignEachActiveTaskToOneClientWhenMoreClientsThanTasks PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignTopicGroupIdEvenlyAcrossClientsWithStandByTasks STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignTopicGroupIdEvenlyAcrossClientsWithStandByTasks PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignTasksNotPreviouslyActiveToNewClient STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignTasksNotPreviouslyActiveToNewClient PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignOneActiveTaskToEachProcessWhenTaskCountSameAsProcessCount
> STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignOneActiveTaskToEachProcessWhenTaskCountSameAsProcessCount
> PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldReBalanceTasksAcrossClientsWhenCapacityLessThanTaskCount STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldReBalanceTasksAcrossClientsWhenCapacityLessThanTaskCount PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldBalanceActiveAndStandbyTasksAcrossAvailableClients STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldBalanceActiveAndStandbyTasksAcrossAvailableClients PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignActiveAndStandbyTasks STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignActiveAndStandbyTasks PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousActiveTasks
> STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousActiveTasks
> PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignTasksToClientWithPreviousStandbyTasks STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignTasksToClientWithPreviousStandbyTasks PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldReBalanceTasksAcrossAllClientsWhenCapacityAndTaskCountTheSame
> STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldReBalanceTasksAcrossAllClientsWhenCapacityAndTaskCountTheSame PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignMoreTasksToClientWithMoreCapacity STARTED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldAssignMoreTasksToClientWithMoreCapacity PASSED
>
> org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest
> > shouldMigrateActiveTasksToNewProcessWithoutChangingAllAssignments STARTED

Build failed in Jenkins: kafka-trunk-jdk10 #429

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-7316: Fix Streams Scala filter recursive call #5538

--
[...truncated 1.98 MB...]
org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldNotHaveSameAssignmentOnAnyTwoHosts PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldRebalanceTasksToClientsBasedOnCapacity STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldRebalanceTasksToClientsBasedOnCapacity PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousStandbyTasks 
STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousStandbyTasks 
PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> 
shouldNotAssignStandbyTaskReplicasWhenNoClientAvailableWithoutHavingTheTaskAssigned
 STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> 
shouldNotAssignStandbyTaskReplicasWhenNoClientAvailableWithoutHavingTheTaskAssigned
 PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignEachActiveTaskToOneClientWhenMoreClientsThanTasks STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignEachActiveTaskToOneClientWhenMoreClientsThanTasks PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignTopicGroupIdEvenlyAcrossClientsWithStandByTasks STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignTopicGroupIdEvenlyAcrossClientsWithStandByTasks PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignTasksNotPreviouslyActiveToNewClient STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignTasksNotPreviouslyActiveToNewClient PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignOneActiveTaskToEachProcessWhenTaskCountSameAsProcessCount STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignOneActiveTaskToEachProcessWhenTaskCountSameAsProcessCount PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldReBalanceTasksAcrossClientsWhenCapacityLessThanTaskCount STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldReBalanceTasksAcrossClientsWhenCapacityLessThanTaskCount PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldBalanceActiveAndStandbyTasksAcrossAvailableClients STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldBalanceActiveAndStandbyTasksAcrossAvailableClients PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignActiveAndStandbyTasks STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignActiveAndStandbyTasks PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousActiveTasks 
STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldNotHaveSameAssignmentOnAnyTwoHostsWhenThereArePreviousActiveTasks PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignTasksToClientWithPreviousStandbyTasks STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignTasksToClientWithPreviousStandbyTasks PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldReBalanceTasksAcrossAllClientsWhenCapacityAndTaskCountTheSame STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldReBalanceTasksAcrossAllClientsWhenCapacityAndTaskCountTheSame PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignMoreTasksToClientWithMoreCapacity STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldAssignMoreTasksToClientWithMoreCapacity PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldMigrateActiveTasksToNewProcessWithoutChangingAllAssignments STARTED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> shouldMigrateActiveTasksToNewProcessWithoutChangingAllAssignments PASSED

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignorTest 
> 

[DISCUSS] KIP-363: Make FunctionConversions private

[Joan Goyeau]

Hi,

As pointed out in this comment #5539 (comment)
 the
object FunctionConversions is only of internal use and therefore should be
private to the lib only so that we can do changes without going through KIP
like this one.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+FunctionConversions+private

Thanks

[jira] [Created] (KAFKA-7333) Protocol changes for KIP-320

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-7333:
--

 Summary: Protocol changes for KIP-320
 Key: KAFKA-7333
 URL: https://issues.apache.org/jira/browse/KAFKA-7333
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Implement protocol changes for 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A+Allow+fetchers+to+detect+and+handle+log+truncation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7332) Improve error message when trying to produce message without key for compacted topic

[Patrik Kleindl (JIRA)]

Patrik Kleindl created KAFKA-7332:
-

 Summary: Improve error message when trying to produce message 
without key for compacted topic
 Key: KAFKA-7332
 URL: https://issues.apache.org/jira/browse/KAFKA-7332
 Project: Kafka
  Issue Type: Improvement
  Components: producer 
Affects Versions: 1.1.0
Reporter: Patrik Kleindl


Goal:

Return a specific error message like e.g. "Message without a key is not valid 
for a compacted topic" when trying to produce such a message instead of a 
CorruptRecordException.

 

> Yesterday we had the following exception:
> 
> Exception thrown when sending a message with key='null' and payload='...'
> to topic sometopic:: org.apache.kafka.common.errors.CorruptRecordException:
> This message has failed its CRC checksum, exceeds the valid size, or is
> otherwise corrupt.
> 
> The cause was identified with the help of
> 
>[https://stackoverflow.com/questions/49098274/kafka-stream-get-corruptrecordexception]
> 
> Is it possible / would it makes sense to open an issue to improve the error
> message for this case?
> A simple "Message without a key is not valid for a compacted topic" would
> suffice and point a user  in the right direction.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7331) Kafka does not detect broker loss in the event of a network partition within the cluster

[Kevin Li (JIRA)]

Kevin Li created KAFKA-7331:
---

 Summary: Kafka does not detect broker loss in the event of a 
network partition within the cluster
 Key: KAFKA-7331
 URL: https://issues.apache.org/jira/browse/KAFKA-7331
 Project: Kafka
  Issue Type: Bug
  Components: controller, network
Affects Versions: 1.0.1
Reporter: Kevin Li


We ran into this issue on our production cluster and had to manually remove the 
broker and enable unclean leader elections to get the cluster working again. 
Ideally, Kafka itself could handle network partitions without manual 
intervention.

The issue is reproducible with the following cross datacenter Kafka cluster 
setup:
DC 1: Kafka brokers + ZK nodes
DC 2: Kafka brokers + ZK nodes
DC 3: Kafka brokers + ZK nodes

Introduce a network partition on a Kafka broker (brokerA) in DC 1 where it 
cannot reach any hosts (brokers and ZK nodes) in the other 2 datacenters. The 
cluster goes into a state where partitions that brokerA is a leader for will 
only contain brokerA in its ISR. Since brokerA is still reachable by ZK nodes 
in DC 1, it still shows up when querying ZK. The controller thinks brokerA is 
still up and does not elect new leaders for partitions that brokerA is a leader 
for. This causes all those partitions to be down until brokerA is back or 
completely removed from the cluster (in which case unclean leader election can 
elect new leaders for those partitions).

A faster recovery scenario could be for a majority of hosts (zk nodes?) to 
realize that brokerA is unreachable, and mark it as down so elections for 
partitions it is a leader for could be triggered. This avoids waiting 
indefinitely for the broker to come back or taking action to remove the broker 
from the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

[Matthias J. Sax]

It seems nobody has any objections against the change.

That's for the KIP improvement. I'll go ahead and merge the PR.


-Matthias

On 8/21/18 2:44 PM, John Roesler wrote:
> Hello again, all,
> 
> I belatedly had a better idea for adding grace period to the Windows class
> hierarchy (TimeWindows, UnlimitedWindows, JoinWindows). Instead of
> providing the grace-setter in the abstract class and having to retract it
> in UnlimitedWindows, I've made the getter abstract method in Windows and
> only added setters to Time and Join windows.
> 
> This should not only improve the ergonomics of grace period, but make the
> whole class hierarchy more maintainable.
> 
> See the PR for more details: https://github.com/apache/kafka/pull/5536
> 
> I've updated the KIP accordingly. Here's the diff:
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=87295409=11=9
> 
> Please let me know if this changes your vote.
> 
> Thanks,
> -John
> 
> On Mon, Aug 13, 2018 at 5:20 PM John Roesler  wrote:
> 
>> Hey all,
>>
>> I just wanted to let you know that a few small issues surfaced during
>> implementation and review. I've updated the KIP. Here's the diff:
>> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=87295409=9=8
>>
>> Basically:
>> * the metrics named "*-event-*" are inconsistent with existing
>> nomenclature, and will be "*-record-*" instead (late records instead of
>> late events, for example)
>> * the apis taking and returning Duration will use long millis instead. We
>> do want to transition to Duration in the future, but we shouldn't do it
>> piecemeal.
>>
>> Thanks,
>> -John
>>
>> On Tue, Aug 7, 2018 at 12:07 PM John Roesler  wrote:
>>
>>> Thanks everyone, KIP-328 has passed with 3 binding votes (Guozhang,
>>> Damian, and Matthias) and 3 non-binding (Ted, Bill, and me).
>>>
>>> Thanks for your time,
>>> -John
>>>
>>> On Mon, Aug 6, 2018 at 6:35 PM Matthias J. Sax 
>>> wrote:
>>>
 +1 (binding)

 Thanks for the KIP.


 -Matthias

 On 8/3/18 12:52 AM, Damian Guy wrote:
> Thanks John! +1
>
> On Mon, 30 Jul 2018 at 23:58 Guozhang Wang  wrote:
>
>> Yes, the addendum lgtm as well. Thanks!
>>
>> On Mon, Jul 30, 2018 at 3:34 PM, John Roesler 
 wrote:
>>
>>> Another thing that came up after I started working on an
 implementation
>> is
>>> that in addition to deprecating "retention" from the Windows
 interface,
>> we
>>> also need to deprecate "segmentInterval", for the same reasons. I
 simply
>>> overlooked it previously. I've updated the KIP accordingly.
>>>
>>> Hopefully, this doesn't change anyone's vote.
>>>
>>> Thanks,
>>> -John
>>>
>>> On Mon, Jul 30, 2018 at 5:31 PM John Roesler 
 wrote:
>>>
 Thanks Guozhang,

 Thanks for that catch. to clarify, currently, events are "late" only
>> when
 they are older than the retention period. Currently, we detect this
 in
>>> the
 processor and record it as a "skipped-record". We then do not
 attempt
>> to
 store the event in the window store. If a user provided a
>> pre-configured
 window store with a retention period smaller than the one they
 specify
>>> via
 Windows#until, the segmented store will drop the update with no
 metric
>>> and
 record a debug-level log.

 With KIP-328, with the introduction of "grace period" and moving
>>> retention
 fully into the state store, we need to have metrics for both "late
>>> events"
 (new records older than the grace period) and "expired window
 events"
>>> (new
 records for windows that are no longer retained in the state
 store). I
 already proposed metrics for the late events, and I've just updated
 the
>>> KIP
 with metrics for the expired window events. I also updated the KIP
 to
>>> make
 it clear that neither late nor expired events will count as
 "skipped-records" any more.

 -John

 On Mon, Jul 30, 2018 at 4:22 PM Guozhang Wang 
>>> wrote:

> Hi John,
>
> Thanks for the updated KIP, +1 from me, and one minor suggestion:
>
> Following your suggestion of the differentiation of
 `skipped-records`
>>> v.s.
> `late-event-drop`, we should probably consider moving the scenarios
>>> where
> records got ignored due the window not being available any more in
> windowed
> aggregation operators from the `skipped-records` metrics recording
 to
>>> the
> `late-event-drop` metrics recording.
>
>
>
> Guozhang
>
>
> On Mon, Jul 30, 2018 at 1:36 PM, Bill Bejeck 
>> wrote:
>
>> Thanks for the KIP!
>>
>> +1

Build failed in Jenkins: kafka-trunk-jdk8 #2919

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Eliminate warnings from KafkaProducerTest (#5548)

--
[...truncated 870.50 KB...]

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartString STARTED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartString PASSED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartWithEmbeddedSeparators 
STARTED

kafka.security.auth.ResourceTest > shouldParseOldTwoPartWithEmbeddedSeparators 
PASSED

kafka.security.auth.ResourceTest > 
shouldThrowOnTwoPartStringWithUnknownResourceType STARTED

kafka.security.auth.ResourceTest > 
shouldThrowOnTwoPartStringWithUnknownResourceType PASSED

kafka.security.auth.ResourceTest > shouldThrowOnBadResourceTypeSeparator STARTED

kafka.security.auth.ResourceTest > shouldThrowOnBadResourceTypeSeparator PASSED

kafka.security.auth.ResourceTest > shouldParseThreePartString STARTED

kafka.security.auth.ResourceTest > shouldParseThreePartString PASSED

kafka.security.auth.ResourceTest > shouldRoundTripViaString STARTED

kafka.security.auth.ResourceTest > shouldRoundTripViaString PASSED

kafka.security.auth.ResourceTest > shouldParseThreePartWithEmbeddedSeparators 
STARTED

kafka.security.auth.ResourceTest > shouldParseThreePartWithEmbeddedSeparators 
PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTED

kafka.security.auth.PermissionTypeTest > testFromString PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
PASSED

> Task :kafka-trunk-jdk8:spotlessScala UP-TO-DATE
> Task :kafka-trunk-jdk8:spotlessScalaCheck UP-TO-DATE
> Task :kafka-trunk-jdk8:core:copyDependantLibs
> Task :kafka-trunk-jdk8:core:jar
> Task :kafka-trunk-jdk8:connect:api:compileJava UP-TO-DATE
> Task :kafka-trunk-jdk8:connect:api:processResources NO-SOURCE
> Task :kafka-trunk-jdk8:connect:api:classes UP-TO-DATE
> Task :kafka-trunk-jdk8:connect:api:copyDependantLibs UP-TO-DATE
> Task :kafka-trunk-jdk8:connect:api:jar UP-TO-DATE
> Task :kafka-trunk-jdk8:connect:json:compileJava UP-TO-DATE
> Task :kafka-trunk-jdk8:connect:json:processResources NO-SOURCE
> Task :kafka-trunk-jdk8:connect:json:classes UP-TO-DATE
> Task :kafka-trunk-jdk8:connect:json:copyDependantLibs UP-TO-DATE
> Task :kafka-trunk-jdk8:connect:json:jar UP-TO-DATE
> Task :kafka-trunk-jdk8:streams:compileJava UP-TO-DATE
> Task :kafka-trunk-jdk8:streams:processResources NO-SOURCE
> Task :kafka-trunk-jdk8:streams:classes UP-TO-DATE
> Task :kafka-trunk-jdk8:streams:copyDependantLibs
> Task :kafka-trunk-jdk8:streams:jar UP-TO-DATE
> Task :kafka-trunk-jdk8:streams:test-utils:compileJava UP-TO-DATE
> Task :kafka-trunk-jdk8:streams:test-utils:processResources NO-SOURCE
> Task :kafka-trunk-jdk8:streams:test-utils:classes UP-TO-DATE
> Task :kafka-trunk-jdk8:streams:test-utils:copyDependantLibs
> Task :kafka-trunk-jdk8:streams:test-utils:jar UP-TO-DATE

> Task :kafka-trunk-jdk8:streams:compileTestJava
:171:
 warning: non-varargs call of varargs method with inexact argument type for 
last parameter;
builder.addProcessor("processor", new MockProcessorSupplier(), null);
   ^
  cast to String for a varargs call
  cast to String[] for a non-varargs call and to suppress this 

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Ismael Juma]

Yeah, the reason why we want to deprecate the auto create functionality is
that it happens when a metadata request is done instead of when a write
operation happens. So, there's no reason to differentiate between the two.

Ismael

On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto  wrote:

> Ah, I just realized that as proposed this is only for the Java consumer
> client, correct?  Would it be possible to make this a broker config, like
> the current one?  Something like:
>
> auto.create.topics.enable=true # allow both producer and consumer to create
> auto.create.topics.enable=consumer # allow only consumer to create
> auto.create.topics.enable=producer # allow only producer to create
> auto.create.topics.enable=false # deny any auto topic creation
>
> Perhaps the broker doesn’t differentiate between the type of client
> connection. If not, I guess this wouldn’t be possible.
>
>
>
> On Thu, Aug 23, 2018 at 11:08 AM Andrew Otto  wrote:
>
> > Yup :)
> >
> > On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma  wrote:
> >
> >> Andrew, one question: you are relying on auto topic creation for the
> >> producer and that's why you can't just disable it?
> >>
> >> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma  wrote:
> >>
> >> > Thanks for sharing Andrew!
> >> >
> >> > Ismael
> >> >
> >> > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto 
> wrote:
> >> >
> >> >> We recently had a pretty serious Kafka outage
> >> >> <
> >> >>
> >>
> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
> >> >> >
> >> >> caused by a bug in one of our consumers that caused it to create new
> >> >> topics
> >> >> in an infinite loop AKA a topic bomb!  Having consumers restricted
> from
> >> >> creating topics would have prevented this for us.
> >> >>
> >> >> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma 
> wrote:
> >> >>
> >> >> > Generally, I think positive configs (`allow` instead of `suppress`)
> >> are
> >> >> > easier to understand.
> >> >> >
> >> >> > Ismael
> >> >> >
> >> >> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <
> >> matth...@confluent.io
> >> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Thanks for the summary!
> >> >> > >
> >> >> > > We might want to add a diagram/table to the docs when we add this
> >> >> > > feature (with whatever config name we choose) to explain how
> broker
> >> >> > > config `auto.create.topics.enable` and the consumer config work
> >> >> together.
> >> >> > >
> >> >> > > I think both options are equally easy to understand. "allow"
> means
> >> >> > > follow the broker config, while "suppress" implies ignore the
> >> broker
> >> >> > > config and don't auto-create.
> >> >> > >
> >> >> > >
> >> >> > > -Matthias
> >> >> > >
> >> >> > >
> >> >> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> >> >> > > > *"suppress" is the opposite of "allow", so
> >> >> > > > setting suppress.auto.create.topics=false would mean that we do
> >> >> _not_
> >> >> > > allow
> >> >> > > > auto topic creation; when set to true, the server configuration
> >> will
> >> >> > > > determine whether we allow automatic creation or not.*
> >> >> > > >
> >> >> > > > Sorry, I meant suppress.auto.create.topics=true above to
> disallow
> >> >> auto
> >> >> > > > topic creation.
> >> >> > > >
> >> >> > > >
> >> >> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <
> >> dhru...@confluent.io
> >> >> >
> >> >> > > wrote:
> >> >> > > >
> >> >> > > >> To be clear, we will allow auto topic creation only when
> server
> >> >> config
> >> >> > > >> auto.create.topics.enable=true and consumer config
> >> >> > > >> allow.auto.create.topics=true; when either is false, we would
> >> not
> >> >> > create
> >> >> > > >> the topic if it does not exist.
> >> >> > > >>
> >> >> > > >> "suppress" is the opposite of "allow", so
> >> >> > > >> setting suppress.auto.create.topics=false would mean that we
> do
> >> >> _not_
> >> >> > > allow
> >> >> > > >> auto topic creation; when set to true, the server
> configuration
> >> >> will
> >> >> > > >> determine whether we allow automatic creation or not.
> >> >> > > >>
> >> >> > > >> I think "allow" is easier to understand but I am open to
> >> >> suggestions.
> >> >> > > >>
> >> >> > > >> - Dhruvil
> >> >> > > >>
> >> >> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> >> >> > > >> brandon.kirch...@gmail.com> wrote:
> >> >> > > >>
> >> >> > > >>> “allow=false” seems a bit more intuitive to me than
> >> >> “suppress=false”
> >> >> > > >>>
> >> >> > > >>> Brandon
> >> >> > > >>>
> >> >> > >  On Aug 22, 2018, at 8:48 PM, Ted Yu 
> >> wrote:
> >> >> > > 
> >> >> > >  We may also consider :
> >> >> > > 
> >> >> > >  "suppress.auto.topic.creation"
> >> >> > > 
> >> >> > >  or
> >> >> > > 
> >> >> > >  "allow.auto.topic.creation"
> >> >> > > 
> >> >> > >  w.r.t. suppress or allow, I don't have strong opinion
> either.
> >> >> It's
> >> >> > > just
> >> >> > > >>> a
> >> >> > >  matter of choosing the proper default value.
> >> >> > > 

[jira] [Resolved] (KAFKA-6309) add support for getting topic defaults from AdminClient

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6309.
--
Resolution: Fixed

Closing as per above comment. We can use "describeConfigs(brokerResource, new 
DescribeConfigsOptions().includeSynonyms(true))" to list all the configured 
values and default configs in synonyms. Please reopen if you think the issue 
still exists

> add support for getting topic defaults from AdminClient
> ---
>
> Key: KAFKA-6309
> URL: https://issues.apache.org/jira/browse/KAFKA-6309
> Project: Kafka
>  Issue Type: Improvement
>Reporter: dan norwood
>Assignee: dan norwood
>Priority: Major
>
> kip here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-234%3A+add+support+for+getting+topic+defaults+from+AdminClient



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Andrew Otto]

Ah, I just realized that as proposed this is only for the Java consumer
client, correct?  Would it be possible to make this a broker config, like
the current one?  Something like:

auto.create.topics.enable=true # allow both producer and consumer to create
auto.create.topics.enable=consumer # allow only consumer to create
auto.create.topics.enable=producer # allow only producer to create
auto.create.topics.enable=false # deny any auto topic creation

Perhaps the broker doesn’t differentiate between the type of client
connection. If not, I guess this wouldn’t be possible.



On Thu, Aug 23, 2018 at 11:08 AM Andrew Otto  wrote:

> Yup :)
>
> On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma  wrote:
>
>> Andrew, one question: you are relying on auto topic creation for the
>> producer and that's why you can't just disable it?
>>
>> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma  wrote:
>>
>> > Thanks for sharing Andrew!
>> >
>> > Ismael
>> >
>> > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto  wrote:
>> >
>> >> We recently had a pretty serious Kafka outage
>> >> <
>> >>
>> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
>> >> >
>> >> caused by a bug in one of our consumers that caused it to create new
>> >> topics
>> >> in an infinite loop AKA a topic bomb!  Having consumers restricted from
>> >> creating topics would have prevented this for us.
>> >>
>> >> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma  wrote:
>> >>
>> >> > Generally, I think positive configs (`allow` instead of `suppress`)
>> are
>> >> > easier to understand.
>> >> >
>> >> > Ismael
>> >> >
>> >> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <
>> matth...@confluent.io
>> >> >
>> >> > wrote:
>> >> >
>> >> > > Thanks for the summary!
>> >> > >
>> >> > > We might want to add a diagram/table to the docs when we add this
>> >> > > feature (with whatever config name we choose) to explain how broker
>> >> > > config `auto.create.topics.enable` and the consumer config work
>> >> together.
>> >> > >
>> >> > > I think both options are equally easy to understand. "allow" means
>> >> > > follow the broker config, while "suppress" implies ignore the
>> broker
>> >> > > config and don't auto-create.
>> >> > >
>> >> > >
>> >> > > -Matthias
>> >> > >
>> >> > >
>> >> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
>> >> > > > *"suppress" is the opposite of "allow", so
>> >> > > > setting suppress.auto.create.topics=false would mean that we do
>> >> _not_
>> >> > > allow
>> >> > > > auto topic creation; when set to true, the server configuration
>> will
>> >> > > > determine whether we allow automatic creation or not.*
>> >> > > >
>> >> > > > Sorry, I meant suppress.auto.create.topics=true above to disallow
>> >> auto
>> >> > > > topic creation.
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <
>> dhru...@confluent.io
>> >> >
>> >> > > wrote:
>> >> > > >
>> >> > > >> To be clear, we will allow auto topic creation only when server
>> >> config
>> >> > > >> auto.create.topics.enable=true and consumer config
>> >> > > >> allow.auto.create.topics=true; when either is false, we would
>> not
>> >> > create
>> >> > > >> the topic if it does not exist.
>> >> > > >>
>> >> > > >> "suppress" is the opposite of "allow", so
>> >> > > >> setting suppress.auto.create.topics=false would mean that we do
>> >> _not_
>> >> > > allow
>> >> > > >> auto topic creation; when set to true, the server configuration
>> >> will
>> >> > > >> determine whether we allow automatic creation or not.
>> >> > > >>
>> >> > > >> I think "allow" is easier to understand but I am open to
>> >> suggestions.
>> >> > > >>
>> >> > > >> - Dhruvil
>> >> > > >>
>> >> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
>> >> > > >> brandon.kirch...@gmail.com> wrote:
>> >> > > >>
>> >> > > >>> “allow=false” seems a bit more intuitive to me than
>> >> “suppress=false”
>> >> > > >>>
>> >> > > >>> Brandon
>> >> > > >>>
>> >> > >  On Aug 22, 2018, at 8:48 PM, Ted Yu 
>> wrote:
>> >> > > 
>> >> > >  We may also consider :
>> >> > > 
>> >> > >  "suppress.auto.topic.creation"
>> >> > > 
>> >> > >  or
>> >> > > 
>> >> > >  "allow.auto.topic.creation"
>> >> > > 
>> >> > >  w.r.t. suppress or allow, I don't have strong opinion either.
>> >> It's
>> >> > > just
>> >> > > >>> a
>> >> > >  matter of choosing the proper default value.
>> >> > > 
>> >> > >  Cheers
>> >> > > 
>> >> > > > On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah <
>> >> dhru...@confluent.io
>> >> > >
>> >> > > >>> wrote:
>> >> > > >
>> >> > > > Hi Matthias,
>> >> > > >
>> >> > > > Do you mean something like "suppress.auto.create.topic"? I am
>> >> > leaning
>> >> > > >>> a bit
>> >> > > > towards "allow.auto.create.topics" but I don't have a strong
>> >> > > preference
>> >> > > > either. Let's wait to hear if anyone else has an opinion on
>> >> this.
>> >> > > >
>> >> > > > Thanks,
>> >> > > 

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Andrew Otto]

Yup :)

On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma  wrote:

> Andrew, one question: you are relying on auto topic creation for the
> producer and that's why you can't just disable it?
>
> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma  wrote:
>
> > Thanks for sharing Andrew!
> >
> > Ismael
> >
> > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto  wrote:
> >
> >> We recently had a pretty serious Kafka outage
> >> <
> >>
> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
> >> >
> >> caused by a bug in one of our consumers that caused it to create new
> >> topics
> >> in an infinite loop AKA a topic bomb!  Having consumers restricted from
> >> creating topics would have prevented this for us.
> >>
> >> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma  wrote:
> >>
> >> > Generally, I think positive configs (`allow` instead of `suppress`)
> are
> >> > easier to understand.
> >> >
> >> > Ismael
> >> >
> >> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <
> matth...@confluent.io
> >> >
> >> > wrote:
> >> >
> >> > > Thanks for the summary!
> >> > >
> >> > > We might want to add a diagram/table to the docs when we add this
> >> > > feature (with whatever config name we choose) to explain how broker
> >> > > config `auto.create.topics.enable` and the consumer config work
> >> together.
> >> > >
> >> > > I think both options are equally easy to understand. "allow" means
> >> > > follow the broker config, while "suppress" implies ignore the broker
> >> > > config and don't auto-create.
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> >> > >
> >> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> >> > > > *"suppress" is the opposite of "allow", so
> >> > > > setting suppress.auto.create.topics=false would mean that we do
> >> _not_
> >> > > allow
> >> > > > auto topic creation; when set to true, the server configuration
> will
> >> > > > determine whether we allow automatic creation or not.*
> >> > > >
> >> > > > Sorry, I meant suppress.auto.create.topics=true above to disallow
> >> auto
> >> > > > topic creation.
> >> > > >
> >> > > >
> >> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <
> dhru...@confluent.io
> >> >
> >> > > wrote:
> >> > > >
> >> > > >> To be clear, we will allow auto topic creation only when server
> >> config
> >> > > >> auto.create.topics.enable=true and consumer config
> >> > > >> allow.auto.create.topics=true; when either is false, we would not
> >> > create
> >> > > >> the topic if it does not exist.
> >> > > >>
> >> > > >> "suppress" is the opposite of "allow", so
> >> > > >> setting suppress.auto.create.topics=false would mean that we do
> >> _not_
> >> > > allow
> >> > > >> auto topic creation; when set to true, the server configuration
> >> will
> >> > > >> determine whether we allow automatic creation or not.
> >> > > >>
> >> > > >> I think "allow" is easier to understand but I am open to
> >> suggestions.
> >> > > >>
> >> > > >> - Dhruvil
> >> > > >>
> >> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> >> > > >> brandon.kirch...@gmail.com> wrote:
> >> > > >>
> >> > > >>> “allow=false” seems a bit more intuitive to me than
> >> “suppress=false”
> >> > > >>>
> >> > > >>> Brandon
> >> > > >>>
> >> > >  On Aug 22, 2018, at 8:48 PM, Ted Yu 
> wrote:
> >> > > 
> >> > >  We may also consider :
> >> > > 
> >> > >  "suppress.auto.topic.creation"
> >> > > 
> >> > >  or
> >> > > 
> >> > >  "allow.auto.topic.creation"
> >> > > 
> >> > >  w.r.t. suppress or allow, I don't have strong opinion either.
> >> It's
> >> > > just
> >> > > >>> a
> >> > >  matter of choosing the proper default value.
> >> > > 
> >> > >  Cheers
> >> > > 
> >> > > > On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah <
> >> dhru...@confluent.io
> >> > >
> >> > > >>> wrote:
> >> > > >
> >> > > > Hi Matthias,
> >> > > >
> >> > > > Do you mean something like "suppress.auto.create.topic"? I am
> >> > leaning
> >> > > >>> a bit
> >> > > > towards "allow.auto.create.topics" but I don't have a strong
> >> > > preference
> >> > > > either. Let's wait to hear if anyone else has an opinion on
> >> this.
> >> > > >
> >> > > > Thanks,
> >> > > > Dhruvil
> >> > > >
> >> > > > On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
> >> > > matth...@confluent.io
> >> > > 
> >> > > > wrote:
> >> > > >
> >> > > >> Thanks for the KIP Dhruvil!
> >> > > >>
> >> > > >> I agree with Jason's comment. An alternative might be to use
> >> > > >>> "suppress"
> >> > > >> what would revert the logic of "allow". Not sure which one is
> >> more
> >> > > >> intuitive and I am fine with both (no personal preference).
> >> Just
> >> > > >>> wanted
> >> > > >> to mention it as an alternative.
> >> > > >>
> >> > > >> Don't have any further comments/question so far.
> >> > > >>
> >> > > >>
> >> > > >> -Matthias
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > 

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Ismael Juma]

Andrew, one question: you are relying on auto topic creation for the
producer and that's why you can't just disable it?

On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma  wrote:

> Thanks for sharing Andrew!
>
> Ismael
>
> On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto  wrote:
>
>> We recently had a pretty serious Kafka outage
>> <
>> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
>> >
>> caused by a bug in one of our consumers that caused it to create new
>> topics
>> in an infinite loop AKA a topic bomb!  Having consumers restricted from
>> creating topics would have prevented this for us.
>>
>> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma  wrote:
>>
>> > Generally, I think positive configs (`allow` instead of `suppress`) are
>> > easier to understand.
>> >
>> > Ismael
>> >
>> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax > >
>> > wrote:
>> >
>> > > Thanks for the summary!
>> > >
>> > > We might want to add a diagram/table to the docs when we add this
>> > > feature (with whatever config name we choose) to explain how broker
>> > > config `auto.create.topics.enable` and the consumer config work
>> together.
>> > >
>> > > I think both options are equally easy to understand. "allow" means
>> > > follow the broker config, while "suppress" implies ignore the broker
>> > > config and don't auto-create.
>> > >
>> > >
>> > > -Matthias
>> > >
>> > >
>> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
>> > > > *"suppress" is the opposite of "allow", so
>> > > > setting suppress.auto.create.topics=false would mean that we do
>> _not_
>> > > allow
>> > > > auto topic creation; when set to true, the server configuration will
>> > > > determine whether we allow automatic creation or not.*
>> > > >
>> > > > Sorry, I meant suppress.auto.create.topics=true above to disallow
>> auto
>> > > > topic creation.
>> > > >
>> > > >
>> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah > >
>> > > wrote:
>> > > >
>> > > >> To be clear, we will allow auto topic creation only when server
>> config
>> > > >> auto.create.topics.enable=true and consumer config
>> > > >> allow.auto.create.topics=true; when either is false, we would not
>> > create
>> > > >> the topic if it does not exist.
>> > > >>
>> > > >> "suppress" is the opposite of "allow", so
>> > > >> setting suppress.auto.create.topics=false would mean that we do
>> _not_
>> > > allow
>> > > >> auto topic creation; when set to true, the server configuration
>> will
>> > > >> determine whether we allow automatic creation or not.
>> > > >>
>> > > >> I think "allow" is easier to understand but I am open to
>> suggestions.
>> > > >>
>> > > >> - Dhruvil
>> > > >>
>> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
>> > > >> brandon.kirch...@gmail.com> wrote:
>> > > >>
>> > > >>> “allow=false” seems a bit more intuitive to me than
>> “suppress=false”
>> > > >>>
>> > > >>> Brandon
>> > > >>>
>> > >  On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:
>> > > 
>> > >  We may also consider :
>> > > 
>> > >  "suppress.auto.topic.creation"
>> > > 
>> > >  or
>> > > 
>> > >  "allow.auto.topic.creation"
>> > > 
>> > >  w.r.t. suppress or allow, I don't have strong opinion either.
>> It's
>> > > just
>> > > >>> a
>> > >  matter of choosing the proper default value.
>> > > 
>> > >  Cheers
>> > > 
>> > > > On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah <
>> dhru...@confluent.io
>> > >
>> > > >>> wrote:
>> > > >
>> > > > Hi Matthias,
>> > > >
>> > > > Do you mean something like "suppress.auto.create.topic"? I am
>> > leaning
>> > > >>> a bit
>> > > > towards "allow.auto.create.topics" but I don't have a strong
>> > > preference
>> > > > either. Let's wait to hear if anyone else has an opinion on
>> this.
>> > > >
>> > > > Thanks,
>> > > > Dhruvil
>> > > >
>> > > > On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
>> > > matth...@confluent.io
>> > > 
>> > > > wrote:
>> > > >
>> > > >> Thanks for the KIP Dhruvil!
>> > > >>
>> > > >> I agree with Jason's comment. An alternative might be to use
>> > > >>> "suppress"
>> > > >> what would revert the logic of "allow". Not sure which one is
>> more
>> > > >> intuitive and I am fine with both (no personal preference).
>> Just
>> > > >>> wanted
>> > > >> to mention it as an alternative.
>> > > >>
>> > > >> Don't have any further comments/question so far.
>> > > >>
>> > > >>
>> > > >> -Matthias
>> > > >>
>> > > >>
>> > > >>
>> > > >>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
>> > > >>> Hey Dhruvil,
>> > > >>>
>> > > >>> I would suggest using the verb "allow" rather than "enable.
>> The
>> > > > consumer
>> > > >>> cannot enable auto topic creation because it is configured on
>> the
>> > > > broker.
>> > > >>> All it can do is prevent it from happening if it is enabled.
>> > > >>>
>> > > >>> -Jason
>> > > 

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Ismael Juma]

Thanks for sharing Andrew!

Ismael

On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto  wrote:

> We recently had a pretty serious Kafka outage
> <
> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
> >
> caused by a bug in one of our consumers that caused it to create new topics
> in an infinite loop AKA a topic bomb!  Having consumers restricted from
> creating topics would have prevented this for us.
>
> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma  wrote:
>
> > Generally, I think positive configs (`allow` instead of `suppress`) are
> > easier to understand.
> >
> > Ismael
> >
> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax 
> > wrote:
> >
> > > Thanks for the summary!
> > >
> > > We might want to add a diagram/table to the docs when we add this
> > > feature (with whatever config name we choose) to explain how broker
> > > config `auto.create.topics.enable` and the consumer config work
> together.
> > >
> > > I think both options are equally easy to understand. "allow" means
> > > follow the broker config, while "suppress" implies ignore the broker
> > > config and don't auto-create.
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> > > > *"suppress" is the opposite of "allow", so
> > > > setting suppress.auto.create.topics=false would mean that we do _not_
> > > allow
> > > > auto topic creation; when set to true, the server configuration will
> > > > determine whether we allow automatic creation or not.*
> > > >
> > > > Sorry, I meant suppress.auto.create.topics=true above to disallow
> auto
> > > > topic creation.
> > > >
> > > >
> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah 
> > > wrote:
> > > >
> > > >> To be clear, we will allow auto topic creation only when server
> config
> > > >> auto.create.topics.enable=true and consumer config
> > > >> allow.auto.create.topics=true; when either is false, we would not
> > create
> > > >> the topic if it does not exist.
> > > >>
> > > >> "suppress" is the opposite of "allow", so
> > > >> setting suppress.auto.create.topics=false would mean that we do
> _not_
> > > allow
> > > >> auto topic creation; when set to true, the server configuration will
> > > >> determine whether we allow automatic creation or not.
> > > >>
> > > >> I think "allow" is easier to understand but I am open to
> suggestions.
> > > >>
> > > >> - Dhruvil
> > > >>
> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> > > >> brandon.kirch...@gmail.com> wrote:
> > > >>
> > > >>> “allow=false” seems a bit more intuitive to me than
> “suppress=false”
> > > >>>
> > > >>> Brandon
> > > >>>
> > >  On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:
> > > 
> > >  We may also consider :
> > > 
> > >  "suppress.auto.topic.creation"
> > > 
> > >  or
> > > 
> > >  "allow.auto.topic.creation"
> > > 
> > >  w.r.t. suppress or allow, I don't have strong opinion either. It's
> > > just
> > > >>> a
> > >  matter of choosing the proper default value.
> > > 
> > >  Cheers
> > > 
> > > > On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah <
> dhru...@confluent.io
> > >
> > > >>> wrote:
> > > >
> > > > Hi Matthias,
> > > >
> > > > Do you mean something like "suppress.auto.create.topic"? I am
> > leaning
> > > >>> a bit
> > > > towards "allow.auto.create.topics" but I don't have a strong
> > > preference
> > > > either. Let's wait to hear if anyone else has an opinion on this.
> > > >
> > > > Thanks,
> > > > Dhruvil
> > > >
> > > > On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
> > > matth...@confluent.io
> > > 
> > > > wrote:
> > > >
> > > >> Thanks for the KIP Dhruvil!
> > > >>
> > > >> I agree with Jason's comment. An alternative might be to use
> > > >>> "suppress"
> > > >> what would revert the logic of "allow". Not sure which one is
> more
> > > >> intuitive and I am fine with both (no personal preference). Just
> > > >>> wanted
> > > >> to mention it as an alternative.
> > > >>
> > > >> Don't have any further comments/question so far.
> > > >>
> > > >>
> > > >> -Matthias
> > > >>
> > > >>
> > > >>
> > > >>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
> > > >>> Hey Dhruvil,
> > > >>>
> > > >>> I would suggest using the verb "allow" rather than "enable. The
> > > > consumer
> > > >>> cannot enable auto topic creation because it is configured on
> the
> > > > broker.
> > > >>> All it can do is prevent it from happening if it is enabled.
> > > >>>
> > > >>> -Jason
> > > >>>
> > > >>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah <
> > > dhru...@confluent.io>
> > > >> wrote:
> > > >>>
> > >  Hi,
> > > 
> > >  I would like to start discussion on KIP-361 that proposes we
> > add a
> > > >> consumer
> > >  configuration to disable auto topic creation.
> > > 
> > > 

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Andrew Otto]

We recently had a pretty serious Kafka outage

caused by a bug in one of our consumers that caused it to create new topics
in an infinite loop AKA a topic bomb!  Having consumers restricted from
creating topics would have prevented this for us.

On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma  wrote:

> Generally, I think positive configs (`allow` instead of `suppress`) are
> easier to understand.
>
> Ismael
>
> On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax 
> wrote:
>
> > Thanks for the summary!
> >
> > We might want to add a diagram/table to the docs when we add this
> > feature (with whatever config name we choose) to explain how broker
> > config `auto.create.topics.enable` and the consumer config work together.
> >
> > I think both options are equally easy to understand. "allow" means
> > follow the broker config, while "suppress" implies ignore the broker
> > config and don't auto-create.
> >
> >
> > -Matthias
> >
> >
> > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> > > *"suppress" is the opposite of "allow", so
> > > setting suppress.auto.create.topics=false would mean that we do _not_
> > allow
> > > auto topic creation; when set to true, the server configuration will
> > > determine whether we allow automatic creation or not.*
> > >
> > > Sorry, I meant suppress.auto.create.topics=true above to disallow auto
> > > topic creation.
> > >
> > >
> > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah 
> > wrote:
> > >
> > >> To be clear, we will allow auto topic creation only when server config
> > >> auto.create.topics.enable=true and consumer config
> > >> allow.auto.create.topics=true; when either is false, we would not
> create
> > >> the topic if it does not exist.
> > >>
> > >> "suppress" is the opposite of "allow", so
> > >> setting suppress.auto.create.topics=false would mean that we do _not_
> > allow
> > >> auto topic creation; when set to true, the server configuration will
> > >> determine whether we allow automatic creation or not.
> > >>
> > >> I think "allow" is easier to understand but I am open to suggestions.
> > >>
> > >> - Dhruvil
> > >>
> > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> > >> brandon.kirch...@gmail.com> wrote:
> > >>
> > >>> “allow=false” seems a bit more intuitive to me than “suppress=false”
> > >>>
> > >>> Brandon
> > >>>
> >  On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:
> > 
> >  We may also consider :
> > 
> >  "suppress.auto.topic.creation"
> > 
> >  or
> > 
> >  "allow.auto.topic.creation"
> > 
> >  w.r.t. suppress or allow, I don't have strong opinion either. It's
> > just
> > >>> a
> >  matter of choosing the proper default value.
> > 
> >  Cheers
> > 
> > > On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah  >
> > >>> wrote:
> > >
> > > Hi Matthias,
> > >
> > > Do you mean something like "suppress.auto.create.topic"? I am
> leaning
> > >>> a bit
> > > towards "allow.auto.create.topics" but I don't have a strong
> > preference
> > > either. Let's wait to hear if anyone else has an opinion on this.
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
> > matth...@confluent.io
> > 
> > > wrote:
> > >
> > >> Thanks for the KIP Dhruvil!
> > >>
> > >> I agree with Jason's comment. An alternative might be to use
> > >>> "suppress"
> > >> what would revert the logic of "allow". Not sure which one is more
> > >> intuitive and I am fine with both (no personal preference). Just
> > >>> wanted
> > >> to mention it as an alternative.
> > >>
> > >> Don't have any further comments/question so far.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >>
> > >>
> > >>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
> > >>> Hey Dhruvil,
> > >>>
> > >>> I would suggest using the verb "allow" rather than "enable. The
> > > consumer
> > >>> cannot enable auto topic creation because it is configured on the
> > > broker.
> > >>> All it can do is prevent it from happening if it is enabled.
> > >>>
> > >>> -Jason
> > >>>
> > >>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah <
> > dhru...@confluent.io>
> > >> wrote:
> > >>>
> >  Hi,
> > 
> >  I would like to start discussion on KIP-361 that proposes we
> add a
> > >> consumer
> >  configuration to disable auto topic creation.
> > 
> >  Link to the KIP:
> > 
> > >>
> > >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> >  Configuration+to+Disable+Auto+Topic+Creation
> > 
> >  Suggestions and feedback are welcome!
> > 
> >  Thanks,
> >  Dhruvil
> > 
> > >>>
> > >>
> > >>
> > >
> > >>>
> > >>
> > >
> >
> >
>

Re: [DISCUSS] KIP-358: Migrate Streams API to Duration instead of long ms times

[Nikolay Izhikov]

Hello, Mathias.

Thanks for your feedback.

> Thus, it might make sense to keep old and just add new ones? 

As far as I understand, we will keep old methods anyway to prevent public API 
backward compatibility.
I agree with you, methods that used internally shouldn't be deprecated.

> End users can use the "nicer" new ones, while we can still use the existing 
> ones internally? 
> Not sure if it would be possible to keep the old ones without exposing them 
> as public API?

I think, when we decide to remove methods with `long` from public API, we can 
do the following:

1. Create an interface like `WindowsInternal`.
2. Change Windows to an interface.
3. Create package-private implementation `WindowsImpl`.

```
package org.apache.kafka.streams.kstream.internals;
public interface WindowsInternal {
public long start();
public long end();
//etc...
}

package org.apache.kafka.streams.kstream;
public interface Windows {
public Instant start();
public Instant end();
//...
}

class WindowsImpl implements Windows, 
WindowsInternal {

}
```

So, in public API we will expose only `Windows` interface and internally we can 
use `WindowsInternal`
But, of course, this will be huge changes in public API.

> Let me know what you think about this.

I think in this KIP we shouldn't deprecate methods, that are used internally.
I changed it, now my proposal is just add new methods.

Please, let me know if anything more need to be done.

В Ср, 22/08/2018 в 17:29 -0700, Matthias J. Sax пишет:
> Thanks a lot for the KIP.
> 
> From my understanding, the idea of the KIP is to improve the public API
> at DSL level. However, not all public methods listed are part of DSL
> level API, but part of runtime API. Those methods are called during
> processing and are on the hot code path. I am not sure, if we want to
> update those methods. We should carefully think about this, and consider
> to keep Long/long type to keep runtime overhead small. Note, that the
> methods I mention are not required to specify a program using the DSL
> and thus is questionable if the DSL API would be improved if we change
> the methods.
> 
> It's unfortunate, that some part of the public API stretch the DSL
> builder part as well as the runtime part...
> 
> This affects the following methods (please double check if I missed any):
> 
>  - Windows#windowsFor()
>  - Window#start()
>  - Window#end()
>  - JoinWindows#windowFor()
>  - SessionWindows#inactivitiyGap()
>  - TimeWindows#windowFor()
>  - UnlimitedWindows#windowFor()
>  - ProcessorContext#schedule()
>  - ReadOnlyWindowStore#fetch() (2x) and #fetchAll()
>  - SessionStore#findSessions() (2x)
> 
> maybe
>  - TimeWindowedDeserializer#getWindowSize() (it's unused atm, but I
> could imagine that it might be use on the hot code path in the furture)
> 
> So methods have "dual" use and might be called externally and internally:
> 
>  - Window#start()
>  - Window#end()
>  - ReadOnlyWindowStore#fetch() (2x) and #fetchAll()
>  - SessionStore#findSessions() (2x)
> 
> Thus, it might make sense to keep old and just add new ones? End users
> can use the "nicer" new ones, while we can still use the existing ones
> internally? Not sure if it would be possible to keep the old ones
> without exposing them as public API?
> 
> Let me know what you think about this.
> 
> 
> -Matthias
> 
> 
> 
> On 8/21/18 11:41 PM, Nikolay Izhikov wrote:
> > Dear, commiters.
> > 
> > Please, pay attention to this KIP and share your opinion.
> > 
> > В Вт, 21/08/2018 в 11:14 -0500, John Roesler пишет:
> > > I'll solicit more reviews. Let's get at least one committer to chime in
> > > before we start a vote (since we need their approval anyway).
> > > -John
> > > 
> > > On Mon, Aug 20, 2018 at 12:39 PM Nikolay Izhikov 
> > > wrote:
> > > 
> > > > Hello, Ted.
> > > > 
> > > > Thanks for the comment.
> > > > 
> > > > I've edit KIP and change proposal to `windowSize`.
> > > > 
> > > > Guys, any other comments?
> > > > 
> > > > 
> > > > В Вс, 19/08/2018 в 14:57 -0700, Ted Yu пишет:
> > > > > bq. // or just Duration windowSize();
> > > > > 
> > > > > +1 to the above choice.
> > > > > The duration is obvious from the return type. For getter methods, we
> > > > 
> > > > don't
> > > > > use get as prefix (as least for new code).
> > > > > 
> > > > > Cheers
> > > > > 
> > > > > On Sun, Aug 19, 2018 at 8:03 AM Nikolay Izhikov 
> > > > 
> > > > wrote:
> > > > > 
> > > > > > Hello, John.
> > > > > > 
> > > > > > Thank you very much for your feedback!
> > > > > > I've addressed all your comments.
> > > > > > Please, see my answers and let my know is anything in KIP [1] needs 
> > > > > > to
> > > > 
> > > > be
> > > > > > improved.
> > > > > > 
> > > > > > > The correct choice is actually "Instant", not> "LocalDateTime"
> > > > > > 
> > > > > > I've changed the methods proposed in KIP [1] to use 

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ismael Juma]

Also, we may consider deprecating the deserialize method that does not take
headers. Yes, it's a convenience, but it also adds confusion.

Ismael

On Thu, Aug 23, 2018 at 6:48 AM Ismael Juma  wrote:

> I think the KIP needs the rejected alternatives section to have more
> detail. For example, another option would be something like the following,
> which works great as long as one overrides one of the methods, but pretty
> bad if one doesn't. :)
>
> default T deserialize(String topic, byte[] data) {
> return deserialize(topic, null, data);
> }
>
> default T deserialize(String topic, Headers headers, byte[] data) { //
> This is the new method
> return deserialize(topic, data);
> }
>
>
> On Thu, Aug 23, 2018 at 3:57 AM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
>
>> Hi Jason,
>>
>> Thanks for the feedback.
>> 1. I chose to return null here because according to the documentation it
>> may return null data, therefore the users of this methods are perpared for
>> getting a null. Thinking of it though it may be better to throw an
>> exception by default because it'd indicate a programming error. However,
>> would that be a backward incompatible change? I'm simply thinking of this
>> because this is a new behavior that we'd introduce but I'm not sure yet if
>> it'd cause problems.
>> Do you think it'd make sense to do the same in `serialize`?
>> 2. Yes, I believe that is covered in KIP-331:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>>
>> Cheers,
>> Viktor
>>
>> On Wed, Aug 22, 2018 at 6:11 PM Jason Gustafson 
>> wrote:
>>
>> > Hey Viktor,
>> >
>> > This is a nice cleanup. Just a couple quick questions:
>> >
>> > 1. Rather than returning null for the default `deserialize(topic,
>> data)`,
>> > would it be better to throw UnsupportedOperationException? I assume that
>> > internally we'll always invoke the api which takes headers. Similarly
>> for
>> > `serialize(topic, data)`.
>> > 2. Would it make sense to have default no-op implementations for
>> > `configure` and `close`?
>> >
>> > Thanks,
>> > Jason
>> >
>> > On Wed, Aug 22, 2018 at 5:27 AM, Satish Duggana <
>> satish.dugg...@gmail.com>
>> > wrote:
>> >
>> > > +1
>> > >
>> > > On Wed, Aug 22, 2018 at 4:45 PM, Ted Yu  wrote:
>> > >
>> > > > +1
>> > > >  Original message From: Kamal Chandraprakash <
>> > > > kamal.chandraprak...@gmail.com> Date: 8/22/18  3:19 AM  (GMT-08:00)
>> > To:
>> > > > dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate
>> > > > ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer
>> > > > +1
>> > > >
>> > > > Thanks for the KIP!
>> > > >
>> > > > On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass <
>> > > > viktorsomo...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi All,
>> > > > >
>> > > > > I'd like to start a vote on this KIP (
>> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
>> > > > action?pageId=87298242)
>> > > > > which aims to refactor ExtendedSerializer/Serializer and
>> > > > > ExtendedDeserializer/Deserializer.
>> > > > >
>> > > > > To summarize what's the motivation:
>> > > > >
>> > > > > When headers were introduced by KIP-82 the ExtendedSerializer and
>> > > > > ExtendedDeserializer classes were created in order to keep
>> interface
>> > > > > compatibility but still add `T deserialize(String topic, Headers
>> > > headers,
>> > > > > byte[] data);` and `byte[] serialize(String topic, Headers
>> headers, T
>> > > > > data);` methods that consume the headers for
>> > > > serialization/deserialization.
>> > > > > The reason for doing so was that Kafka at that time needed be
>> > > compatbile
>> > > > > with Java 7. Since we're not compiling on Java 7 anymore
>> (KAFKA-4423)
>> > > > we'll
>> > > > > try consolidate the way we're using these in a backward compatible
>> > > > fashion:
>> > > > > deprecating the Extended* classes and moving the aforementioned
>> > methods
>> > > > up
>> > > > > in the class hierarchy.
>> > > > >
>> > > > > I'd be happy to get votes or additional feedback on this.
>> > > > >
>> > > > > Viktor
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ismael Juma]

I think the KIP needs the rejected alternatives section to have more
detail. For example, another option would be something like the following,
which works great as long as one overrides one of the methods, but pretty
bad if one doesn't. :)

default T deserialize(String topic, byte[] data) {
return deserialize(topic, null, data);
}

default T deserialize(String topic, Headers headers, byte[] data) { // This
is the new method
return deserialize(topic, data);
}


On Thu, Aug 23, 2018 at 3:57 AM Viktor Somogyi-Vass 
wrote:

> Hi Jason,
>
> Thanks for the feedback.
> 1. I chose to return null here because according to the documentation it
> may return null data, therefore the users of this methods are perpared for
> getting a null. Thinking of it though it may be better to throw an
> exception by default because it'd indicate a programming error. However,
> would that be a backward incompatible change? I'm simply thinking of this
> because this is a new behavior that we'd introduce but I'm not sure yet if
> it'd cause problems.
> Do you think it'd make sense to do the same in `serialize`?
> 2. Yes, I believe that is covered in KIP-331:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>
> Cheers,
> Viktor
>
> On Wed, Aug 22, 2018 at 6:11 PM Jason Gustafson 
> wrote:
>
> > Hey Viktor,
> >
> > This is a nice cleanup. Just a couple quick questions:
> >
> > 1. Rather than returning null for the default `deserialize(topic, data)`,
> > would it be better to throw UnsupportedOperationException? I assume that
> > internally we'll always invoke the api which takes headers. Similarly for
> > `serialize(topic, data)`.
> > 2. Would it make sense to have default no-op implementations for
> > `configure` and `close`?
> >
> > Thanks,
> > Jason
> >
> > On Wed, Aug 22, 2018 at 5:27 AM, Satish Duggana <
> satish.dugg...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > On Wed, Aug 22, 2018 at 4:45 PM, Ted Yu  wrote:
> > >
> > > > +1
> > > >  Original message From: Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> Date: 8/22/18  3:19 AM  (GMT-08:00)
> > To:
> > > > dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate
> > > > ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer
> > > > +1
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass <
> > > > viktorsomo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a vote on this KIP (
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=87298242)
> > > > > which aims to refactor ExtendedSerializer/Serializer and
> > > > > ExtendedDeserializer/Deserializer.
> > > > >
> > > > > To summarize what's the motivation:
> > > > >
> > > > > When headers were introduced by KIP-82 the ExtendedSerializer and
> > > > > ExtendedDeserializer classes were created in order to keep
> interface
> > > > > compatibility but still add `T deserialize(String topic, Headers
> > > headers,
> > > > > byte[] data);` and `byte[] serialize(String topic, Headers
> headers, T
> > > > > data);` methods that consume the headers for
> > > > serialization/deserialization.
> > > > > The reason for doing so was that Kafka at that time needed be
> > > compatbile
> > > > > with Java 7. Since we're not compiling on Java 7 anymore
> (KAFKA-4423)
> > > > we'll
> > > > > try consolidate the way we're using these in a backward compatible
> > > > fashion:
> > > > > deprecating the Extended* classes and moving the aforementioned
> > methods
> > > > up
> > > > > in the class hierarchy.
> > > > >
> > > > > I'd be happy to get votes or additional feedback on this.
> > > > >
> > > > > Viktor
> > > > >
> > > >
> > >
> >
>

Re: Current Kafka Steams and KSQL Performance Metrics / Benchmarks?

[Adam Bellemare]

Thanks Guozhang!

I am asking primarily because I have seen Flink & Spark Streaming users
boasting of millions of records / second being processed and was interested
to learn where Kafka Streams / KSQL stands. This would also help a lot in
capability planning for teams looking to use Kafka Streams. Is there a
general rule of thumb for upper performance on a Kafka Streams app, for
say, single stream to table join?


As for KIP-213... I need something else to start taking a look at while it
awaits review :).

Thanks, I'll look into what you posted.

Adam


On Wed, Aug 22, 2018 at 7:42 PM, Guozhang Wang  wrote:

> Hello Adam,
>
> Thanks for your interests in working on Kafka Streams / KSQL potential
> performance improvements (I thought the non-key joining will take most of
> your time :P )
>
> Currently there is no published performance numbers for latest versions of
> Streams AFAIK. Personally I ran the Streams SimpleBenchmark (
> https://github.com/apache/kafka/blob/trunk/tests/
> kafkatest/benchmarks/streams/streams_simple_benchmark_test.py) and profile
> it if necessary trying to figure out the performance bottlenecks. If you
> are interested you can follow similar approaches, there are also some JIRAs
> open for potential performance improvements as well:
>
> https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%
> 20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%
> 20AND%20component%20%3D%20%22streams%22%20%20AND%
> 20labels%20%3D%20performance%20%20
>
>
> Guozhang
>
> On Wed, Aug 22, 2018 at 7:02 AM, Adam Bellemare 
> wrote:
>
> > Blog post in question:
> > https://www.confluent.io/blog/ksql-february-release-streamin
> > g-sql-for-apache-kafka/
> >
> > On Wed, Aug 22, 2018 at 10:01 AM, Adam Bellemare <
> adam.bellem...@gmail.com
> > >
> > wrote:
> >
> > > Hi All
> > >
> > > I am looking for performance metrics related to Kafka Streams and
> KSQL. I
> > > have been scouring various blogs, including the confluent one, looking
> > for
> > > any current performance metrics or benchmarks, official or otherwise,
> on
> > > both Kafka Streams and KSQL for Kafka 2.x +. Unfortunately, almost
> > > everything I am finding is 0.x.
> > >
> > > In this particular blog post on KSQL, there is the following quotation:
> > >
> > > > For example, our soak testing cluster has racked up over 1,000 hours
> > > and runs KSQL workloads 24×7. The performance tests we conduct allow us
> > to
> > > understand performance characteristics of stateless and stateful KSQL
> > > queries. We currently run over 42 different tests that collect more
> than
> > > 700 metrics.
> > >
> > > I assume that there is also some information related to Kafka Streams
> in
> > > similar tests. Does anyone know where I can find these results? Or does
> > > anyone have any blog posts or other materials that look at the
> > performance
> > > of either one of these for Kafka 2.x ?
> > >
> > > For context, I am asking this question to get a better understanding of
> > > current Kafka Streams / KSQL performance, such that contributors can
> > > understand the prioritization of performance-related improvements vs.
> > > feature-related improvements.
> > >
> > > Thanks
> > > Adam
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Viktor Somogyi-Vass]

Hi Jason,

Thanks for the feedback.
1. I chose to return null here because according to the documentation it
may return null data, therefore the users of this methods are perpared for
getting a null. Thinking of it though it may be better to throw an
exception by default because it'd indicate a programming error. However,
would that be a backward incompatible change? I'm simply thinking of this
because this is a new behavior that we'd introduce but I'm not sure yet if
it'd cause problems.
Do you think it'd make sense to do the same in `serialize`?
2. Yes, I believe that is covered in KIP-331:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde

Cheers,
Viktor

On Wed, Aug 22, 2018 at 6:11 PM Jason Gustafson  wrote:

> Hey Viktor,
>
> This is a nice cleanup. Just a couple quick questions:
>
> 1. Rather than returning null for the default `deserialize(topic, data)`,
> would it be better to throw UnsupportedOperationException? I assume that
> internally we'll always invoke the api which takes headers. Similarly for
> `serialize(topic, data)`.
> 2. Would it make sense to have default no-op implementations for
> `configure` and `close`?
>
> Thanks,
> Jason
>
> On Wed, Aug 22, 2018 at 5:27 AM, Satish Duggana 
> wrote:
>
> > +1
> >
> > On Wed, Aug 22, 2018 at 4:45 PM, Ted Yu  wrote:
> >
> > > +1
> > >  Original message From: Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> Date: 8/22/18  3:19 AM  (GMT-08:00)
> To:
> > > dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate
> > > ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer
> > > +1
> > >
> > > Thanks for the KIP!
> > >
> > > On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'd like to start a vote on this KIP (
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=87298242)
> > > > which aims to refactor ExtendedSerializer/Serializer and
> > > > ExtendedDeserializer/Deserializer.
> > > >
> > > > To summarize what's the motivation:
> > > >
> > > > When headers were introduced by KIP-82 the ExtendedSerializer and
> > > > ExtendedDeserializer classes were created in order to keep interface
> > > > compatibility but still add `T deserialize(String topic, Headers
> > headers,
> > > > byte[] data);` and `byte[] serialize(String topic, Headers headers, T
> > > > data);` methods that consume the headers for
> > > serialization/deserialization.
> > > > The reason for doing so was that Kafka at that time needed be
> > compatbile
> > > > with Java 7. Since we're not compiling on Java 7 anymore (KAFKA-4423)
> > > we'll
> > > > try consolidate the way we're using these in a backward compatible
> > > fashion:
> > > > deprecating the Extended* classes and moving the aforementioned
> methods
> > > up
> > > > in the class hierarchy.
> > > >
> > > > I'd be happy to get votes or additional feedback on this.
> > > >
> > > > Viktor
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Ismael Juma]

Generally, I think positive configs (`allow` instead of `suppress`) are
easier to understand.

Ismael

On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax 
wrote:

> Thanks for the summary!
>
> We might want to add a diagram/table to the docs when we add this
> feature (with whatever config name we choose) to explain how broker
> config `auto.create.topics.enable` and the consumer config work together.
>
> I think both options are equally easy to understand. "allow" means
> follow the broker config, while "suppress" implies ignore the broker
> config and don't auto-create.
>
>
> -Matthias
>
>
> On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> > *"suppress" is the opposite of "allow", so
> > setting suppress.auto.create.topics=false would mean that we do _not_
> allow
> > auto topic creation; when set to true, the server configuration will
> > determine whether we allow automatic creation or not.*
> >
> > Sorry, I meant suppress.auto.create.topics=true above to disallow auto
> > topic creation.
> >
> >
> > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah 
> wrote:
> >
> >> To be clear, we will allow auto topic creation only when server config
> >> auto.create.topics.enable=true and consumer config
> >> allow.auto.create.topics=true; when either is false, we would not create
> >> the topic if it does not exist.
> >>
> >> "suppress" is the opposite of "allow", so
> >> setting suppress.auto.create.topics=false would mean that we do _not_
> allow
> >> auto topic creation; when set to true, the server configuration will
> >> determine whether we allow automatic creation or not.
> >>
> >> I think "allow" is easier to understand but I am open to suggestions.
> >>
> >> - Dhruvil
> >>
> >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> >> brandon.kirch...@gmail.com> wrote:
> >>
> >>> “allow=false” seems a bit more intuitive to me than “suppress=false”
> >>>
> >>> Brandon
> >>>
>  On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:
> 
>  We may also consider :
> 
>  "suppress.auto.topic.creation"
> 
>  or
> 
>  "allow.auto.topic.creation"
> 
>  w.r.t. suppress or allow, I don't have strong opinion either. It's
> just
> >>> a
>  matter of choosing the proper default value.
> 
>  Cheers
> 
> > On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah 
> >>> wrote:
> >
> > Hi Matthias,
> >
> > Do you mean something like "suppress.auto.create.topic"? I am leaning
> >>> a bit
> > towards "allow.auto.create.topics" but I don't have a strong
> preference
> > either. Let's wait to hear if anyone else has an opinion on this.
> >
> > Thanks,
> > Dhruvil
> >
> > On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
> matth...@confluent.io
> 
> > wrote:
> >
> >> Thanks for the KIP Dhruvil!
> >>
> >> I agree with Jason's comment. An alternative might be to use
> >>> "suppress"
> >> what would revert the logic of "allow". Not sure which one is more
> >> intuitive and I am fine with both (no personal preference). Just
> >>> wanted
> >> to mention it as an alternative.
> >>
> >> Don't have any further comments/question so far.
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
> >>> Hey Dhruvil,
> >>>
> >>> I would suggest using the verb "allow" rather than "enable. The
> > consumer
> >>> cannot enable auto topic creation because it is configured on the
> > broker.
> >>> All it can do is prevent it from happening if it is enabled.
> >>>
> >>> -Jason
> >>>
> >>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah <
> dhru...@confluent.io>
> >> wrote:
> >>>
>  Hi,
> 
>  I would like to start discussion on KIP-361 that proposes we add a
> >> consumer
>  configuration to disable auto topic creation.
> 
>  Link to the KIP:
> 
> >>
> >
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
>  Configuration+to+Disable+Auto+Topic+Creation
> 
>  Suggestions and feedback are welcome!
> 
>  Thanks,
>  Dhruvil
> 
> >>>
> >>
> >>
> >
> >>>
> >>
> >
>
>

Re: [VOTE] KIP-349 Priorities for Source Topics

[Jan Filipiak]

also:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-353%3A+Improve+Kafka+Streams+Timestamp+Synchronization


On 20.08.2018 15:01, Thomas Becker wrote:

I agree with Jan. A strategy interface for choosing processing order is nice, 
and would hopefully be a step towards getting this in streams.

-Tommy

On Mon, 2018-08-20 at 12:52 +0200, Jan Filipiak wrote:

On 20.08.2018 00:19, Matthias J. Sax wrote:

@Nick: A KIP is only accepted if it got 3 binding votes, ie, votes from

committers. If you close the vote before that, the KIP would not be

accepted. Note that committers need to pay attention to a lot of KIPs

and it can take a while until people can look into it. Thanks for your

understanding.


@Jan: Can you give a little bit more context on your concerns? It's

unclear why you mean atm.

Just saying that we should peek at the Samza approach, it's a much more

powerful abstraction. We can ship a default MessageChooser

that looks at the topics priority.

@Adam: anyone can vote :)




-Matthias


On 8/19/18 9:58 AM, Adam Bellemare wrote:

While I am not sure if I can or can’t vote, my question re: Jan’s comment is, 
“should we be implementing it as Samza does?”


I am not familiar with the drawbacks of the current approach vs how samza does 
it.


On Aug 18, 2018, at 5:06 PM, n...@afshartous.com 
wrote:



I only saw one vote on KIP-349, just checking to see if anyone else would like 
to vote before closing this out.

--

   Nick



On Aug 13, 2018, at 9:19 PM, n...@afshartous.com 
wrote:



Hi All,


Calling for a vote on KIP-349


https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics


--

  Nick











This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

[jira] [Resolved] (KAFKA-6314) Add a tool to delete kafka based consumer offsets for a given group

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6314.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-6275

> Add a tool to delete kafka based consumer offsets for a given group
> ---
>
> Key: KAFKA-6314
> URL: https://issues.apache.org/jira/browse/KAFKA-6314
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer, core, tools
>Reporter: Tom Scott
>Priority: Minor
>
> Add a tool to delete kafka based consumer offsets for a given group similar 
> to the reset tool. It could look something like this:
> kafka-consumer-groups --bootstrap-server localhost:9092 --delete-offsets 
> --group somegroup
> The case for this is as follows:
> 1. Consumer group with id: group1 subscribes to topic1
> 2. The group is stopped 
> 3. The subscription changed to topic2 but the id is kept as group1
> Now the out output of kafka-consumer-groups --describe for the group will 
> show topic1 even though the group is not subscribed to that topic. This is 
> bad for monitoring as it will show lag on topic1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-1921) Delete offsets for a group with kafka offset storage

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1921.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-6275

> Delete offsets for a group with kafka offset storage
> 
>
> Key: KAFKA-1921
> URL: https://issues.apache.org/jira/browse/KAFKA-1921
> Project: Kafka
>  Issue Type: Improvement
>  Components: offset manager
>Reporter: Onur Karaman
>Priority: Major
>
> There is currently no way to delete offsets for a consumer group when using 
> kafka offset storage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7330) Kakfa 0.10.2.1 producer close method issue

[vinay (JIRA)]

vinay created KAFKA-7330:


 Summary: Kakfa 0.10.2.1 producer close method issue
 Key: KAFKA-7330
 URL: https://issues.apache.org/jira/browse/KAFKA-7330
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.2.1
 Environment: linux
Reporter: vinay


We are creating new producer connections each time but in separate threads(pool 
of 40 and all threads will be used most of the time) and closing it when work 
in done. its a java process.

Not always but sometimes close connection takes more than 10 seconds and 
observerd that particular message was never published.

Could you please help.

 

{{Properties prop = new Properties(); }}

{{prop.put("bootstarp.servers",-);}}

{{prop.put("acks","all"); }}

{{//some ssl properties }}

{{--- }}

{{//ends KafkaProducer }}

{{connection = null; }}

{{try { }}

{{connection = new KafkaProducer(props); msg.setTopic(topic); }}

{{msg.setDate(new Date());}}

{{ connection.send(msg);}}

{{ } catch() {}}

{{ } finally { }}

{{connection.close();// sometimes it takes more than 10 secs and the above 
message was not send }}

{{}}}

Producer config :

max.in.flight.requests.per.connection = 5

acks= all

batch.size = 16384

linger.ms = 0

max.block.ms = 6

request.timeout.ms = 5000

retries = 0

---

 

Even  i see below warning most of the times,

Failed to send SSL Close message:

java.io.IOException: Unexpected status returned by SSLEngine.wrap, expected 
CLOSED, received OK. Will not send close message to peer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

[Matthias J. Sax]

Thanks for the summary!

We might want to add a diagram/table to the docs when we add this
feature (with whatever config name we choose) to explain how broker
config `auto.create.topics.enable` and the consumer config work together.

I think both options are equally easy to understand. "allow" means
follow the broker config, while "suppress" implies ignore the broker
config and don't auto-create.


-Matthias


On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> *"suppress" is the opposite of "allow", so
> setting suppress.auto.create.topics=false would mean that we do _not_ allow
> auto topic creation; when set to true, the server configuration will
> determine whether we allow automatic creation or not.*
> 
> Sorry, I meant suppress.auto.create.topics=true above to disallow auto
> topic creation.
> 
> 
> On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah  wrote:
> 
>> To be clear, we will allow auto topic creation only when server config
>> auto.create.topics.enable=true and consumer config
>> allow.auto.create.topics=true; when either is false, we would not create
>> the topic if it does not exist.
>>
>> "suppress" is the opposite of "allow", so
>> setting suppress.auto.create.topics=false would mean that we do _not_ allow
>> auto topic creation; when set to true, the server configuration will
>> determine whether we allow automatic creation or not.
>>
>> I think "allow" is easier to understand but I am open to suggestions.
>>
>> - Dhruvil
>>
>> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
>> brandon.kirch...@gmail.com> wrote:
>>
>>> “allow=false” seems a bit more intuitive to me than “suppress=false”
>>>
>>> Brandon
>>>
 On Aug 22, 2018, at 8:48 PM, Ted Yu  wrote:

 We may also consider :

 "suppress.auto.topic.creation"

 or

 "allow.auto.topic.creation"

 w.r.t. suppress or allow, I don't have strong opinion either. It's just
>>> a
 matter of choosing the proper default value.

 Cheers

> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah 
>>> wrote:
>
> Hi Matthias,
>
> Do you mean something like "suppress.auto.create.topic"? I am leaning
>>> a bit
> towards "allow.auto.create.topics" but I don't have a strong preference
> either. Let's wait to hear if anyone else has an opinion on this.
>
> Thanks,
> Dhruvil
>
> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax >>>
> wrote:
>
>> Thanks for the KIP Dhruvil!
>>
>> I agree with Jason's comment. An alternative might be to use
>>> "suppress"
>> what would revert the logic of "allow". Not sure which one is more
>> intuitive and I am fine with both (no personal preference). Just
>>> wanted
>> to mention it as an alternative.
>>
>> Don't have any further comments/question so far.
>>
>>
>> -Matthias
>>
>>
>>
>>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
>>> Hey Dhruvil,
>>>
>>> I would suggest using the verb "allow" rather than "enable. The
> consumer
>>> cannot enable auto topic creation because it is configured on the
> broker.
>>> All it can do is prevent it from happening if it is enabled.
>>>
>>> -Jason
>>>
>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah 
>> wrote:
>>>
 Hi,

 I would like to start discussion on KIP-361 that proposes we add a
>> consumer
 configuration to disable auto topic creation.

 Link to the KIP:

>>
>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
 Configuration+to+Disable+Auto+Topic+Creation

 Suggestions and feedback are welcome!

 Thanks,
 Dhruvil

>>>
>>
>>
>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature