Re: [VOTE] KIP-475: New Metric to Measure Number of Tasks on a Connector

[Cyrus Vafadari]

Many thanks for the feedback. Per Gwen's suggestion, I've updated the KIP
to specify that the task count will be per-worker (no additional MBean tag,
since each process is a worker) and per-connector (MBean tag).

On Mon, Jun 17, 2019 at 8:24 PM Cyrus Vafadari  wrote:

> I meant to write:
> I've also updated the KIP to clarify that every task must have exactly one
> non-null *status* at all times.
>
> On Mon, Jun 17, 2019 at 6:55 PM Cyrus Vafadari  wrote:
>
>> Guozhang,
>>
>> Both of Kafka's implementations of "StatusBackingStore" immediately
>> delete the task from the backign store when you try to set it to DESTROYED,
>> so we'd actually expect it to always be zero. A nonzero number of destroyed
>> tasks would either indicate a new implementation of StatusBackingStore, or
>> a malfunctioning StatusBackingStore (e.g. caches out of sync with compacted
>> topic). This metric will usually be uninteresting, and was only included
>> for completeness. It could possibly catch a bug.
>>
>> Gwen,
>> I had not considered this option. I agree there is an advantage to having
>> more granular data about both connector and worker. The main disadvantage
>> would be that it increases the number of metrics by a factor of
>> num_workers, but I'd say this is an acceptable tradeoff. Another advantage
>> of your suggestion is that the public interfaces for WorkerConnector would
>> be unchanged, and the new metrics can be added within the Worker class.
>>
>> I've also updated the KIP to clarify that every task must have exactly
>> one non-null task at all times.
>>
>> On Mon, Jun 17, 2019 at 1:41 PM Guozhang Wang  wrote:
>>
>>> Hello Cyrus,
>>>
>>> Thanks for the KIP. I just have one nit question about Connect destroyed
>>> tasks: is it an ever-increasing number? If yes, the corresponding metric
>>> value would be increasing indefinitely as well. Is that intentional?
>>>
>>> Otherwise, lgtm.
>>>
>>>
>>> Guozhang
>>>
>>> On Mon, Jun 17, 2019 at 1:14 PM Gwen Shapira  wrote:
>>>
>>> > Sorry to join so late, but did we consider a single set of task-count
>>> > metrics and using tags to scope each data point to a specific
>>> > connector and worker (and in the future perhaps also user)?
>>> >
>>> > It will make analysis of the data easier - someone may want to
>>> > breakdown tasks by both worker and connector to detect imbalanced
>>> > assignments.
>>> >
>>> > Are there downsides to this approach?
>>> >
>>> > And a small nit: it will be good to capture in the KIP what are the
>>> > expectations regarding overlap and disjointness of the proposed
>>> > metrics. For example, is running+paused+failed = total? Can a task be
>>> > failed and destroyed and therefore count in 2 of those metrics?
>>> >
>>> > On Thu, Jun 6, 2019 at 12:29 PM Cyrus Vafadari 
>>> wrote:
>>> > >
>>> > > Konstantine,
>>> > >
>>> > > This is a good suggestion. Since the suggestion to add 2 additional
>>> > > statuses analogous to the 3 proposed, it is a very minor change of no
>>> > > structural consequence to the KIP.
>>> > >
>>> > > I've updated the KIP to incorporate your suggestion, and any voters
>>> who
>>> > > disagree should definitely respond in the thread.
>>> > >
>>> > > Cyrus
>>> > >
>>> > > On Thu, Jun 6, 2019 at 11:16 AM Konstantine Karantasis <
>>> > > konstant...@confluent.io> wrote:
>>> > >
>>> > > > Thanks Cyrus,
>>> > > >
>>> > > > this is a nice and straightforward addition.
>>> > > >
>>> > > > I'm +1 too, but I'd like to return with a question here as well
>>> > regarding
>>> > > > whether the unassigned tasks will be taken into account.
>>> > > > Especially after KIP-415 we might start seeing this status for
>>> specific
>>> > > > periods of time. Therefore, I think it's a meaningful addition.
>>> > > > Then there's the `destroyed` status which might be a lot more
>>> > transient but
>>> > > > we could also include for the sake of completion.
>>> > > > Check org.apache.kafka.connect.runtime.AbstractStatus for the list
>>> of
>>> > all
>>> > > > possible statuses.
>>> > > >
>>> > > > Konstantine
>>> > > >
>>> > > > On Wed, Jun 5, 2019 at 4:32 PM Randall Hauch 
>>> wrote:
>>> > > >
>>> > > > > Thanks, Cyrus.
>>> > > > >
>>> > > > > +1 (binding)
>>> > > > >
>>> > > > > Randall Hauch
>>> > > > >
>>> > > > > On Wed, Jun 5, 2019 at 10:36 AM Andrew Schofield <
>>> > > > > andrew_schofi...@live.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > +1 (non-binding)
>>> > > > > >
>>> > > > > > Andrew Schofield
>>> > > > > >
>>> > > > > > On 05/06/2019, 14:04, "Ryanne Dolan" 
>>> > wrote:
>>> > > > > >
>>> > > > > > +1 (non-binding)
>>> > > > > >
>>> > > > > > Thanks
>>> > > > > > Ryanne
>>> > > > > >
>>> > > > > > On Tue, Jun 4, 2019, 11:29 PM Cyrus Vafadari <
>>> > cy...@confluent.io>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi all,
>>> > > > > > >
>>> > > > > > > Like like to start voting in the following KIP:
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> >
>>> https://nam01.safeli

Build failed in Jenkins: kafka-trunk-jdk11 #639

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: rename subscription construction function (#6954)

--
[...truncated 2.87 MB...]

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
PASSED

org.apache.kafka.connect.runtime.StateTrackerTest > calculateDurations STARTED

org.apache.kafka.connect.runtime.StateTrackerTest > calculateDurations PASSED

org.apache.kafka.connect.runtime.StateTrackerTest > currentState STARTED

org.apache.kafka.connect.runtime.StateTrackerTest > currentState PASSED

org.apache.kafka.connect.runtime.StateTrackerTest > 
currentStateIsNullWhenNotInitialized STARTED

org.apache.kafka.connect.runtime.StateTrackerTest > 
currentStateIsNullWhenNotInitialized PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLFirstCancelThenScheduleRestart STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLFirstCancelThenScheduleRestart PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariable STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariable PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLAndScheduleRestart STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLAndScheduleRestart PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTL STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTL PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testTransformNullConfiguration STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testTransformNullConfiguration PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testSaveRestore 
STARTED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testSaveRestore 
PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testGetSet STARTED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testGetSet PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > testStartStop 
STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > testStartStop 
PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutTaskConfigsDoesNotResolveAllInconsistencies STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutTaskConfigsDoesNotResolveAllInconsistencies PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutConnectorConfig STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutTaskConfigs STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutTaskConfigs PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutTaskConfigsZeroTasks STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testPutTaskConfigsZeroTasks PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreTargetState STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreTargetState PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testBackgroundUpdateTargetState STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testBackgroundUpdateTargetState PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testBackgroundConnectorDeletion STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testBackgroundConnectorDeletion PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreTargetStateUnexpectedDeletion STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreTargetStateUnexpectedDeletion PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > testRestore 
STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > testRestore 
PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreConnectorDeletion STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreConnectorDeletion PASSED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreZeroTasks STARTED

org.apache.kafka.connect.storage.KafkaConfigBackingStoreTest > 
testRestoreZeroTasks PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testStartStop 
STARTED

org.apache.kafka.conn

Build failed in Jenkins: kafka-trunk-jdk8 #3730

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-7853: Refactor coordinator config (#6854)

[jason] KAFKA-8539; Add group.instance.id to Subscription (#6936)

[jason] HOTFIX: Fix optional import in ConsumerCoordinator (#6953)

[github] MINOR: Simplify controller election utilities (#6944)

[bbejeck] KAFKA-6958: Overload KTable methods to allow to name operation name

--
[...truncated 5.03 MB...]

org.apache.kafka.connect.json.JsonConverterTest > doubleToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
timestampToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
timestampToConnectOptionalWithDefaultValue PASSED

> Task :streams:examples:test

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test 
STARTED

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test PASSED

> Task :streams:streams-scala:test

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionWithNamedRepartitionTopic STARTED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionWithNamedRepartitionTopic PASSED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionJava STARTED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegionJava PASSED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegion STARTED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes
 > testShouldCountClicksPerRegion PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaJoin STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaJoin PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaSimple STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaSimple PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaAggregate STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaAggregate PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaProperties STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaProperties PASSED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaTransform STARTED

org.apache.kafka.streams.scala.TopologyTest > 
shouldBuildIdenticalTopologyInJavaNScalaTransform PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses PASSED

org.apac

Re: [VOTE] KIP-475: New Metric to Measure Number of Tasks on a Connector

[Cyrus Vafadari]

I meant to write:
I've also updated the KIP to clarify that every task must have exactly one
non-null *status* at all times.

On Mon, Jun 17, 2019 at 6:55 PM Cyrus Vafadari  wrote:

> Guozhang,
>
> Both of Kafka's implementations of "StatusBackingStore" immediately delete
> the task from the backign store when you try to set it to DESTROYED, so
> we'd actually expect it to always be zero. A nonzero number of destroyed
> tasks would either indicate a new implementation of StatusBackingStore, or
> a malfunctioning StatusBackingStore (e.g. caches out of sync with compacted
> topic). This metric will usually be uninteresting, and was only included
> for completeness. It could possibly catch a bug.
>
> Gwen,
> I had not considered this option. I agree there is an advantage to having
> more granular data about both connector and worker. The main disadvantage
> would be that it increases the number of metrics by a factor of
> num_workers, but I'd say this is an acceptable tradeoff. Another advantage
> of your suggestion is that the public interfaces for WorkerConnector would
> be unchanged, and the new metrics can be added within the Worker class.
>
> I've also updated the KIP to clarify that every task must have exactly one
> non-null task at all times.
>
> On Mon, Jun 17, 2019 at 1:41 PM Guozhang Wang  wrote:
>
>> Hello Cyrus,
>>
>> Thanks for the KIP. I just have one nit question about Connect destroyed
>> tasks: is it an ever-increasing number? If yes, the corresponding metric
>> value would be increasing indefinitely as well. Is that intentional?
>>
>> Otherwise, lgtm.
>>
>>
>> Guozhang
>>
>> On Mon, Jun 17, 2019 at 1:14 PM Gwen Shapira  wrote:
>>
>> > Sorry to join so late, but did we consider a single set of task-count
>> > metrics and using tags to scope each data point to a specific
>> > connector and worker (and in the future perhaps also user)?
>> >
>> > It will make analysis of the data easier - someone may want to
>> > breakdown tasks by both worker and connector to detect imbalanced
>> > assignments.
>> >
>> > Are there downsides to this approach?
>> >
>> > And a small nit: it will be good to capture in the KIP what are the
>> > expectations regarding overlap and disjointness of the proposed
>> > metrics. For example, is running+paused+failed = total? Can a task be
>> > failed and destroyed and therefore count in 2 of those metrics?
>> >
>> > On Thu, Jun 6, 2019 at 12:29 PM Cyrus Vafadari 
>> wrote:
>> > >
>> > > Konstantine,
>> > >
>> > > This is a good suggestion. Since the suggestion to add 2 additional
>> > > statuses analogous to the 3 proposed, it is a very minor change of no
>> > > structural consequence to the KIP.
>> > >
>> > > I've updated the KIP to incorporate your suggestion, and any voters
>> who
>> > > disagree should definitely respond in the thread.
>> > >
>> > > Cyrus
>> > >
>> > > On Thu, Jun 6, 2019 at 11:16 AM Konstantine Karantasis <
>> > > konstant...@confluent.io> wrote:
>> > >
>> > > > Thanks Cyrus,
>> > > >
>> > > > this is a nice and straightforward addition.
>> > > >
>> > > > I'm +1 too, but I'd like to return with a question here as well
>> > regarding
>> > > > whether the unassigned tasks will be taken into account.
>> > > > Especially after KIP-415 we might start seeing this status for
>> specific
>> > > > periods of time. Therefore, I think it's a meaningful addition.
>> > > > Then there's the `destroyed` status which might be a lot more
>> > transient but
>> > > > we could also include for the sake of completion.
>> > > > Check org.apache.kafka.connect.runtime.AbstractStatus for the list
>> of
>> > all
>> > > > possible statuses.
>> > > >
>> > > > Konstantine
>> > > >
>> > > > On Wed, Jun 5, 2019 at 4:32 PM Randall Hauch 
>> wrote:
>> > > >
>> > > > > Thanks, Cyrus.
>> > > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > Randall Hauch
>> > > > >
>> > > > > On Wed, Jun 5, 2019 at 10:36 AM Andrew Schofield <
>> > > > > andrew_schofi...@live.com>
>> > > > > wrote:
>> > > > >
>> > > > > > +1 (non-binding)
>> > > > > >
>> > > > > > Andrew Schofield
>> > > > > >
>> > > > > > On 05/06/2019, 14:04, "Ryanne Dolan" 
>> > wrote:
>> > > > > >
>> > > > > > +1 (non-binding)
>> > > > > >
>> > > > > > Thanks
>> > > > > > Ryanne
>> > > > > >
>> > > > > > On Tue, Jun 4, 2019, 11:29 PM Cyrus Vafadari <
>> > cy...@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > Like like to start voting in the following KIP:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> >
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-475%253A%2BNew%2BMetric%2Bto%2BMeasure%2BNumber%2Bof%2BTasks%2Bon%2Ba%2BConnector&data=02%7C01%7C%7C95f8a8ebb4a44882773808d6e9b65983%7C84df9e7fe9f640afb435%7C1%7C0%7C636953366722392496&sdata=vbE%2BjrAapcQ68Vnwh5OkY1FFoOzFHs9rZRaPHlwqxSU%3D&reserved=0
>> > > > > > >
>> > > > > > > Discuss

[jira] [Created] (KAFKA-8551) Comments for connectors() in Herder interface

[Luying Liu (JIRA)]

Luying Liu created KAFKA-8551:
-

 Summary: Comments for connectors() in Herder interface 
 Key: KAFKA-8551
 URL: https://issues.apache.org/jira/browse/KAFKA-8551
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.2.1
Reporter: Luying Liu


There are mistakes in the comments for connectors() in Herder interface.  The 
mistakes are in the  file 
[kafka|https://github.com/apache/kafka]/[connect|https://github.com/apache/kafka/tree/trunk/connect]/[runtime|https://github.com/apache/kafka/tree/trunk/connect/runtime]/[src|https://github.com/apache/kafka/tree/trunk/connect/runtime/src]/[main|https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main]/[java|https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java]/[org|https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java/org]/[apache|https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java/org/apache]/[kafka|https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java/org/apache/kafka]/[connect|https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java/org/apache/kafka/connect]/[runtime|https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime]/*Herder.java.*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 2.3.0 RC2

[Guozhang Wang]

+1 (binding)

Verified release notes, javadoc, and run quick start with scala_2.12 binary.


Guozhang

On Mon, Jun 17, 2019 at 7:47 PM Satish Duggana 
wrote:

> +1 (non-binding)
>
> - Ran testAll/releaseTarGzAll successfully with no failures.
> - Ran through quickstart of core/streams on builds generated from
> 2.3.0-rc2 tag.
> - Ran few internal apps targeting to topics on 3 node cluster.
>
> Thanks,
> Satish.
>
> On Mon, Jun 17, 2019 at 7:25 PM David Arthur 
> wrote:
> >
> > +1 binding
> >
> > Verified signatures, pulled down kafka_2.12-2.3.0 and ran
> producer/consumer
> > perf test scripts.
> >
> > -David
> >
> > On Mon, Jun 17, 2019 at 1:48 AM Vahid Hashemian <
> vahid.hashem...@gmail.com>
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > I also verifies signatures, build from source and tested the Quickstart
> > > successfully on the built binary.
> > >
> > > BTW, I don't see a link to documentation for 2.3. Is there a reason?
> > >
> > > Thanks,
> > > --Vahid
> > >
> > > On Sat, Jun 15, 2019 at 6:38 PM Gwen Shapira 
> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Verified signatures, built from sources, ran quickstart on binary and
> > > > checked out the passing jenkins build on the branch.
> > > >
> > > > Gwen
> > > >
> > > >
> > > > On Thu, Jun 13, 2019 at 11:58 AM Colin McCabe 
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Good news: I have run a junit test build for RC2, and it passed.
> Check
> > > > out https://builds.apache.org/job/kafka-2.3-jdk8/51/
> > > > >
> > > > > Also, the vote will go until Saturday, June 15th (sorry for the
> typo
> > > > earlier in the vote end time).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Wed, Jun 12, 2019, at 15:55, Colin McCabe wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > We discovered some problems with the first release candidate
> (RC1) of
> > > > > > 2.3.0.  Specifically, KAFKA-8484 and KAFKA-8500.  I have created
> a
> > > new
> > > > > > release candidate that includes fixes for these issues.
> > > > > >
> > > > > > Check out the release notes for the 2.3.0 release here:
> > > > > >
> https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/RELEASE_NOTES.html
> > > > > >
> > > > > > The vote will go until Friday, June 7th, or until we create
> another R
> > > > > >
> > > > > > * Kafka's KEYS file containing PGP keys we use to sign the
> release
> > > can
> > > > > > be found here:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * The release artifacts to be voted upon (source and binary) are
> > > here:
> > > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/javadoc/
> > > > > >
> > > > > > * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/2.3.0-rc2
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Gwen Shapira
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter | blog
> > > >
> > >
> > >
> > > --
> > >
> > > Thanks!
> > > --Vahid
> > >
>


-- 
-- Guozhang

Re: [VOTE] 2.3.0 RC2

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll successfully with no failures.
- Ran through quickstart of core/streams on builds generated from 2.3.0-rc2 tag.
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.

On Mon, Jun 17, 2019 at 7:25 PM David Arthur  wrote:
>
> +1 binding
>
> Verified signatures, pulled down kafka_2.12-2.3.0 and ran producer/consumer
> perf test scripts.
>
> -David
>
> On Mon, Jun 17, 2019 at 1:48 AM Vahid Hashemian 
> wrote:
>
> > +1 (non-binding)
> >
> > I also verifies signatures, build from source and tested the Quickstart
> > successfully on the built binary.
> >
> > BTW, I don't see a link to documentation for 2.3. Is there a reason?
> >
> > Thanks,
> > --Vahid
> >
> > On Sat, Jun 15, 2019 at 6:38 PM Gwen Shapira  wrote:
> >
> > > +1 (binding)
> > >
> > > Verified signatures, built from sources, ran quickstart on binary and
> > > checked out the passing jenkins build on the branch.
> > >
> > > Gwen
> > >
> > >
> > > On Thu, Jun 13, 2019 at 11:58 AM Colin McCabe 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Good news: I have run a junit test build for RC2, and it passed.  Check
> > > out https://builds.apache.org/job/kafka-2.3-jdk8/51/
> > > >
> > > > Also, the vote will go until Saturday, June 15th (sorry for the typo
> > > earlier in the vote end time).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Wed, Jun 12, 2019, at 15:55, Colin McCabe wrote:
> > > > > Hi all,
> > > > >
> > > > > We discovered some problems with the first release candidate (RC1) of
> > > > > 2.3.0.  Specifically, KAFKA-8484 and KAFKA-8500.  I have created a
> > new
> > > > > release candidate that includes fixes for these issues.
> > > > >
> > > > > Check out the release notes for the 2.3.0 release here:
> > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/RELEASE_NOTES.html
> > > > >
> > > > > The vote will go until Friday, June 7th, or until we create another R
> > > > >
> > > > > * Kafka's KEYS file containing PGP keys we use to sign the release
> > can
> > > > > be found here:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * The release artifacts to be voted upon (source and binary) are
> > here:
> > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/javadoc/
> > > > >
> > > > > * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> > > > > https://github.com/apache/kafka/releases/tag/2.3.0-rc2
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
> >

Re: [VOTE] KIP-475: New Metric to Measure Number of Tasks on a Connector

[Cyrus Vafadari]

Guozhang,

Both of Kafka's implementations of "StatusBackingStore" immediately delete
the task from the backign store when you try to set it to DESTROYED, so
we'd actually expect it to always be zero. A nonzero number of destroyed
tasks would either indicate a new implementation of StatusBackingStore, or
a malfunctioning StatusBackingStore (e.g. caches out of sync with compacted
topic). This metric will usually be uninteresting, and was only included
for completeness. It could possibly catch a bug.

Gwen,
I had not considered this option. I agree there is an advantage to having
more granular data about both connector and worker. The main disadvantage
would be that it increases the number of metrics by a factor of
num_workers, but I'd say this is an acceptable tradeoff. Another advantage
of your suggestion is that the public interfaces for WorkerConnector would
be unchanged, and the new metrics can be added within the Worker class.

I've also updated the KIP to clarify that every task must have exactly one
non-null task at all times.

On Mon, Jun 17, 2019 at 1:41 PM Guozhang Wang  wrote:

> Hello Cyrus,
>
> Thanks for the KIP. I just have one nit question about Connect destroyed
> tasks: is it an ever-increasing number? If yes, the corresponding metric
> value would be increasing indefinitely as well. Is that intentional?
>
> Otherwise, lgtm.
>
>
> Guozhang
>
> On Mon, Jun 17, 2019 at 1:14 PM Gwen Shapira  wrote:
>
> > Sorry to join so late, but did we consider a single set of task-count
> > metrics and using tags to scope each data point to a specific
> > connector and worker (and in the future perhaps also user)?
> >
> > It will make analysis of the data easier - someone may want to
> > breakdown tasks by both worker and connector to detect imbalanced
> > assignments.
> >
> > Are there downsides to this approach?
> >
> > And a small nit: it will be good to capture in the KIP what are the
> > expectations regarding overlap and disjointness of the proposed
> > metrics. For example, is running+paused+failed = total? Can a task be
> > failed and destroyed and therefore count in 2 of those metrics?
> >
> > On Thu, Jun 6, 2019 at 12:29 PM Cyrus Vafadari 
> wrote:
> > >
> > > Konstantine,
> > >
> > > This is a good suggestion. Since the suggestion to add 2 additional
> > > statuses analogous to the 3 proposed, it is a very minor change of no
> > > structural consequence to the KIP.
> > >
> > > I've updated the KIP to incorporate your suggestion, and any voters who
> > > disagree should definitely respond in the thread.
> > >
> > > Cyrus
> > >
> > > On Thu, Jun 6, 2019 at 11:16 AM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > > Thanks Cyrus,
> > > >
> > > > this is a nice and straightforward addition.
> > > >
> > > > I'm +1 too, but I'd like to return with a question here as well
> > regarding
> > > > whether the unassigned tasks will be taken into account.
> > > > Especially after KIP-415 we might start seeing this status for
> specific
> > > > periods of time. Therefore, I think it's a meaningful addition.
> > > > Then there's the `destroyed` status which might be a lot more
> > transient but
> > > > we could also include for the sake of completion.
> > > > Check org.apache.kafka.connect.runtime.AbstractStatus for the list of
> > all
> > > > possible statuses.
> > > >
> > > > Konstantine
> > > >
> > > > On Wed, Jun 5, 2019 at 4:32 PM Randall Hauch 
> wrote:
> > > >
> > > > > Thanks, Cyrus.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Randall Hauch
> > > > >
> > > > > On Wed, Jun 5, 2019 at 10:36 AM Andrew Schofield <
> > > > > andrew_schofi...@live.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Andrew Schofield
> > > > > >
> > > > > > On 05/06/2019, 14:04, "Ryanne Dolan" 
> > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks
> > > > > > Ryanne
> > > > > >
> > > > > > On Tue, Jun 4, 2019, 11:29 PM Cyrus Vafadari <
> > cy...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Like like to start voting in the following KIP:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-475%253A%2BNew%2BMetric%2Bto%2BMeasure%2BNumber%2Bof%2BTasks%2Bon%2Ba%2BConnector&data=02%7C01%7C%7C95f8a8ebb4a44882773808d6e9b65983%7C84df9e7fe9f640afb435%7C1%7C0%7C636953366722392496&sdata=vbE%2BjrAapcQ68Vnwh5OkY1FFoOzFHs9rZRaPHlwqxSU%3D&reserved=0
> > > > > > >
> > > > > > > Discussion thread:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2Fbf7c92224aa798336c14d7e96ec8f2e3406c61879ec381a50652acfe%40%253Cdev.kafka.apache.org%253E&data=02%7C01%7C%7C95f8a8ebb4a44882773808d6e9b65983%7C84df9e7f

Jenkins build is back to normal : kafka-trunk-jdk11 #638

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk8 #3729

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

[John Roesler]

Thanks for the feedback, Sophie!

I actually felt a little uneasy when I wrote that remark, because it's
not restricted at all in the API, it's just available to you if you
choose to give your stores and context the same parameters. So, I
think your use case is valid, and also perfectly permissable under the
current KIP. Sorry for sowing confusion on my own discussion thread!

I'm not crazy about the package name, either. I went with it only
because there's seemingly nothing special about the new package except
that it can't have the same name as the old one. Otherwise, the
existing "processor" and "Processor" names for the package and class
are perfectly satisfying. Rather than pile on additional semantics, it
seemed cleaner to just add a number to the package name.

This wouldn't be the first project to do something like this... Apache
Commons, for example, has added a "2" to the end of some of their
packages for exactly the same reason.

I'm open to any suggestions. For example, we could do something like
org.apache.kafka.streams.typedprocessor.Processor or
org.apache.kafka.streams.processor.typed.Processor , which would have
just about the same effect. One microscopic thought is that, if
there's another interface in the "processor" package that we wish to
do the same thing to, would _could_ pile it in to "processor2", but we
couldn't do the same if we use a package that has "typed" in the name,
unless that change is _also_ related to types in some way. But this
seems like a very minor concern.

What's your preference?
-John

On Mon, Jun 17, 2019 at 3:56 PM Sophie Blee-Goldman  wrote:
>
> Hey John, thanks for writing this up! I like the proposal but there's one
> point that I think may be too restrictive:
>
> "A processor that happens to use a typed store is actually emitting the
> same types that it is storing."
>
> I can imagine someone could want to leverage this new type safety without
> also limiting how they can interact with/use their store. As an (admittedly
> contrived) example, say you have an input stream of purchases of a certain
> type (entertainment, food, etc), and on seeing a new record you want to
> output how many types of purchase a shopper has made more than 5 purchases
> of in the last month. Your state store will probably be holding some more
> complicated PurchaseHistory object (keyed by user), but your output is just
> a 
>
> I'm also not crazy about "processor2" as the package name ... not sure what
> a better one would be though (something with "typed"?)
>
> On Mon, Jun 17, 2019 at 12:47 PM John Roesler  wrote:
>
> > Hi all,
> >
> > I'd like to propose KIP-478 (https://cwiki.apache.org/confluence/x/2SkLBw
> > ).
> >
> > This proposal would add output type bounds to the Processor interface
> > in Kafka Streams, which enables static checking of a number of useful
> > properties:
> > * A processor B that consumes the output of processor A is actually
> > expecting the same types that processor A produces.
> > * A processor that happens to use a typed store is actually emitting
> > the same types that it is storing.
> > * A processor is simply forwarding the expected types in all code paths.
> > * Processors added via the Streams DSL, which are not permitted to
> > forward results at all are statically prevented from doing so by the
> > compiler
> >
> > Internally, we can use the above properties to achieve a much higher
> > level of confidence in the Streams DSL implementation's correctness.
> > Actually, while doing the POC, I found a few bugs and mistakes, which
> > become structurally impossible with KIP-478.
> >
> > Additionally, the stronger types dramatically improve the
> > self-documentation of our Streams internal implementations, which
> > makes it much easier for new contributors to ramp up with confidence.
> >
> > Thanks so much for your consideration!
> > -John
> >

[jira] [Created] (KAFKA-8550) Connector validation fails with aliased converters

[Chris Egerton (JIRA)]

Chris Egerton created KAFKA-8550:


 Summary: Connector validation fails with aliased converters
 Key: KAFKA-8550
 URL: https://issues.apache.org/jira/browse/KAFKA-8550
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton


During connector config validation, 
[ConfigDef.validateAll(...)|https://github.com/apache/kafka/blob/1ae92914e28919a97520e91bfd0e588d55eb1774/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L497-L513]
 is invoked using a [Connector 
ConfigDef|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java].
 This ConfigDef contains definitions for the [key and value 
converters|https://github.com/apache/kafka/blob/1ae92914e28919a97520e91bfd0e588d55eb1774/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L72-L78],
 which have the type 
[ConfigDef.Type.CLASS|https://github.com/apache/kafka/blob/1ae92914e28919a97520e91bfd0e588d55eb1774/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L167-L168].
 When plugin aliases are used for these configs, an error is encountered and 
the connector configuration is rejected.

This error occurs because 
[Class.forName(...)|https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#forName-java.lang.String-boolean-java.lang.ClassLoader-]
 is used to load the classes for these configs during validation. Even though 
the DelegatingClassLoader used by Connect successfully loads the requested 
class in its 
[loadClass(...)|https://github.com/apache/kafka/blob/1ae92914e28919a97520e91bfd0e588d55eb1774/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/DelegatingClassLoader.java#L358-L376]
 method, some (if not all) implementations of the Java runtime will then 
perform a check in their native Class.forName method to verify that the name of 
the loaded class matches the requested class name. For example, see [this 
section of 
OpenJDK|https://github.com/openjdk/jdk/blob/515e7600df738ebf8c42d38e322def012385e1b9/src/hotspot/share/classfile/systemDictionary.cpp#L1508-L1528]
 that performs the aforementioned check.

A few possible fixes that come to mind include altering the connector 
validation in the AbstractHerder class to not use the 
ConfigDef.validateAll(...) method, or altering the logic used by the ConfigDef 
in its validateAll method for configs of type ConfigDef.Type.CLASS to use 
[ClassLoader.loadClass(...)|https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html#loadClass-java.lang.String-]
 either instead of or in addition to Class.forName(...).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Posted a new article about Kafka Streams

[John Roesler]

Hi all,

Thanks for the heads-up! I've added the article to my reading list.

And thanks for the reminder about your KIP, Paul. I just visited the
discussion and voting threads.

Also, another plug, you might both be interested in my
freshly-proposed KIP-478
(https://cwiki.apache.org/confluence/x/2SkLBw) to add more type safety
to the Processor API.

When I did the POC PR for it, I found it was _super_ useful in
spotting and preventing bugs in the internal code for the Streams DSL.
Maybe you can share some thoughts about whether it would also be
useful as users of the Processor API, what it would need to change to
be useful, or whether I should just propose it as an internal change.

Thanks!
-John

On Mon, Jun 17, 2019 at 1:06 PM Development  wrote:
>
> Hey Paul,
>
> Thank you so much for your input! :)
> Just updated my article about the iterator closing.
>
> Thank you!
>
> Best,
> Daniyar Yeralin
>
> > On Jun 16, 2019, at 4:11 PM, Paul Whalen  wrote:
> >
> > I've only skimmed it so far, but great job!  The community is in serious
> > need of more examples of the Processor API, there really isn't that much
> > out there.
> >
> > One thing I did notice: the iterator you get from kvStore.all() ought to be
> > closed to release resources when you're done with it.  This matters when
> > the underlying store is RocksDB, which as I understand it, allocates
> > additional memory off heap to iterate.  I see this bug everywhere, after
> > writing it many times myself over the course of many months :). it's too
> > bad the API can't be more clear, but I guess there's not a ton you can do
> > in Java.  People think about this kind of thing for DB calls, but when
> > you're using something that's basically a HashMap you really don't think of
> > it at all.
> >
> > Side plug for KIP-401 since you're using the Processor API, it would be
> > interesting to hear on that discussion thread if you find it useful (
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97553756).
> > It seems like there's soft interest, but maybe not yet enough to push it
> > over the finish line.
> >
> > Again, great job!
> >
> > Paul
> >
> > On Fri, Jun 14, 2019 at 10:33 AM Development  wrote:
> >
> >> Bad link:
> >>
> >> https://medium.com/@daniyaryeralin/utilizing-kafka-streams-processor-api-and-implementing-custom-aggregator-6cb23d00eaa7
> >>
> >>> On Jun 14, 2019, at 11:07 AM, Development  wrote:
> >>>
> >>> Hello Kafka Dev community,
> >>>
> >>> I wrote an article on implementing a custom transformer using Processor
> >> API for Kafka Streams!
> >>>
> >> https://medium.com/@daniyaryeralin/utilizing-kafka-streams-processor-api-and-implementing-custom-aggregation-f6a4a6c376be
> >> <
> >> https://medium.com/@daniyaryeralin/utilizing-kafka-streams-processor-api-and-implementing-custom-aggregation-f6a4a6c376be
> >>>
> >>> Feel free to leave a feedback and/or corrections if I wrote something
> >> silly :)
> >>>
> >>> Thank you!
> >>>
> >>> Best,
> >>> Daniyar Yeralin
> >>
> >>
>

Re: [VOTE] KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting

[John Roesler]

I'm +1 (nonbinding) on the current iteration of the proposal.

On Mon, May 27, 2019 at 1:58 PM Paul Whalen  wrote:
>
> I spoke too early a month ago, but I believe the proposal is finalized now
> and ready for voting.
>
> KIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97553756
>
> Discussion:
> https://lists.apache.org/thread.html/600996d83d485f2b8daf45037de64a60cebdfac9b234bf3449b6b753@%3Cdev.kafka.apache.org%3E
>
> Pull request (still a WIP, obviously):
> https://github.com/apache/kafka/pull/6824
>
> Thanks,
> Paul
>
> On Wed, Apr 24, 2019 at 8:00 PM Paul Whalen  wrote:
>
> > Hi all,
> >
> > After some good discussion on and adjustments to KIP-401 (which I renamed
> > slightly for clarity), chatter has died down so I figured I may as well
> > start a vote.
> >
> > KIP:
> > TransformerSupplier/ProcessorSupplier StateStore connecting
> > 
> > Discussion:
> >
> > https://lists.apache.org/thread.html/600996d83d485f2b8daf45037de64a60cebdfac9b234bf3449b6b753@%3Cdev.kafka.apache.org%3E
> >
> > Thanks!
> > Paul
> >

Re: [DISCUSS] KIP-401 TransformerSupplier/ProcessorSupplier enhancements

[John Roesler]

Hey, all,

Sorry I'm late to the party. I meant to read into this KIP before, but
didn't get around to it. I was just reminded when Paul mentioned it in
a different thread. Please feel free to bump a discussion any time it
stalls!

I've just read through the whole discussion so far, and, to echo the
earlier sentiments, the motivation seems very clear. I remember how
hard it was to figure out how to actually wire up a stateful processor
properly the first couple of times. Not a very good user experience.

I looked over the whole conversation to date, as well as the KIP and
the latest PR (https://github.com/apache/kafka/pull/6824). FWIW, The
current approach looks good to me. I was concerned about the "cheat
codes"-style mixin interface. Discoverability would have been a
problem, and it's also not a very normal pattern for Java APIs. It
actually looks a little more like something you'd do with an
annotation.

So the current approach seems good:
* The new interface with a default to return `null` is effectively
shipping the feature flagged "off" (which is nice and safe)
* Shared stores are "supported" the same way they always have been, by
connecting them externally. This makes sense, since those stores
aren't "owned" by any of the connected processors.
* Processors that do own their stores can configure them in the same
file they use them, which decreases the probability of cast exceptions
when they get the stores from the context.
* Stateful processors that own their stores are available for one-shot
definition of the stores and the processor all in the same file (this
is the main point of the KIP)

The runtime check that stores can't be both defined in the processor
and referenced by name might be a little restrictive (since we already
have the restriction that same-name stores can't be registered), but
it would also be easy to remove it later. I'm just thinking that if I
have a processor that owns one store and shares another, it would be
pretty obvious how to hook it up in the proposed API, except for that
check.

One last thought, regarding the all-important interface name: If you
wanted to indicate more that the stores are available for Streams to
connect, rather than that they are already connected, you could call
it ConnectableStoreProvider (similar to AutoCloseable).

I just thought I'd summarize the current state, since it's been a
while and no one has voted yet. I'll go ahead and vote now on the
voting thread, since I'm +1 on the current proposal.

Thanks,
-John

On Mon, May 27, 2019 at 1:59 PM Paul Whalen  wrote:
>
> It wasn't much of a lift changing option B to work for option C, so I
> closed that PR and made a new one, which should be identical to the KIP
> right now: https://github.com/apache/kafka/pull/6824.  There are a few
> todos still which I will hold off until the KIP is accepted.
>
> I created a voting thread about a month ago, so I'll bump that now that
> we're nearly there.
>
> Paul
>
> On Sun, May 26, 2019 at 2:21 PM Paul Whalen  wrote:
>
> > Per Matthias's suggestion from a while ago, I actually implemented a good
> > amount of option B to get a sense of the user experience and documentation
> > requirements.  For a few reasons mentioned below, I think it's not my
> > favorite option, and I prefer option C.  But since I did the work and it
> > can help discussion, I may as well share:
> > https://github.com/apache/kafka/pull/6821.
> >
> > Things I learned along the way implementing Option B:
> >  - For the name of the interface, I like ConnectedStoreProvider.  It isn't
> > perfect but it seems to capture the general gist without being overly
> > verbose.  I get that from a strict standpoint it's not "providing connected
> > stores" but is instead "providing stores to be connected," but I think that
> > in context and with documentation, the risk of someone being confused by
> > that is low.
> >  - I definitely felt the discoverability issue while trying to write clear
> > documentation; you really have to make sure to connect the dots for the
> > user when the interface isn't connected to anything.
> >  - Another problem with a separate interface found while writing
> > tests/examples: defining a ProcessorSupplier that also implements
> > ConnectedStoreProvider cannot be done anonymously, since you can't define
> > an anonymous class in Java that implements multiple interfaces.  I actually
> > consider this a fairly major usability issue - it means a user always has
> > to have a custom class rather than doing it inline.  We could provide an
> > abstract class that implements the two, but at that point, we're not that
> > far from option A or C anyway.
> >
> > I updated the KIP with my current thinking, which as mentioned is
> > Matthias's option C.  Once again for clarity, that *is not* what is in the
> > linked pull request.  The current KIP is my proposal.
> >
> > Thanks everyone for the input!
> >
> > P.S.  What do folks use to edit the HTML documentation, e.g.
> > processor-a

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

[Sophie Blee-Goldman]

Hey John, thanks for writing this up! I like the proposal but there's one
point that I think may be too restrictive:

"A processor that happens to use a typed store is actually emitting the
same types that it is storing."

I can imagine someone could want to leverage this new type safety without
also limiting how they can interact with/use their store. As an (admittedly
contrived) example, say you have an input stream of purchases of a certain
type (entertainment, food, etc), and on seeing a new record you want to
output how many types of purchase a shopper has made more than 5 purchases
of in the last month. Your state store will probably be holding some more
complicated PurchaseHistory object (keyed by user), but your output is just
a 

I'm also not crazy about "processor2" as the package name ... not sure what
a better one would be though (something with "typed"?)

On Mon, Jun 17, 2019 at 12:47 PM John Roesler  wrote:

> Hi all,
>
> I'd like to propose KIP-478 (https://cwiki.apache.org/confluence/x/2SkLBw
> ).
>
> This proposal would add output type bounds to the Processor interface
> in Kafka Streams, which enables static checking of a number of useful
> properties:
> * A processor B that consumes the output of processor A is actually
> expecting the same types that processor A produces.
> * A processor that happens to use a typed store is actually emitting
> the same types that it is storing.
> * A processor is simply forwarding the expected types in all code paths.
> * Processors added via the Streams DSL, which are not permitted to
> forward results at all are statically prevented from doing so by the
> compiler
>
> Internally, we can use the above properties to achieve a much higher
> level of confidence in the Streams DSL implementation's correctness.
> Actually, while doing the POC, I found a few bugs and mistakes, which
> become structurally impossible with KIP-478.
>
> Additionally, the stronger types dramatically improve the
> self-documentation of our Streams internal implementations, which
> makes it much easier for new contributors to ramp up with confidence.
>
> Thanks so much for your consideration!
> -John
>

Re: [VOTE] KIP-475: New Metric to Measure Number of Tasks on a Connector

[Guozhang Wang]

Hello Cyrus,

Thanks for the KIP. I just have one nit question about Connect destroyed
tasks: is it an ever-increasing number? If yes, the corresponding metric
value would be increasing indefinitely as well. Is that intentional?

Otherwise, lgtm.


Guozhang

On Mon, Jun 17, 2019 at 1:14 PM Gwen Shapira  wrote:

> Sorry to join so late, but did we consider a single set of task-count
> metrics and using tags to scope each data point to a specific
> connector and worker (and in the future perhaps also user)?
>
> It will make analysis of the data easier - someone may want to
> breakdown tasks by both worker and connector to detect imbalanced
> assignments.
>
> Are there downsides to this approach?
>
> And a small nit: it will be good to capture in the KIP what are the
> expectations regarding overlap and disjointness of the proposed
> metrics. For example, is running+paused+failed = total? Can a task be
> failed and destroyed and therefore count in 2 of those metrics?
>
> On Thu, Jun 6, 2019 at 12:29 PM Cyrus Vafadari  wrote:
> >
> > Konstantine,
> >
> > This is a good suggestion. Since the suggestion to add 2 additional
> > statuses analogous to the 3 proposed, it is a very minor change of no
> > structural consequence to the KIP.
> >
> > I've updated the KIP to incorporate your suggestion, and any voters who
> > disagree should definitely respond in the thread.
> >
> > Cyrus
> >
> > On Thu, Jun 6, 2019 at 11:16 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Thanks Cyrus,
> > >
> > > this is a nice and straightforward addition.
> > >
> > > I'm +1 too, but I'd like to return with a question here as well
> regarding
> > > whether the unassigned tasks will be taken into account.
> > > Especially after KIP-415 we might start seeing this status for specific
> > > periods of time. Therefore, I think it's a meaningful addition.
> > > Then there's the `destroyed` status which might be a lot more
> transient but
> > > we could also include for the sake of completion.
> > > Check org.apache.kafka.connect.runtime.AbstractStatus for the list of
> all
> > > possible statuses.
> > >
> > > Konstantine
> > >
> > > On Wed, Jun 5, 2019 at 4:32 PM Randall Hauch  wrote:
> > >
> > > > Thanks, Cyrus.
> > > >
> > > > +1 (binding)
> > > >
> > > > Randall Hauch
> > > >
> > > > On Wed, Jun 5, 2019 at 10:36 AM Andrew Schofield <
> > > > andrew_schofi...@live.com>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Andrew Schofield
> > > > >
> > > > > On 05/06/2019, 14:04, "Ryanne Dolan" 
> wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks
> > > > > Ryanne
> > > > >
> > > > > On Tue, Jun 4, 2019, 11:29 PM Cyrus Vafadari <
> cy...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Like like to start voting in the following KIP:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-475%253A%2BNew%2BMetric%2Bto%2BMeasure%2BNumber%2Bof%2BTasks%2Bon%2Ba%2BConnector&data=02%7C01%7C%7C95f8a8ebb4a44882773808d6e9b65983%7C84df9e7fe9f640afb435%7C1%7C0%7C636953366722392496&sdata=vbE%2BjrAapcQ68Vnwh5OkY1FFoOzFHs9rZRaPHlwqxSU%3D&reserved=0
> > > > > >
> > > > > > Discussion thread:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2Fbf7c92224aa798336c14d7e96ec8f2e3406c61879ec381a50652acfe%40%253Cdev.kafka.apache.org%253E&data=02%7C01%7C%7C95f8a8ebb4a44882773808d6e9b65983%7C84df9e7fe9f640afb435%7C1%7C0%7C636953366722402501&sdata=0JpQuCpTKwJyOjWH8cM%2B6eU%2FjNT28eE7xvMOBQgghjA%3D&reserved=0
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Cyrus
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>


-- 
-- Guozhang

Re: Become a contributer

[Anastasia Vela]

My user ID is anastasiavela.

Anastasia

On Thu, Jun 13, 2019 at 11:18 AM Anastasia Vela  wrote:

> Hi,
>
> I would like to contribute to Apache Kafka. Would I be able to get access
> to handle JIRAs?
> Github ID: anatasiavela
> Github Email: anastasiave...@gmail.com
>
> Thanks,
> Anastasia
>

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[John Roesler]

Woah, I wasn't aware of that Hamcrest test style. Awesome!

Thanks for the updates. I look forward to hearing what others think.

-John

On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
 wrote:
>
> Wiki page updated:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
>
>
> ClientRecord removed and replaced with TestRecord in method calls.
> TestRecordFactory removed (time tracking functionality to be included to
> TestInputTopic)
> OutputVerifier deprecated
> TestRecord topic removed and getters added
>
> Getters in TestRecord enable writing test ignoring selected fields with
> hamcrest like this:
>
> assertThat(outputTopic.readRecord(), allOf(
> hasProperty("key", equalTo(1L)),
> hasProperty("value", equalTo("Hello")),
> hasProperty("headers", equalTo(headers;
>
>
> Jukka
>
> la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io) kirjoitti:
>
> > Sounds good. Thanks as always for considering my feedback!
> > -John
> >
> > On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen
> >  wrote:
> > >
> > > Ok, I will modify KIP Public Interface in a wiki based on the feedback.
> > >
> > > TestRecordFactory / ConsumerRecordFactory was used by TestInputTopic with
> > > the version I had with KIP456, but maybe I can merge That functionality
> > to
> > > InputTopic or  TestRecordFactory   can kept non public maybe moving it to
> > > internals package.
> > >
> > > I will make the proposal with a slim down interface.
> > > I don't want to go to so slim as you proposed with only TestRecord or
> > > List, because you then still end up doing helper methods to
> > > construct List of TestRecord.
> > > The list of values is easier to write and clearer to read than if you
> > need
> > > to contruct list of TestRecords.
> > >
> > > For example:
> > >
> > > final List inputValues = Arrays.asList(
> > > "Apache Kafka Streams Example",
> > > "Using Kafka Streams Test Utils",
> > > "Reading and Writing Kafka Topic"
> > > );
> > > inputTopic.pipeValueList(inputValues);
> > >
> > >
> > > Let's check after the next iteration is it still worth reducing the
> > methods.
> > >
> > >
> > > Jukka
> > >
> > >
> > > pe 14. kesäk. 2019 klo 18.27 John Roesler (j...@confluent.io) kirjoitti:
> > >
> > > > Thanks, Jukka,
> > > >
> > > > Ok, I buy this reasoning.
> > > >
> > > > Just to echo what I think I read, you would just drop ClientRecord
> > > > from the proposal, and TestRecord would stand on its own, with the
> > > > same methods and properties you proposed, and the "input topic" would
> > > > accept TestRecord, and the "output topic" would produce TestRecord?
> > > > Further, the "input and output topic" classes would internally handle
> > > > the conversion to and from ConsumerRecord and ProducerRecord to pass
> > > > to and from the TopologyTestDriver?
> > > >
> > > > This seems good to me.
> > > >
> > > > Since the object coming out of the "output topic" is much more
> > > > ergonomic, I suspect we won't need the OutputVerifier at all. It was
> > > > mostly needed because of all the extra junk in ProducerRecord you need
> > > > to ignore. It seems better to just deprecate it. If in the future it
> > > > turns out there is some actual use case for a verifier, we can have a
> > > > very small KIP to add one. But reading your response, I suspect that
> > > > existing test verification libraries would be sufficient on their own.
> > > >
> > > > Similarly, it seems like we can shrink the total interface by removing
> > > > the TestRecordFactory from the proposal. If TestRecord already offers
> > > > all the constructors you'd want, then the only benefit of the factory
> > > > is to auto-increment the timestamps, but then again, the "input topic"
> > > > can already do that (e.g., it can do it if the record timestamp is not
> > > > set).
> > > >
> > > > Likewise, if the TestRecords are easy to create, then we don't need
> > > > all the redundant methods in "input topic" to pipe values, or
> > > > key/values, or key/value/timestamp, etc. We can do with just two
> > > > methods, one for a single TestRecord and one for a collection of them.
> > > > This reduces API ambiguity and also dramatically decreases the surface
> > > > area of the interface, which ultimately makes it much easier to use.
> > > >
> > > > It's best to start with the smallest interface that will do the job
> > > > and expand it upon request, rather than throwing in everything you can
> > > > think of up front. The extra stuff would be confusing to people facing
> > > > two practically identical paths to accomplish the same goal, and it's
> > > > very difficult to slim the interface down later, because we don't
> > > > really know which parts are more popular (i.e., no one submits
> > > > "feature requests" to _remove_ stuff they don't need, only to _add_
> > > > stuff that they need.
> > > >
> > > > But overall, I really like the structure of this 

Re: [VOTE] KIP-475: New Metric to Measure Number of Tasks on a Connector

[Gwen Shapira]

Sorry to join so late, but did we consider a single set of task-count
metrics and using tags to scope each data point to a specific
connector and worker (and in the future perhaps also user)?

It will make analysis of the data easier - someone may want to
breakdown tasks by both worker and connector to detect imbalanced
assignments.

Are there downsides to this approach?

And a small nit: it will be good to capture in the KIP what are the
expectations regarding overlap and disjointness of the proposed
metrics. For example, is running+paused+failed = total? Can a task be
failed and destroyed and therefore count in 2 of those metrics?

On Thu, Jun 6, 2019 at 12:29 PM Cyrus Vafadari  wrote:
>
> Konstantine,
>
> This is a good suggestion. Since the suggestion to add 2 additional
> statuses analogous to the 3 proposed, it is a very minor change of no
> structural consequence to the KIP.
>
> I've updated the KIP to incorporate your suggestion, and any voters who
> disagree should definitely respond in the thread.
>
> Cyrus
>
> On Thu, Jun 6, 2019 at 11:16 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Thanks Cyrus,
> >
> > this is a nice and straightforward addition.
> >
> > I'm +1 too, but I'd like to return with a question here as well regarding
> > whether the unassigned tasks will be taken into account.
> > Especially after KIP-415 we might start seeing this status for specific
> > periods of time. Therefore, I think it's a meaningful addition.
> > Then there's the `destroyed` status which might be a lot more transient but
> > we could also include for the sake of completion.
> > Check org.apache.kafka.connect.runtime.AbstractStatus for the list of all
> > possible statuses.
> >
> > Konstantine
> >
> > On Wed, Jun 5, 2019 at 4:32 PM Randall Hauch  wrote:
> >
> > > Thanks, Cyrus.
> > >
> > > +1 (binding)
> > >
> > > Randall Hauch
> > >
> > > On Wed, Jun 5, 2019 at 10:36 AM Andrew Schofield <
> > > andrew_schofi...@live.com>
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Andrew Schofield
> > > >
> > > > On 05/06/2019, 14:04, "Ryanne Dolan"  wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks
> > > > Ryanne
> > > >
> > > > On Tue, Jun 4, 2019, 11:29 PM Cyrus Vafadari 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Like like to start voting in the following KIP:
> > > > >
> > > > >
> > > >
> > >
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-475%253A%2BNew%2BMetric%2Bto%2BMeasure%2BNumber%2Bof%2BTasks%2Bon%2Ba%2BConnector&data=02%7C01%7C%7C95f8a8ebb4a44882773808d6e9b65983%7C84df9e7fe9f640afb435%7C1%7C0%7C636953366722392496&sdata=vbE%2BjrAapcQ68Vnwh5OkY1FFoOzFHs9rZRaPHlwqxSU%3D&reserved=0
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2Fbf7c92224aa798336c14d7e96ec8f2e3406c61879ec381a50652acfe%40%253Cdev.kafka.apache.org%253E&data=02%7C01%7C%7C95f8a8ebb4a44882773808d6e9b65983%7C84df9e7fe9f640afb435%7C1%7C0%7C636953366722402501&sdata=0JpQuCpTKwJyOjWH8cM%2B6eU%2FjNT28eE7xvMOBQgghjA%3D&reserved=0
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Cyrus
> > > > >
> > > >
> > > >
> > > >
> > >
> >



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[DISCUSS] KIP-478 Strongly Typed Processor API

[John Roesler]

Hi all,

I'd like to propose KIP-478 (https://cwiki.apache.org/confluence/x/2SkLBw).

This proposal would add output type bounds to the Processor interface
in Kafka Streams, which enables static checking of a number of useful
properties:
* A processor B that consumes the output of processor A is actually
expecting the same types that processor A produces.
* A processor that happens to use a typed store is actually emitting
the same types that it is storing.
* A processor is simply forwarding the expected types in all code paths.
* Processors added via the Streams DSL, which are not permitted to
forward results at all are statically prevented from doing so by the
compiler

Internally, we can use the above properties to achieve a much higher
level of confidence in the Streams DSL implementation's correctness.
Actually, while doing the POC, I found a few bugs and mistakes, which
become structurally impossible with KIP-478.

Additionally, the stronger types dramatically improve the
self-documentation of our Streams internal implementations, which
makes it much easier for new contributors to ramp up with confidence.

Thanks so much for your consideration!
-John

Build failed in Jenkins: kafka-trunk-jdk11 #637

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-7853: Refactor coordinator config (#6854)

[jason] KAFKA-8539; Add group.instance.id to Subscription (#6936)

--
[...truncated 4.28 KB...]
> Task :streams:upgrade-system-tests-10:clean UP-TO-DATE
> Task :streams:upgrade-system-tests-11:clean UP-TO-DATE
> Task :streams:upgrade-system-tests-20:clean UP-TO-DATE
> Task :streams:upgrade-system-tests-21:clean UP-TO-DATE
> Task :streams:upgrade-system-tests-22:clean UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE

> Task :rat
Rat report: 


> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test NO-SOURCE
> Task :generator:compileJava
> Task :generator:processResources NO-SOURCE
> Task :generator:classes

> Task :clients:processMessages
MessageGenerator: processed 93 Kafka message JSON files(s).

> Task :clients:compileJava
:393:
 error: cannot find symbol

 Optional.ofNullable(memberSubscription.groupInstanceId()));

 ^
  symbol:   variable Optional
  location: class ConsumerCoordinator
1 error

> Task :clients:compileJava FAILED
> Task :clients:processResources
> Task :clients:processTestResources
> Task :connect:compileJava NO-SOURCE
> Task :connect:processResources NO-SOURCE
> Task :connect:classes UP-TO-DATE
> Task :connect:checkstyleMain NO-SOURCE
> Task :connect:compileTestJava NO-SOURCE
> Task :connect:processTestResources NO-SOURCE
> Task :connect:testClasses UP-TO-DATE
> Task :connect:checkstyleTest NO-SOURCE
> Task :connect:spotbugsMain NO-SOURCE
> Task :connect:test NO-SOURCE
> Task :clients:determineCommitId UP-TO-DATE
> Task :clients:createVersionFile
> Task :core:processResources NO-SOURCE
> Task :core:processTestResources
> Task :examples:processResources NO-SOURCE
> Task :examples:processTestResources NO-SOURCE
> Task :generator:checkstyleMain
> Task :generator:compileTestJava
> Task :generator:processTestResources NO-SOURCE
> Task :generator:testClasses
> Task :generator:checkstyleTest
> Task :connect:api:processResources NO-SOURCE
> Task :connect:json:processResources NO-SOURCE
> Task :streams:processResources NO-SOURCE
> Task :jmh-benchmarks:processResources NO-SOURCE
> Task :jmh-benchmarks:processTestResources NO-SOURCE
> Task :log4j-appender:processResources NO-SOURCE
> Task :log4j-appender:processTestResources NO-SOURCE
> Task :streams:test-utils:processResources NO-SOURCE
> Task :streams:processTestResources
> Task :tools:processResources NO-SOURCE
> Task :tools:processTestResources
> Task :connect:api:processTestResources NO-SOURCE
> Task :connect:basic-auth-extension:processResources
> Task :connect:basic-auth-extension:processTestResources NO-SOURCE
> Task :connect:file:processResources NO-SOURCE
> Task :connect:file:processTestResources NO-SOURCE
> Task :connect:json:processTestResources
> Task :connect:transforms:processResources NO-SOURCE
> Task :connect:runtime:processResources
> Task :connect:runtime:processTestResources
> Task :connect:transforms:processTestResources NO-SOURCE
> Task :streams:examples:processResources NO-SOURCE
> Task :streams:examples:processTestResources NO-SOURCE
> Task :spotlessScala
> Task :spotlessScalaCheck
> Task :streams:streams-scala:processResources NO-SOURCE
> Task :streams:streams-scala:processTestResources
> Task :streams:test-utils:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest

Build failed in Jenkins: kafka-trunk-jdk11 #636

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8457; Move `Log' reference from `Replica` into `Partition` (#6841)

--
[...truncated 2.52 MB...]

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessString PASSED

org.apache.kafka.connect.transforms.CastTest > castFieldsWithSchema STARTED

org.apache.kafka.connect.transforms.CastTest > castFieldsWithSchema PASSED

org.apache.kafka.connect.transforms.ValueToKeyTest > withSchema STARTED

org.apache.kafka.connect.transforms.ValueToKeyTest > withSchema PASSED

org.apache.kafka.connect.transforms.ValueToKeyTest > schemaless STARTED

org.apache.kafka.connect.transforms.ValueToKeyTest > schemaless PASSED

org.apache.kafka.connect.transforms.RegexRouterTest > doesntMatch STARTED

org.apache.kafka.connect.transforms.RegexRouterTest > doesntMatch PASSED

org.apache.kafka.connect.transforms.RegexRouterTest > identity STARTED

org.apache.kafka.connect.transforms.RegexRouterTest > identity PASSED

org.apache.kafka.connect.transforms.RegexRouterTest > addPrefix STARTED

org.apache.kafka.connect.transforms.RegexRouterTest > addPrefix PASSED

org.apache.kafka.connect.transforms.RegexRouterTest > addSuffix STARTED

org.apache.kafka.connect.transforms.RegexRouterTest > addSuffix PASSED

org.apache.kafka.connect.transforms.RegexRouterTest > slice STARTED

org.apache.kafka.connect.transforms.RegexRouterTest > slice PASSED

org.apache.kafka.connect.transforms.RegexRouterTest > staticReplacement STARTED

org.apache.kafka.connect.transforms.RegexRouterTest > staticReplacement PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessFieldConversion STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessFieldConversion PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidTargetType STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidTargetType PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessStringToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessStringToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > testKey STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > testKey PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessDateToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessDateToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToString STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToString PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimeToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimeToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToDate STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToDate PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToTime STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToTime PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToUnix STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToUnix PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaUnixToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaUnixToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessIdentity STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessIdentity PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaFieldConversion STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaFieldConversion PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaDateToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaDateToTimestamp PASSED

org.apache.kafka.con

[jira] [Resolved] (KAFKA-8539) Add `group.instance.id` to Subscription class

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8539.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Add `group.instance.id` to Subscription class
> -
>
> Key: KAFKA-8539
> URL: https://issues.apache.org/jira/browse/KAFKA-8539
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
> Fix For: 2.4.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Posted a new article about Kafka Streams

[Development]

Hey Paul,

Thank you so much for your input! :)
Just updated my article about the iterator closing.

Thank you!

Best,
Daniyar Yeralin

> On Jun 16, 2019, at 4:11 PM, Paul Whalen  wrote:
> 
> I've only skimmed it so far, but great job!  The community is in serious
> need of more examples of the Processor API, there really isn't that much
> out there.
> 
> One thing I did notice: the iterator you get from kvStore.all() ought to be
> closed to release resources when you're done with it.  This matters when
> the underlying store is RocksDB, which as I understand it, allocates
> additional memory off heap to iterate.  I see this bug everywhere, after
> writing it many times myself over the course of many months :). it's too
> bad the API can't be more clear, but I guess there's not a ton you can do
> in Java.  People think about this kind of thing for DB calls, but when
> you're using something that's basically a HashMap you really don't think of
> it at all.
> 
> Side plug for KIP-401 since you're using the Processor API, it would be
> interesting to hear on that discussion thread if you find it useful (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97553756).
> It seems like there's soft interest, but maybe not yet enough to push it
> over the finish line.
> 
> Again, great job!
> 
> Paul
> 
> On Fri, Jun 14, 2019 at 10:33 AM Development  wrote:
> 
>> Bad link:
>> 
>> https://medium.com/@daniyaryeralin/utilizing-kafka-streams-processor-api-and-implementing-custom-aggregator-6cb23d00eaa7
>> 
>>> On Jun 14, 2019, at 11:07 AM, Development  wrote:
>>> 
>>> Hello Kafka Dev community,
>>> 
>>> I wrote an article on implementing a custom transformer using Processor
>> API for Kafka Streams!
>>> 
>> https://medium.com/@daniyaryeralin/utilizing-kafka-streams-processor-api-and-implementing-custom-aggregation-f6a4a6c376be
>> <
>> https://medium.com/@daniyaryeralin/utilizing-kafka-streams-processor-api-and-implementing-custom-aggregation-f6a4a6c376be
>>> 
>>> Feel free to leave a feedback and/or corrections if I wrote something
>> silly :)
>>> 
>>> Thank you!
>>> 
>>> Best,
>>> Daniyar Yeralin
>> 
>> 

[jira] [Resolved] (KAFKA-7853) Refactor ConsumerCoordinator/AbstractCoordinator to reduce constructor parameter list

[Boyang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-7853.

Resolution: Fixed

> Refactor ConsumerCoordinator/AbstractCoordinator to reduce constructor 
> parameter list
> -
>
> Key: KAFKA-7853
> URL: https://issues.apache.org/jira/browse/KAFKA-7853
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>
> The parameter lists for class ConsumerCoordinator/AbstractCoordinator are 
> growing over time. We should think of reducing the parameter size by 
> introducing some intermediate data structs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Ryanne Dolan]

Andy, while I agree that the new interface is useful, I'm not convinced
adding an interface requires deprecating AdminClient and changing so much
client code. Why not just add the Admin interface, have AdminClient
implement it, and have done?

Ryanne

On Mon, Jun 17, 2019 at 12:09 PM Andy Coates  wrote:

> Hi all,
>
> I think I've addressed all concerns. Let me know if I've not.  Can I call
> another round of votes please?
>
> Thanks,
>
> Andy
>
> On Fri, 14 Jun 2019 at 04:55, Satish Duggana 
> wrote:
>
> > Hi Andy,
> > Thanks for the KIP. This is a good change and it gives the user a better
> > handle on Admin client usage. I agree with the proposal except the new
> > `Admin` interface having all the methods from `AdminClient` abstract
> class.
> > It should be kept clean having only the admin operations as methods from
> > KafkaClient abstract class but not the factory methods as mentioned in
> the
> > earlier mail.
> >
> > I know about dynamic proxies(which were widely used in RMI/EJB world). I
> am
> > curious about the usecase using dynamic proxies with Admin client
> > interface. Dynamic proxy can have performance penalty if it is used in
> > critical path. Is that the primary motivation for creating the KIP?
> >
> > Thanks,
> > Satish.
> >
> > On Wed, Jun 12, 2019 at 8:43 PM Andy Coates  wrote:
> >
> > > I'm not married to that part.  That was only done to keep it more or
> less
> > > inline with what's already there, (an abstract class that has a factory
> > > method that returns a subclass sounds like the same anti-pattern
> ;))
> > >
> > > An alternative would to have an `AdminClients` utility class to create
> > the
> > > admin client.
> > >
> > > On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax 
> > > wrote:
> > >
> > > > Hmmm...
> > > >
> > > > So the new interface, returns an instance of a class that implements
> > the
> > > > interface. This sounds a little bit like an anti-pattern? Shouldn't
> > > > interfaces actually not know anything about classes that implement
> the
> > > > interface?
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 6/10/19 11:22 AM, Andy Coates wrote:
> > > > > `AdminClient` would be deprecated purely because it would no longer
> > > serve
> > > > > any purpose and would be virtually empty, getting all of its
> > > > implementation
> > > > > from the new interfar. It would be nice to remove this from the API
> > at
> > > > the
> > > > > next major version bump, hence the need to deprecate.
> > > > >
> > > > > `AdminClient.create()` would return what it does today, (so not a
> > > > breaking
> > > > > change).
> > > > >
> > > > > On Tue, 4 Jun 2019 at 22:24, Ryanne Dolan 
> > > wrote:
> > > > >
> > > > >>> The existing `AdminClient` will be marked as deprecated.
> > > > >>
> > > > >> What's the reasoning behind this? I'm fine with the other changes,
> > but
> > > > >> would prefer to keep the existing public API intact if it's not
> > > hurting
> > > > >> anything.
> > > > >>
> > > > >> Also, what will AdminClient.create() return? Would it be a
> breaking
> > > > change?
> > > > >>
> > > > >> Ryanne
> > > > >>
> > > > >> On Tue, Jun 4, 2019, 11:17 AM Andy Coates 
> > wrote:
> > > > >>
> > > > >>> Hi folks
> > > > >>>
> > > > >>> As there's been no chatter on this KIP I'm assuming it's
> > > > non-contentious,
> > > > >>> (or just boring), hence I'd like to call a vote for KIP-476:
> > > > >>>
> > > > >>>
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-476%3A+Add+Java+AdminClient+Interface
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Andy
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Development]

bump

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Andy Coates]

Hi all,

I think I've addressed all concerns. Let me know if I've not.  Can I call
another round of votes please?

Thanks,

Andy

On Fri, 14 Jun 2019 at 04:55, Satish Duggana 
wrote:

> Hi Andy,
> Thanks for the KIP. This is a good change and it gives the user a better
> handle on Admin client usage. I agree with the proposal except the new
> `Admin` interface having all the methods from `AdminClient` abstract class.
> It should be kept clean having only the admin operations as methods from
> KafkaClient abstract class but not the factory methods as mentioned in the
> earlier mail.
>
> I know about dynamic proxies(which were widely used in RMI/EJB world). I am
> curious about the usecase using dynamic proxies with Admin client
> interface. Dynamic proxy can have performance penalty if it is used in
> critical path. Is that the primary motivation for creating the KIP?
>
> Thanks,
> Satish.
>
> On Wed, Jun 12, 2019 at 8:43 PM Andy Coates  wrote:
>
> > I'm not married to that part.  That was only done to keep it more or less
> > inline with what's already there, (an abstract class that has a factory
> > method that returns a subclass sounds like the same anti-pattern ;))
> >
> > An alternative would to have an `AdminClients` utility class to create
> the
> > admin client.
> >
> > On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax 
> > wrote:
> >
> > > Hmmm...
> > >
> > > So the new interface, returns an instance of a class that implements
> the
> > > interface. This sounds a little bit like an anti-pattern? Shouldn't
> > > interfaces actually not know anything about classes that implement the
> > > interface?
> > >
> > >
> > > -Matthias
> > >
> > > On 6/10/19 11:22 AM, Andy Coates wrote:
> > > > `AdminClient` would be deprecated purely because it would no longer
> > serve
> > > > any purpose and would be virtually empty, getting all of its
> > > implementation
> > > > from the new interfar. It would be nice to remove this from the API
> at
> > > the
> > > > next major version bump, hence the need to deprecate.
> > > >
> > > > `AdminClient.create()` would return what it does today, (so not a
> > > breaking
> > > > change).
> > > >
> > > > On Tue, 4 Jun 2019 at 22:24, Ryanne Dolan 
> > wrote:
> > > >
> > > >>> The existing `AdminClient` will be marked as deprecated.
> > > >>
> > > >> What's the reasoning behind this? I'm fine with the other changes,
> but
> > > >> would prefer to keep the existing public API intact if it's not
> > hurting
> > > >> anything.
> > > >>
> > > >> Also, what will AdminClient.create() return? Would it be a breaking
> > > change?
> > > >>
> > > >> Ryanne
> > > >>
> > > >> On Tue, Jun 4, 2019, 11:17 AM Andy Coates 
> wrote:
> > > >>
> > > >>> Hi folks
> > > >>>
> > > >>> As there's been no chatter on this KIP I'm assuming it's
> > > non-contentious,
> > > >>> (or just boring), hence I'd like to call a vote for KIP-476:
> > > >>>
> > > >>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-476%3A+Add+Java+AdminClient+Interface
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Andy
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Andy Coates]

Hi All,

I've updated the KIP to move the `create` factory method implementation
into a new `AdminClients` utility class, rather than on the new `Admin`
interface.

Satish,

As above, the KIP has been updated to only have the operations on the
`Admin` api. As for the overhead of dynamic proxies... the use of dynamic
proxies is totally up to the users of the library. In KSQL we use dynamic
proxies because the overhead is acceptable and it decouples us from
additions to the client interfaces. Others may decide otherwise for their
project. By making the admin api an interface we're empowering users to
choose the right approach for them.

This is the primary motivation for the KIP from my point of view. However,
it also brings it inline with the other Kafka clients, and gives users the
freedom to do what they want, rather than requiring the use of an abstract
base class.

Thanks,

Andy


On Fri, 14 Jun 2019 at 04:55, Satish Duggana 
wrote:

> Hi Andy,
> Thanks for the KIP. This is a good change and it gives the user a better
> handle on Admin client usage. I agree with the proposal except the new
> `Admin` interface having all the methods from `AdminClient` abstract class.
> It should be kept clean having only the admin operations as methods from
> KafkaClient abstract class but not the factory methods as mentioned in the
> earlier mail.
>
> I know about dynamic proxies(which were widely used in RMI/EJB world). I am
> curious about the usecase using dynamic proxies with Admin client
> interface. Dynamic proxy can have performance penalty if it is used in
> critical path. Is that the primary motivation for creating the KIP?
>
> Thanks,
> Satish.
>
> On Wed, Jun 12, 2019 at 8:43 PM Andy Coates  wrote:
>
> > I'm not married to that part.  That was only done to keep it more or less
> > inline with what's already there, (an abstract class that has a factory
> > method that returns a subclass sounds like the same anti-pattern ;))
> >
> > An alternative would to have an `AdminClients` utility class to create
> the
> > admin client.
> >
> > On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax 
> > wrote:
> >
> > > Hmmm...
> > >
> > > So the new interface, returns an instance of a class that implements
> the
> > > interface. This sounds a little bit like an anti-pattern? Shouldn't
> > > interfaces actually not know anything about classes that implement the
> > > interface?
> > >
> > >
> > > -Matthias
> > >
> > > On 6/10/19 11:22 AM, Andy Coates wrote:
> > > > `AdminClient` would be deprecated purely because it would no longer
> > serve
> > > > any purpose and would be virtually empty, getting all of its
> > > implementation
> > > > from the new interfar. It would be nice to remove this from the API
> at
> > > the
> > > > next major version bump, hence the need to deprecate.
> > > >
> > > > `AdminClient.create()` would return what it does today, (so not a
> > > breaking
> > > > change).
> > > >
> > > > On Tue, 4 Jun 2019 at 22:24, Ryanne Dolan 
> > wrote:
> > > >
> > > >>> The existing `AdminClient` will be marked as deprecated.
> > > >>
> > > >> What's the reasoning behind this? I'm fine with the other changes,
> but
> > > >> would prefer to keep the existing public API intact if it's not
> > hurting
> > > >> anything.
> > > >>
> > > >> Also, what will AdminClient.create() return? Would it be a breaking
> > > change?
> > > >>
> > > >> Ryanne
> > > >>
> > > >> On Tue, Jun 4, 2019, 11:17 AM Andy Coates 
> wrote:
> > > >>
> > > >>> Hi folks
> > > >>>
> > > >>> As there's been no chatter on this KIP I'm assuming it's
> > > non-contentious,
> > > >>> (or just boring), hence I'd like to call a vote for KIP-476:
> > > >>>
> > > >>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-476%3A+Add+Java+AdminClient+Interface
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Andy
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>

[jira] [Resolved] (KAFKA-8548) Inconsistency in Kafka Documentation

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-8548.

Resolution: Not A Problem

Closing this. The docs are correct as pointed out by [~ckamal].

> Inconsistency in Kafka Documentation
> 
>
> Key: KAFKA-8548
> URL: https://issues.apache.org/jira/browse/KAFKA-8548
> Project: Kafka
>  Issue Type: Task
>  Components: documentation
>Affects Versions: 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Minor
>
> Dears,
> Two parts (referenced below) of [documentation 
> |http://kafka.apache.org/documentation/] are not quite consistent.
> In one text we can read, that max.poll.interval.ms has defaut value 
> Integer.MAX_VALUE, in the other it is 300 000.
> Part 1.
> {quote}
> The default values for two configurations of the StreamsConfig class were 
> changed to improve the resiliency of Kafka Streams applications. The internal 
> Kafka Streams producer retries default value was changed from 0 to 10. The 
> internal Kafka Streams consumer max.poll.interval.ms default value was 
> changed from 30 to {color:#FF}Integer.MAX_VALUE{color}.
> {quote}
>  
> Part 2. - Table
> |max.poll.interval.ms|The maximum delay between invocations of poll() when 
> using consumer group management. This places an upper bound on the amount of 
> time that the consumer can be idle before fetching more records. If poll() is 
> not called before expiration of this timeout, then the consumer is considered 
> failed and the group will rebalance in order to reassign the partitions to 
> another member.|int|{color:#FF}30{color}|[1,...]|medium|
> Which value is then default :-)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8457) Remove Log dependency from Replica

[Vikas Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Singh resolved KAFKA-8457.

Resolution: Fixed

Fixed in commit 57baa4079d9fc14103411f790b9a025c9f2146a4

> Remove Log dependency from Replica
> --
>
> Key: KAFKA-8457
> URL: https://issues.apache.org/jira/browse/KAFKA-8457
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Vikas Singh
>Assignee: Vikas Singh
>Priority: Major
>
> A partition can have one log but many replicas. Putting log in replica meant 
> that we have to have if-else each time we need to access log. Moving the log 
> out of replica and in partition will make code simpler and it will also help 
> in testing where mocks will get simplified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 2.3.0 RC2

[David Arthur]

+1 binding

Verified signatures, pulled down kafka_2.12-2.3.0 and ran producer/consumer
perf test scripts.

-David

On Mon, Jun 17, 2019 at 1:48 AM Vahid Hashemian 
wrote:

> +1 (non-binding)
>
> I also verifies signatures, build from source and tested the Quickstart
> successfully on the built binary.
>
> BTW, I don't see a link to documentation for 2.3. Is there a reason?
>
> Thanks,
> --Vahid
>
> On Sat, Jun 15, 2019 at 6:38 PM Gwen Shapira  wrote:
>
> > +1 (binding)
> >
> > Verified signatures, built from sources, ran quickstart on binary and
> > checked out the passing jenkins build on the branch.
> >
> > Gwen
> >
> >
> > On Thu, Jun 13, 2019 at 11:58 AM Colin McCabe 
> wrote:
> > >
> > > Hi all,
> > >
> > > Good news: I have run a junit test build for RC2, and it passed.  Check
> > out https://builds.apache.org/job/kafka-2.3-jdk8/51/
> > >
> > > Also, the vote will go until Saturday, June 15th (sorry for the typo
> > earlier in the vote end time).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Wed, Jun 12, 2019, at 15:55, Colin McCabe wrote:
> > > > Hi all,
> > > >
> > > > We discovered some problems with the first release candidate (RC1) of
> > > > 2.3.0.  Specifically, KAFKA-8484 and KAFKA-8500.  I have created a
> new
> > > > release candidate that includes fixes for these issues.
> > > >
> > > > Check out the release notes for the 2.3.0 release here:
> > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/RELEASE_NOTES.html
> > > >
> > > > The vote will go until Friday, June 7th, or until we create another R
> > > >
> > > > * Kafka's KEYS file containing PGP keys we use to sign the release
> can
> > > > be found here:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * The release artifacts to be voted upon (source and binary) are
> here:
> > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/javadoc/
> > > >
> > > > * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/2.3.0-rc2
> > > >
> > > > best,
> > > > Colin
> > > >
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
>
> --
>
> Thanks!
> --Vahid
>

[jira] [Created] (KAFKA-8549) Kafka Windows start up failed due to topic name conflict

[prehistoricpenguin (JIRA)]

prehistoricpenguin created KAFKA-8549:
-

 Summary: Kafka Windows start up failed due to topic name conflict 
 Key: KAFKA-8549
 URL: https://issues.apache.org/jira/browse/KAFKA-8549
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.2.1
Reporter: prehistoricpenguin


We are running Kafka server on windows, we got this exception during Kafka 
server start up:
{code:java}
2019-06-11 14:50:48,537] ERROR Error while creating log for 
this_is_a_topic_name in dir C:\Program Files (x86)\dummy_path\tmp\kafka-logs 
(kafka.server.LogDirFailureChannel)
java.io.IOException: The requested operation cannot be performed on a file with 
a user-mapped section open
at java.io.RandomAccessFile.setLength(Native Method)
at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175)
at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:238)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:238)
at kafka.log.LogSegment.recover(LogSegment.scala:377)
at kafka.log.Log.recoverSegment(Log.scala:500)
at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:482)
at 
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:792)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:791)
at kafka.log.Log.loadSegmentFiles(Log.scala:454)
at kafka.log.Log.$anonfun$loadSegments$1(Log.scala:565)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2034)
at kafka.log.Log.loadSegments(Log.scala:559)
at kafka.log.Log.(Log.scala:292)
at kafka.log.Log$.apply(Log.scala:2168)
at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:716)
at scala.Option.getOrElse(Option.scala:138)
at kafka.log.LogManager.getOrCreateLog(LogManager.scala:674)
at kafka.cluster.Partition.$anonfun$getOrCreateReplica$1(Partition.scala:202)
at kafka.utils.Pool$$anon$1.apply(Pool.scala:61)
at 
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at kafka.utils.Pool.getAndMaybePut(Pool.scala:60)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:198)
at kafka.cluster.Partition.$anonfun$makeLeader$3(Partition.scala:376)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:237)
at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at kafka.cluster.Partition.$anonfun$makeLeader$1(Partition.scala:376)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:259)
at kafka.cluster.Partition.makeLeader(Partition.scala:370)
at kafka.server.ReplicaManager.$anonfun$makeLeaders$5(ReplicaManager.scala:1188)
at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:1186)
at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:1098)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:195)
at kafka.server.KafkaApis.handle(KafkaApis.scala:112)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
at java.lang.Thread.run(Thread.java:748)
[2019-06-11 14:50:48,542] INFO [ReplicaManager broker=0] Stopping serving 
replicas in dir C:\Program Files (x86)\dummy_path\tmp\kafka-logs 
(kafka.server.ReplicaManager)
[2019-06-11 14:50:48,543] ERROR [ReplicaManager broker=0] Error while making 
broker the leader for partition Topic: this_is_a_topic_name; Partition: 0; 
Leader: None; AllReplicas: ; InSyncReplicas: in dir None 
(kafka.server

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[Jukka Karvanen]

Wiki page updated:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements


ClientRecord removed and replaced with TestRecord in method calls.
TestRecordFactory removed (time tracking functionality to be included to
TestInputTopic)
OutputVerifier deprecated
TestRecord topic removed and getters added

Getters in TestRecord enable writing test ignoring selected fields with
hamcrest like this:

assertThat(outputTopic.readRecord(), allOf(
hasProperty("key", equalTo(1L)),
hasProperty("value", equalTo("Hello")),
hasProperty("headers", equalTo(headers;


Jukka

la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io) kirjoitti:

> Sounds good. Thanks as always for considering my feedback!
> -John
>
> On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen
>  wrote:
> >
> > Ok, I will modify KIP Public Interface in a wiki based on the feedback.
> >
> > TestRecordFactory / ConsumerRecordFactory was used by TestInputTopic with
> > the version I had with KIP456, but maybe I can merge That functionality
> to
> > InputTopic or  TestRecordFactory   can kept non public maybe moving it to
> > internals package.
> >
> > I will make the proposal with a slim down interface.
> > I don't want to go to so slim as you proposed with only TestRecord or
> > List, because you then still end up doing helper methods to
> > construct List of TestRecord.
> > The list of values is easier to write and clearer to read than if you
> need
> > to contruct list of TestRecords.
> >
> > For example:
> >
> > final List inputValues = Arrays.asList(
> > "Apache Kafka Streams Example",
> > "Using Kafka Streams Test Utils",
> > "Reading and Writing Kafka Topic"
> > );
> > inputTopic.pipeValueList(inputValues);
> >
> >
> > Let's check after the next iteration is it still worth reducing the
> methods.
> >
> >
> > Jukka
> >
> >
> > pe 14. kesäk. 2019 klo 18.27 John Roesler (j...@confluent.io) kirjoitti:
> >
> > > Thanks, Jukka,
> > >
> > > Ok, I buy this reasoning.
> > >
> > > Just to echo what I think I read, you would just drop ClientRecord
> > > from the proposal, and TestRecord would stand on its own, with the
> > > same methods and properties you proposed, and the "input topic" would
> > > accept TestRecord, and the "output topic" would produce TestRecord?
> > > Further, the "input and output topic" classes would internally handle
> > > the conversion to and from ConsumerRecord and ProducerRecord to pass
> > > to and from the TopologyTestDriver?
> > >
> > > This seems good to me.
> > >
> > > Since the object coming out of the "output topic" is much more
> > > ergonomic, I suspect we won't need the OutputVerifier at all. It was
> > > mostly needed because of all the extra junk in ProducerRecord you need
> > > to ignore. It seems better to just deprecate it. If in the future it
> > > turns out there is some actual use case for a verifier, we can have a
> > > very small KIP to add one. But reading your response, I suspect that
> > > existing test verification libraries would be sufficient on their own.
> > >
> > > Similarly, it seems like we can shrink the total interface by removing
> > > the TestRecordFactory from the proposal. If TestRecord already offers
> > > all the constructors you'd want, then the only benefit of the factory
> > > is to auto-increment the timestamps, but then again, the "input topic"
> > > can already do that (e.g., it can do it if the record timestamp is not
> > > set).
> > >
> > > Likewise, if the TestRecords are easy to create, then we don't need
> > > all the redundant methods in "input topic" to pipe values, or
> > > key/values, or key/value/timestamp, etc. We can do with just two
> > > methods, one for a single TestRecord and one for a collection of them.
> > > This reduces API ambiguity and also dramatically decreases the surface
> > > area of the interface, which ultimately makes it much easier to use.
> > >
> > > It's best to start with the smallest interface that will do the job
> > > and expand it upon request, rather than throwing in everything you can
> > > think of up front. The extra stuff would be confusing to people facing
> > > two practically identical paths to accomplish the same goal, and it's
> > > very difficult to slim the interface down later, because we don't
> > > really know which parts are more popular (i.e., no one submits
> > > "feature requests" to _remove_ stuff they don't need, only to _add_
> > > stuff that they need.
> > >
> > > But overall, I really like the structure of this design. I'm super
> > > excited about this KIP.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Fri, Jun 14, 2019 at 2:55 AM Jukka Karvanen
> > >  wrote:
> > > >
> > > > Hi,
> > > >
> > > > I am not a fan of swapping only ProducerRecord and ConsumerRecord.
> > > > As a test writer point of view I do not want to care about the
> difference
> > > > of those and
> > > > that way I like to have object type w

[jira] [Created] (KAFKA-8548) Inconsistency in Kafka Documentation

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-8548:
-

 Summary: Inconsistency in Kafka Documentation
 Key: KAFKA-8548
 URL: https://issues.apache.org/jira/browse/KAFKA-8548
 Project: Kafka
  Issue Type: Task
  Components: documentation
Affects Versions: 2.2.1
Reporter: Seweryn Habdank-Wojewodzki


Dears,

Two parts (referenced below) of [documentation 
|http://kafka.apache.org/documentation/] are not quite consistent.

In one text we can read, that max.poll.interval.ms has defaut value 
Integer.MAX_VALUE, in the other it is 300 000.

Part 1.

{quote}
The default values for two configurations of the StreamsConfig class were 
changed to improve the resiliency of Kafka Streams applications. The internal 
Kafka Streams producer retries default value was changed from 0 to 10. The 
internal Kafka Streams consumer max.poll.interval.ms default value was changed 
from 30 to {color:#FF}Integer.MAX_VALUE{color}.
{quote}
 
Part 2. - Table

|max.poll.interval.ms|The maximum delay between invocations of poll() when 
using consumer group management. This places an upper bound on the amount of 
time that the consumer can be idle before fetching more records. If poll() is 
not called before expiration of this timeout, then the consumer is considered 
failed and the group will rebalance in order to reassign the partitions to 
another member.|int|{color:#FF}30{color}|[1,...]|medium|

Which value is then default :-)




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Kafka streams rebalancing issue

[Matthias J. Sax]

> This leads to lockexception in
>> consumer2 and consumer2 remains in livelock to create state directories for
>> those two partitons.

There is a fix for a `LockException` during rebalance in 0.11.0.1:
https://issues.apache.org/jira/browse/KAFKA-5167

Maybe upgrading helps? Note, that you can upgrade Kafka Streams
independent of your brokers. Btw: I would recommend to upgrade to at
least to 0.11.0.3 what is the latest bug-fix release for 0.11.0; in
general, if there is a bug fix release, its recommended to upgrade.

Besides bug-fix release, I would recommend to upgrade to newer version
anyway (including broker if possible) as 0.11.0 is already 2 years old...



-Matthias


On 6/15/19 12:20 AM, Aadhil RF wrote:
> Hi All,
> 
>  We have two consumers in a consumer group subscribed to the topic.
> Both the consumers are in different servers. The topic consists of 11
> partitions and 1 replication. Normally, 5 partitions are consumed in
> consumer 1 and remaining in consumer 2. Whenever there is a connection
> glitch between consumers and the coordinator, the rebalance procedure is
> running on both consumers. During this procedure, all the 11 partitions are
> assigned to consumer1 and two of the partitions (which are assigned to
> consumer1) are assigned to consumer2. This leads to lockexception in
> consumer2 and consumer2 remains in livelock to create state directories for
> those two partitons.
> 
> Kafka version: 0.11.0.0
> Zookeeper: 3.4.10
> 



signature.asc
Description: OpenPGP digital signature