[GitHub] [kafka-site] mjsax merged pull request #283: MINOR: fix HTML

[GitBox]

mjsax merged pull request #283:
URL: https://github.com/apache/kafka-site/pull/283


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is back to normal : kafka-trunk-jdk8 #4768

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-2.6-jdk8 #107

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: fix HTML (#9127)


--
[...truncated 8.54 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

[jira] [Created] (KAFKA-10371) Partition reassignments can result in crashed ReplicaFetcherThreads.

[Steve Rodrigues (Jira)]

Steve Rodrigues created KAFKA-10371:
---

 Summary: Partition reassignments can result in crashed 
ReplicaFetcherThreads.
 Key: KAFKA-10371
 URL: https://issues.apache.org/jira/browse/KAFKA-10371
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.7.0
Reporter: Steve Rodrigues


A Kafka system doing partition reassignments got stuck with the reassignment 
partially done and the system with a non-zero number of URPs and increasing max 
lag.

Looking in the logs, we see: 
{noformat}
[ERROR] 2020-07-31 21:22:23,984 [ReplicaFetcherThread-0-3] 
kafka.server.ReplicaFetcherThread - [ReplicaFetcher replicaId=4, leaderId=3, 
fetcherId=0] Error due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: Error while 
fetching partition state for foo
[INFO] 2020-07-31 21:22:23,986 [ReplicaFetcherThread-0-3] 
kafka.server.ReplicaFetcherThread - [ReplicaFetcher replicaId=4, leaderId=3, 
fetcherId=0] Stopped
{noformat}

Investigating further and with some helpful changes to the exception (which was 
not generating a stack trace because it was a client-side exception), we see on 
a test run:

{noformat}
[2020-08-06 19:58:21,592] ERROR [ReplicaFetcher replicaId=2, leaderId=1, 
fetcherId=0] Error due to (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.NotLeaderOrFollowerException: Error while 
fetching partition state for topic-test-topic-85
at org.apache.kafka.common.protocol.Errors.exception(Errors.java:415)
at 
kafka.server.ReplicaManager.getPartitionOrException(ReplicaManager.scala:645)
at 
kafka.server.ReplicaManager.localLogOrException(ReplicaManager.scala:672)
at 
kafka.server.ReplicaFetcherThread.logStartOffset(ReplicaFetcherThread.scala:133)
at 
kafka.server.ReplicaFetcherThread.$anonfun$buildFetch$1(ReplicaFetcherThread.scala:316)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
at 
kafka.server.ReplicaFetcherThread.buildFetch(ReplicaFetcherThread.scala:309)
{noformat}

It appears that the fetcher is attempting to fetch for a partition that has 
been getting reassigned away. From further investigation, it seems that in 
KAFKA-10002 the StopReplica code was changed from:
1. Remove partition from fetcher
2. Remove partition from partition map
to the other way around, but now the fetcher may race and attempt to build a 
fetch for a partition that's no longer mapped.  In particular, since the 
logOrException code is being called from logStartOffset which isn't protected 
against NotLeaderOrFollowerException, just against KafkaStorageException, the 
exception isn't caught and throws all the way out, killing the replica fetcher 
thread.
We need to switch this back.






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10370) WorkerSinkTask: IllegalStateException cased by consumer.seek(tp, offsets) when (tp, offsets) are supplied by WorkerSinkTaskContext

[Ning Zhang (Jira)]

Ning Zhang created KAFKA-10370:
--

 Summary: WorkerSinkTask: IllegalStateException cased by 
consumer.seek(tp, offsets) when (tp, offsets) are supplied by 
WorkerSinkTaskContext
 Key: KAFKA-10370
 URL: https://issues.apache.org/jira/browse/KAFKA-10370
 Project: Kafka
  Issue Type: New Feature
  Components: KafkaConnect
Affects Versions: 2.5.0
Reporter: Ning Zhang
Assignee: Ning Zhang
 Fix For: 2.6.0


In WorkerSinkTask.java, when we want the consumer to start consuming from 
certain offsets, rather than from the last committed offset, 
[WorkerSinkTaskContext|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L295]
 is used to carry the offsets from external world (e.g. implementation of 
SinkTask).

In the [poll() 
method|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L312],
 it first call "rewind()" to (1) read the offsets from WorkerSinkTaskContext, 
(2) consumer.seek(tp, offset) to rewind the consumer.

when running (2), we saw the following IllegalStateException:
```
java.lang.IllegalStateException: No current assignment for partition mytopic-1
at 
org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:251)
at 
org.apache.kafka.clients.consumer.internals.SubscriptionState.seek(SubscriptionState.java:276)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1135)
```

As suggested in 
https://stackoverflow.com/questions/41008610/kafkaconsumer-0-10-java-api-error-message-no-current-assignment-for-partition/41010594,
 the resolution is to use *consumer.assign* with *consumer.seek* , instead of 
*consumer.subscribe*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

[Boyang Chen]

Hey there,

we are going to introduce a minor change to bump the version of several
RPCs which are currently not supporting flexible versions. It is necessary
because they need to be able to construct request header with initial
principal name and client id as optional fields for redirection. The are
only two of them:

1. AlterConfig
2. AlterClientQuotas

Let me know if you have any questions.

Boyang

On Fri, Jul 31, 2020 at 11:42 AM Boyang Chen 
wrote:

> Hey David,
>
> After discussing with Colin offline, I would like to correct one case in
> the described workflow, where the CLUSTER_ACTION authorization would not be
> based on the initial principal field check, because it is not a secured
> condition which anyone could forge. The revised workflow shall be:
>
> Step 1. Filter out resources that are authorized
>  1.1 Use traditional principals to verify first. If authorized,
> continue
>  1.2 If not authorized, check whether the request is from the
> control plane. Note that this is a best-effort to verify whether the
> request is internal.
>  1.3 If the request is not from the control plane, return
> authorization failure
>  1.4 If the request is from the control plane, use CLUSTER_ACTION
> to verify and determine the result
>
> Step 2. Check the request context to see if this is a forwarding request,
> by checking whether it is from control plane and uses extra header fields
> 2.1 if the resource is authorized, and if this is the active
> controller, process it
> 2.2 if the resource is authorized but this is not the active
> controller, return NOT_CONTROLLER to the sender (forwarding broker) for
> retry
> 2.3 if the resource is not authorized, return
> CLUSTER_AUTHORIZATION_FAILURE to propagate back to the original client
> through forwarding broker
> Step 3. If the request is not a forwarding request
> 3.1 If the resource is authorized, and this is the active
> controller, process it
> 3.2 If the resource is authorized, but this is not active
> controller, put the resource into the preparation for a new AlterConfig
> request for forwarding
> 3.3 If the resource is not authorized, reply the original client
> AUTHORIZATION_FAILURE when the forwarding request is returned
>
> On Thu, Jul 30, 2020 at 3:47 PM Boyang Chen 
> wrote:
>
>>
>>
>> On Thu, Jul 30, 2020 at 7:18 AM David Jacot  wrote:
>>
>>> Hi Boyang,
>>>
>>> Thanks for your answers.
>>>
>>> > The point for using the listener name is more of a security purpose, to
>>> > detect any forged request to our best effort.
>>> > For throttling I think we could just check the request header for
>>> > *InitialClientId* existence, to distinguish whether to apply
>>> > throttling strategy as forwarded request or direct request.
>>>
>>> Reading "security" and "best effort" in the same sentence makes me a
>>> little nervous :).
>>>
>>> The identification issue is also valid for quota as we don't want one to
>>> be
>>> able to bypass the quota by forging a request as well, isn't it?
>>> Otherwise,
>>> anyone could just set the InitialPrincipal to bypass it. I think that we
>>> should
>>> only use InitialPrincipal and/or InitialClientId when we know that they
>>> come
>>> from another broker. Based on what I read in the KIP, it looks like we
>>> could
>>> only use them when the principal has CLUSTER_ACTION privilege. Do I
>>> understand it correctly?
>>>
>>> There is no 100% safe way to distinguish between raw request and forward
>> request if you take malicious request into consideration, which could happen
>> anyway if the attacker prepares those requests to pass the checks for a
>> pre-KIP-500 cluster.
>> We could at most know whether it is sent to the control plane, or the
>> data plane, and whether it has extra header fields such as initial
>> principal and client id defined. For a non-malicious request
>> going into the control plane, it must be sent from a valid broker, which
>> is a prerequisite to check its
>> CLUSTER_ACTION principal. Take AlterConfig for an example, the intended
>> workflow for a *KIP-590 broker* would be:
>>
>> Step 1. Check the request context to see if this is a forwarding request,
>> by checking whether it is from control plane and uses extra header fields
>> 1.1 if it is a forwarding request, use CLUSTER_ACTION to verify
>> the given resource
>> 1.2 if the resource is authorized, and if this is the active
>> controller, process it
>> 1.3 if the resource is authorized but this is not the active
>> controller, return NOT_CONTROLLER to the sender (forwarding broker) for
>> retry
>> 1.4 if the resource is not authorized, return
>> CLUSTER_AUTHORIZATION_FAILURE to propagate back to the original client
>> through forwarding broker
>> Step 2. If the request is not a forwarding request
>> 2.1 Verify with normal principal as ALTER on the given resources
>> 2.2 If the resource is authorized, and this is the active

Re: [VOTE] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[Yuriy Badalyantc]

Hi everybody.

There was some minor change since the voting process started (nullSerde
added). Let's continue to vote.

-Yuriy.

On Thu, Jul 9, 2020 at 10:00 PM John Roesler  wrote:

> Thanks Yuriy,
>
> I'm +1 (binding)
>
> -John
>
> On Wed, Jul 8, 2020, at 23:08, Yuriy Badalyantc wrote:
> > Hi everybody
> >
> > I would like to start a vote  for KIP-616:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-616%3A+Rename+implicit+Serdes+instances+in+kafka-streams-scala
> >
> > This KIP fixes name clash in the org.apache.kafka.streams.scala.Serdes.
> >
> > -Yuriy
> >
>

Build failed in Jenkins: kafka-trunk-jdk14 #344

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10162; Use Token Bucket algorithm for controller mutation quota

[github] KAFKA-10316: Consider renaming getter method for Interactive Queries

[github] MINOR: fix HTML (#9127)

[github] KAFKA-9274: Remove `retries` from InternalTopicManager (#9060)


--
[...truncated 3.21 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

Build failed in Jenkins: kafka-trunk-jdk11 #1692

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10162; Use Token Bucket algorithm for controller mutation quota

[github] KAFKA-10316: Consider renaming getter method for Interactive Queries

[github] MINOR: fix HTML (#9127)

[github] KAFKA-9274: Remove `retries` from InternalTopicManager (#9060)


--
[...truncated 3.21 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

Re: [DISCUSS] Apache Kafka 2.6.0 release

[Randall Hauch]

Just to follow up, we've released AK 2.6.0 today. See the announcement
thread.

On Tue, Jul 14, 2020 at 4:01 PM Randall Hauch  wrote:

> I've just announced 2.6.0 RC0 in a vote thread to this list. If you find
> any issues, please reply to that "[VOTE] 2.6.0 RC0" thread.
>
> Thanks, and best regards!
>
> Randall
>
> On Fri, Jul 10, 2020 at 7:44 PM Matthias J. Sax  wrote:
>
>> Randall,
>>
>> we found another blocker:
>> https://issues.apache.org/jira/browse/KAFKA-10262
>>
>> Luckily, we have already a PR for it.
>>
>>
>> -Matthias
>>
>>
>> On 7/8/20 3:05 PM, Sophie Blee-Goldman wrote:
>> > Hey Randall,
>> >
>> > We just discovered another regression in 2.6:
>> > https://issues.apache.org/jira/browse/KAFKA-10249
>> >
>> > The fix is extremely straightforward -- only about two lines of actual
>> > code -- and low risk. It is a new regression introduced in 2.6 and
>> affects
>> > all Streams apps with any suppression or other in-memory state.
>> >
>> > The PR is already ready here: https://github.com/apache/kafka/pull/8996
>> >
>> > Best,
>> > Sophie
>> >
>> > On Wed, Jul 8, 2020 at 10:59 AM John Roesler 
>> wrote:
>> >
>> >> Hi Randall,
>> >>
>> >> While developing system tests, I've just unearthed a new 2.6
>> regression:
>> >> https://issues.apache.org/jira/browse/KAFKA-10247
>> >>
>> >> I've got a PR in progress. Hoping to finish it up today:
>> >> https://github.com/apache/kafka/pull/8994
>> >>
>> >> Sorry for the trouble,
>> >> -John
>> >>
>> >> On Mon, Jun 29, 2020, at 09:29, Randall Hauch wrote:
>> >>> Thanks for raising this, David. I agree it makes sense to include this
>> >> fix
>> >>> in 2.6, so I've adjusted the "Fix Version(s)" field to include
>> '2.6.0'.
>> >>>
>> >>> Best regards,
>> >>>
>> >>> Randall
>> >>>
>> >>> On Mon, Jun 29, 2020 at 8:25 AM David Jacot 
>> wrote:
>> >>>
>>  Hi Randall,
>> 
>>  We have discovered an annoying issue that we introduced in 2.5:
>> 
>>  Describing topics with the command line tool fails if the user does
>> not
>>  have the
>>  privileges to access the ListPartitionReassignments API. I believe
>> that
>>  this is the
>>  case for most non-admin users.
>> 
>>  I propose to include the fix in 2.6. The fix is trivial so low risk.
>> >> What
>>  do you think?
>> 
>>  JIRA: https://issues.apache.org/jira/browse/KAFKA-10212
>>  PR: https://github.com/apache/kafka/pull/8947
>> 
>>  Best,
>>  David
>> 
>>  On Sat, Jun 27, 2020 at 4:39 AM John Roesler 
>> >> wrote:
>> 
>> > Hi Randall,
>> >
>> > I neglected to notify this thread when I merged the fix for
>> > https://issues.apache.org/jira/browse/KAFKA-10185
>> > on June 19th. I'm sorry about that oversight. It is marked with
>> > a fix version of 2.6.0.
>> >
>> > On a side node, I have a fix for KAFKA-10173, which I'm merging
>> > and backporting right now.
>> >
>> > Thanks for managing the release,
>> > -John
>> >
>> > On Thu, Jun 25, 2020, at 10:23, Randall Hauch wrote:
>> >> Thanks for the update, folks!
>> >>
>> >> Based upon Jira [1], we currently have 4 issues that are considered
>> >> blockers for the 2.6.0 release and production of RCs:
>> >>
>> >>- https://issues.apache.org/jira/browse/KAFKA-10134 - High CPU
>>  issue
>> >>during rebalance in Kafka consumer after upgrading to 2.5
>>  (unassigned)
>> >>- https://issues.apache.org/jira/browse/KAFKA-10143 - Can no
>> >> longer
>> >>change replication throttle with reassignment tool (Jason G)
>> >>- https://issues.apache.org/jira/browse/KAFKA-10166 - Excessive
>> >>TaskCorruptedException seen in testing (Sophie, Bruno)
>> >>- https://issues.apache.org/jira/browse/KAFKA-10173
>> >>- BufferUnderflowException during Kafka Streams Upgrade (John R)
>> >>
>> >> and one critical issue that may be a regression that at this time
>> >> will
>> > not
>> >> block production of RCs:
>> >>
>> >>- https://issues.apache.org/jira/browse/KAFKA-10017 - Flaky
>> >> Test
>> >>EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta
>> > (Matthias)
>> >>
>> >> and one build/release issue we'd like to fix if possible but will
>> >> not
>> > block
>> >> RCs or the release:
>> >>
>> >>- https://issues.apache.org/jira/browse/KAFKA-9381
>> >>- kafka-streams-scala: Javadocs + Scaladocs not published on
>> >> maven
>> > central
>> >>(me)
>> >>
>> >> I'm working with the assignees and reporters of these issues (via
>> > comments
>> >> on the issues) to identify an ETA and to track progress. Anyone is
>> > welcome
>> >> to chime in on those issues.
>> >>
>> >> At this time, no other changes (other than PRs that only
>> >> fix/improve
>> > tests)
>> >> should be merged to the `2.6` branch. If you think you've
>> >> identified a
>> 

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Gwen Shapira]

Yay!

Congratulations everyone :) And thank you for the release, Randall!

On Thu, Aug 6, 2020 at 7:21 AM Randall Hauch  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.6.0
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Significant performance improvements, especially when the broker has
> large numbers of partitions
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes other features, 74 improvements, 175 bug fixes,
> plus other changes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.6.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 127 contributors to this release!
>
> 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
> Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
> Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
> Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
> Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
> Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
> feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
> Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
> Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
> Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
> Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
> Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
> Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
> Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
> Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
> Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
> Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
> Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> Randall Hauch
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-651 - Support PEM format for SSL certificates and private key

[David Jacot]

Supporting PEM is really nice. Thanks, Rajini.

+1 (non-binding)

On Thu, Aug 6, 2020 at 9:18 PM Gwen Shapira  wrote:

> +1 (binding)
> Thank you for driving this, Rajini
>
> On Thu, Aug 6, 2020 at 10:43 AM Rajini Sivaram 
> wrote:
>
> > Hi all,
> >
> > I would like to start vote on KIP-651 to support SSL key stores and trust
> > stores in PEM format:
> >
> >-
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> >
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
> >
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [VOTE] KIP-651 - Support PEM format for SSL certificates and private key

[Gwen Shapira]

+1 (binding)
Thank you for driving this, Rajini

On Thu, Aug 6, 2020 at 10:43 AM Rajini Sivaram 
wrote:

> Hi all,
>
> I would like to start vote on KIP-651 to support SSL key stores and trust
> stores in PEM format:
>
>-
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
>
>
> Thank you...
>
> Regards,
>
> Rajini
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-651 - Support PEM format for SSL certificates and private key

[Maulin Vasavada]

+1 (non-binding)

On Thu, Aug 6, 2020 at 11:36 AM Ron Dagostino  wrote:

> +1 (non-binding)
>
> Ron
>
> > On Aug 6, 2020, at 1:42 PM, Rajini Sivaram 
> wrote:
> >
> > Hi all,
> >
> > I would like to start vote on KIP-651 to support SSL key stores and trust
> > stores in PEM format:
> >
> >   -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> >
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
>

Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and private key

[Maulin Vasavada]

Looks great Rajini. Indeed PEM is a standard needed to be supported in a
first class way on Kafka.

On Wed, Aug 5, 2020 at 9:55 PM Manikumar  wrote:

> Thanks for the KIP. LGTM.
>
> On Wed, Aug 5, 2020 at 3:43 AM Ron Dagostino  wrote:
>
> > Yes, Rajinit, it looks good -- an excellently-written KIP.  Thanks!
> >
> > Ron
> >
> > On Mon, Aug 3, 2020 at 9:17 AM Andrew Otto  wrote:
> >
> > > Yes please!  I had to build custom tooling to automate generation and
> > > distribution of java truststore and keystore files at Wikimedia for
> Kafka
> > > brokers and clients.  PEM is much more standard outside of the Java
> world
> > > and it would be simpler to not have to maintain custom tooling for
> > > keystores.
> > >
> > > - Andrew Otto
> > >
> > > On Mon, Aug 3, 2020 at 7:29 AM Rajini Sivaram  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have submitted KIP-651 to support PEM format for SSL key and trust
> > > > stores:
> > > >
> > > >-
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> > > >
> > > > This enables better integration with other Kafka features like
> dynamic
> > > > config update and password protection.
> > > >
> > > > Feedback and suggestions are welcome.
> > > >
> > > > Thank you...
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
>

Re: New Website Layout

[Ben Weintraub]

Plus one to Tom's request - the ability to easily generate links to
specific config options is extremely valuable.

On Thu, Aug 6, 2020 at 10:09 AM Tom Bentley  wrote:

> Hi Ben,
>
> The documentation for the configs (broker, producer etc) used to function
> as links as well as anchors, which made the url fragments more
> discoverable, because you could click on the link and then copy+paste the
> browser URL:
>
> 
>href="#batch.size">batch.size
> 
>
> What seems to have happened with the new layout is the  tags are empty,
> and no longer enclose the config name,
>
> 
>   
>   batch.size
> 
>
> meaning you can't click on the link to copy and paste the URL. Could the
> old behaviour be restored?
>
> Thanks,
>
> Tom
>
> On Wed, Aug 5, 2020 at 12:43 PM Luke Chen  wrote:
>
> > When entering streams doc, it'll always show:
> > *You're viewing documentation for an older version of Kafka - check out
> our
> > current documentation here.*
> >
> >
> >
> > On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford  wrote:
> >
> > > Thanks for the PR and feedback Michael. Appreciated.
> > >
> > > On Wed, 5 Aug 2020 at 10:49, Mickael Maison 
> > > wrote:
> > >
> > > > Thank you, it looks great!
> > > >
> > > > I found a couple of small issues:
> > > > - It's not rendering correctly with http.
> > > > - It's printing "called" to the console. I opened a PR to remove the
> > > > console.log() call: https://github.com/apache/kafka-site/pull/278
> > > >
> > > > On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford 
> wrote:
> > > > >
> > > > > The new website layout has gone live as you may have seen. There
> are
> > a
> > > > > couple of rendering issues in the streams developer guide that
> we're
> > > > > getting addressed. If anyone spots anything else could they please
> > > reply
> > > > to
> > > > > this thread.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Ben
> > > > >
> > > > > On Fri, 26 Jun 2020 at 11:48, Ben Stopford 
> wrote:
> > > > >
> > > > > > Hey folks
> > > > > >
> > > > > > We've made some updates to the website's look and feel. There is
> a
> > > > staged
> > > > > > version in the link below.
> > > > > >
> > > > > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > > > > username: kafka
> > > > > > password: streaming
> > > > > >
> > > > > > Comments welcomed.
> > > > > >
> > > > > > Ben
> > > > > >
> > > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Ben Stopford
> > >
> > > Lead Technologist, Office of the CTO
> > >
> > > 
> > >
> >
>

Re: [VOTE] KIP-651 - Support PEM format for SSL certificates and private key

[Ron Dagostino]

+1 (non-binding)

Ron

> On Aug 6, 2020, at 1:42 PM, Rajini Sivaram  wrote:
> 
> Hi all,
> 
> I would like to start vote on KIP-651 to support SSL key stores and trust
> stores in PEM format:
> 
>   -
>   
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> 
> 
> Thank you...
> 
> Regards,
> 
> Rajini

[jira] [Created] (KAFKA-10369) Introduce Distinct operation in KStream

[Ivan Ponomarev (Jira)]

Ivan Ponomarev created KAFKA-10369:
--

 Summary: Introduce Distinct operation in KStream
 Key: KAFKA-10369
 URL: https://issues.apache.org/jira/browse/KAFKA-10369
 Project: Kafka
  Issue Type: Improvement
Reporter: Ivan Ponomarev
Assignee: Ivan Ponomarev


Message deduplication is a common task.

One example: we might have multiple data sources each reporting its state 
periodically with a relatively high frequency, their current states should be 
stored in a database. In case the actual change of the state occurs with a 
lower frequency than it is reported, in order to reduce the number of writes to 
the database we might want to filter out duplicated messages using Kafka 
Streams.

'Distinct' operation is common in data processing, e. g.
 * Java Stream has [distinct() 
|https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#distinct--]
 operation,
 * SQL has DISTINCT keyword.

 

Hence it is natural to expect the similar functionality from Kafka Streams.

Although Kafka Streams Tutorials contains an 
[example|https://kafka-tutorials.confluent.io/finding-distinct-events/kstreams.html]
 of how distinct can be emulated , but this example is complicated: it involves 
low-level coding with local state store and a custom transformer. It might be 
much more convenient to have distinct as a first-class DSL operation.

Due to 'infinite' nature of KStream, distinct operation should be windowed, 
similar to windowed joins and aggregations for KStreams.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Jun,
Thanks for your comments.

> At the high level, that approach sounds reasonable to
me. It would be useful to document how RLMM handles overlapping archived
offset ranges and how those overlapping segments are deleted through
retention.

Sure, we will document that in the KIP.

>How is the remaining part of the KIP coming along? To me, the two biggest
missing items are (1) more detailed documentation on how all the new APIs
are being used and (2) metadata format and usage in the internal
topic __remote_log_metadata.

We are working on updating APIs based on the recent discussions and
get the perf numbers by plugging in rocksdb as a cache store for RLMM.
We will update the KIP with the updated APIs and with the above
requested details in a few days and let you know.

Thanks,
Satish.




On Wed, Aug 5, 2020 at 12:49 AM Jun Rao  wrote:
>
> Hi, Ying, Satish,
>
> Thanks for the reply. At the high level, that approach sounds reasonable to
> me. It would be useful to document how RLMM handles overlapping archived
> offset ranges and how those overlapping segments are deleted through
> retention.
>
> How is the remaining part of the KIP coming along? To me, the two biggest
> missing items are (1) more detailed documentation on how all the new APIs
> are being used and (2) metadata format and usage in the internal
> topic __remote_log_metadata.
>
> Thanks,
>
> Jun
>
> On Tue, Aug 4, 2020 at 8:32 AM Satish Duggana 
> wrote:
>
> > Hi Jun,
> > Thanks for your comment,
> >
> > 1001. Using the new leader as the source of truth may be fine too. What's
> > not clear to me is when a follower takes over as the new leader, from which
> > offset does it start archiving to the block storage. I assume that the new
> > leader starts from the latest archived ooffset by the previous leader, but
> > it seems that's not the case. It would be useful to document this in the
> > Wiki.
> >
> > When a follower becomes a leader it needs to findout the offset from
> > which the segments to be copied to remote storage. This is found by
> > traversing from the the latest leader epoch from leader epoch history
> > and find the highest offset of a segment with that epoch copied into
> > remote storage by using respective RLMM APIs. If it can not find an
> > entry then it checks for the previous leader epoch till it finds an
> > entry, If there are no entries till the earliest leader epoch in
> > leader epoch cache then it starts copying the segments from the
> > earliest epoch entry’s offset.
> > Added an example in the KIP here[1]. We will update RLMM APIs in the KIP.
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Followertoleadertransition
> >
> > Satish.
> >
> >
> > On Tue, Aug 4, 2020 at 9:00 PM Satish Duggana 
> > wrote:
> > >
> > > Hi Ying,
> > > Thanks for your comment.
> > >
> > > 1001. Using the new leader as the source of truth may be fine too. What's
> > > not clear to me is when a follower takes over as the new leader, from
> > which
> > > offset does it start archiving to the block storage. I assume that the
> > new
> > > leader starts from the latest archived ooffset by the previous leader,
> > but
> > > it seems that's not the case. It would be useful to document this in the
> > > Wiki.
> > >
> > > When a follower becomes a leader it needs to findout the offset from
> > > which the segments to be copied to remote storage. This is found by
> > > traversing from the the latest leader epoch from leader epoch history
> > > and find the highest offset of a segment with that epoch copied into
> > > remote storage by using respective RLMM APIs. If it can not find an
> > > entry then it checks for the previous leader epoch till it finds an
> > > entry, If there are no entries till the earliest leader epoch in
> > > leader epoch cache then it starts copying the segments from the
> > > earliest epoch entry’s offset.
> > > Added an example in the KIP here[1]. We will update RLMM APIs in the KIP.
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Followertoleadertransition
> > >
> > >
> > > Satish.
> > >
> > >
> > > On Tue, Aug 4, 2020 at 10:28 AM Ying Zheng 
> > wrote:
> > > >
> > > > Hi Jun,
> > > >
> > > > Thank you for the comment! The current KIP is not very clear about this
> > > > part.
> > > >
> > > > 1001. The new leader will start archiving from the earliest local
> > segment
> > > > that is not fully
> > > > covered by the "valid" remote data. "valid" means the (offset, leader
> > > > epoch) pair is valid
> > > > based on the leader-epoch history.
> > > >
> > > > There are some edge cases where the same offset range (with the same
> > leader
> > > > epoch) can
> > > > be copied to the remote storage more than once. But this kind of
> > > > duplication shouldn't be a
> > > > problem.
> > > >
> > > > Staish is going to explain the details in the KIP with examples.
> > > >
> > > >

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Konstantine Karantasis]

Thanks for the great release management Randall and thanks to the
contributors for all the nice improvements and new features!

Konstantine


On Thu, Aug 6, 2020 at 9:40 AM Guozhang Wang  wrote:

> Thank you so much for driving the release Randall!
>
> Guozhang
>
> On Thu, Aug 6, 2020 at 7:35 AM Ismael Juma  wrote:
>
> > Thanks for driving the releas, Randall. Congratulations to all the
> > contributors! :)
> >
> > Ismael
> >
> > On Thu, Aug 6, 2020, 7:21 AM Randall Hauch  wrote:
> >
> > > The Apache Kafka community is pleased to announce the release for
> Apache
> > > Kafka 2.6.0
> > >
> > > * TLSv1.3 has been enabled by default for Java 11 or newer.
> > > * Significant performance improvements, especially when the broker has
> > > large numbers of partitions
> > > * Smooth scaling out of Kafka Streams applications
> > > * Kafka Streams support for emit on change
> > > * New metrics for better operational insight
> > > * Kafka Connect can automatically create topics for source connectors
> > > * Improved error reporting options for sink connectors in Kafka Connect
> > > * New Filter and conditional SMTs in Kafka Connect
> > > * The default value for the `client.dns.lookup` configuration is
> > > now `use_all_dns_ips`
> > > * Upgrade Zookeeper to 3.5.8
> > >
> > > This release also includes other features, 74 improvements, 175 bug
> > fixes,
> > > plus other changes.
> > >
> > > All of the changes in this release can be found in the release notes:
> > > https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
> > >
> > >
> > > You can download the source and binary release (Scala 2.12 and 2.13)
> > from:
> > > https://kafka.apache.org/downloads#2.6.0
> > >
> > >
> > >
> >
> ---
> > >
> > >
> > > Apache Kafka is a distributed streaming platform with four core APIs:
> > >
> > >
> > > ** The Producer API allows an application to publish a stream of
> records
> > to
> > > one or more Kafka topics.
> > >
> > > ** The Consumer API allows an application to subscribe to one or more
> > > topics and process the stream of records produced to them.
> > >
> > > ** The Streams API allows an application to act as a stream processor,
> > > consuming an input stream from one or more topics and producing an
> > > output stream to one or more output topics, effectively transforming
> the
> > > input streams to output streams.
> > >
> > > ** The Connector API allows building and running reusable producers or
> > > consumers that connect Kafka topics to existing applications or data
> > > systems. For example, a connector to a relational database might
> > > capture every change to a table.
> > >
> > >
> > > With these APIs, Kafka can be used for two broad classes of
> application:
> > >
> > > ** Building real-time streaming data pipelines that reliably get data
> > > between systems or applications.
> > >
> > > ** Building real-time streaming applications that transform or react
> > > to the streams of data.
> > >
> > >
> > > Apache Kafka is in use at large and small companies worldwide,
> including
> > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> Rabobank,
> > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > >
> > > A big thank you for the following 127 contributors to this release!
> > >
> > > 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam
> Brahma,
> > > Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> > > Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston,
> > avalsa,
> > > Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen,
> > Brian
> > > Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai,
> > Chris
> > > Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur,
> David
> > > Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> > > Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava,
> > fantayeneh,
> > > feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> > > Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma,
> > Jason
> > > Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder,
> Jiamei
> > > Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge
> > Esteban
> > > Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis,
> > Kowshik
> > > Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> > > Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong,
> Magnus
> > > Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> > > maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch,
> Navina
> > > Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> > > Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram,
> Randall
> > > Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> 

Re: [VOTE] KIP-632: Add DirectoryConfigProvider

[Konstantine Karantasis]

A bit late on this thread, but I've reviewed the KIP and I'm also +1.

Thanks Tom. Happy to see this improvement making it in.

Konstantine

On Thu, Aug 6, 2020 at 10:27 AM Tom Bentley  wrote:

> With 3 binding +1 (Manikumar, Mickael and Rajini) and 2 non-binding +1
> (David and William) the vote passes.
>
> Thanks to all who voted.
>
> Tom
>
> On Thu, Aug 6, 2020 at 12:39 PM Rajini Sivaram 
> wrote:
>
> > Hi Tom,
> >
> > +1 (binding)
> >
> > Thanks for the KIP. Apart from being useful when using frameworks like
> > Kubernetes, it also seems like a neater way of storing complex
> > multi-line configs without embedding into a property file.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Thu, Aug 6, 2020 at 12:29 PM William Reynolds <
> > william.reyno...@instaclustr.com> wrote:
> >
> > > +1 (non binding) Looks like a good addition!
> > >
> > > On 06/08/2020, Tom Bentley  wrote:
> > > > This pretty minor change has 2 binding and 1 non-binding votes. It
> > would
> > > be
> > > > great if more people could take a look and either vote or give
> feedback
> > > > about how it should be improved.
> > > >
> > > > Many thanks,
> > > >
> > > > Tom
> > > >
> > > > On Wed, Jul 8, 2020 at 7:07 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> +1 (binding)
> > > >> Thanks
> > > >>
> > > >> On Wed, Jul 8, 2020 at 11:31 AM Manikumar <
> manikumar.re...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > +1 (bindig)
> > > >> >
> > > >> > Thanks for the KIP.
> > > >> >
> > > >> > On Tue, Jul 7, 2020 at 10:30 PM David Jacot 
> > > >> > wrote:
> > > >> >
> > > >> > > +1 (non-binding). Thanks for the KIP!
> > > >> > >
> > > >> > > On Tue, Jul 7, 2020 at 12:54 PM Tom Bentley <
> tbent...@redhat.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > Hi,
> > > >> > > >
> > > >> > > > I'd like to start a vote on KIP-632, which is about making the
> > > >> > > > config
> > > >> > > > provider mechanism more ergonomic on Kubernetes:
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+DirectoryConfigProvider
> > > >> > > >
> > > >> > > > Please take a look if you have time.
> > > >> > > >
> > > >> > > > Many thanks,
> > > >> > > >
> > > >> > > > Tom
> > > >> > > >
> > > >> > >
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > >
> > >
> > >
> > > *William Reynolds**Technical Operations Engineer*
> > >
> > >
> > >    <
> > https://twitter.com/instaclustr>
> > > 
> > >
> > > Read our latest technical blog posts here
> > > .
> > >
> > > This email has been sent on behalf of Instaclustr Pty. Limited
> > (Australia)
> > > and Instaclustr Inc (USA).
> > >
> > > This email and any attachments may contain confidential and legally
> > > privileged information.  If you are not the intended recipient, do not
> > copy
> > > or disclose its content, but please reply to this email immediately and
> > > highlight the error to the sender and then immediately delete the
> > message.
> > >
> > > Instaclustr values your privacy. Our privacy policy can be found at
> > > https://www.instaclustr.com/company/policies/privacy-policy
> > >
> >
>

[VOTE] KIP-651 - Support PEM format for SSL certificates and private key

[Rajini Sivaram]

Hi all,

I would like to start vote on KIP-651 to support SSL key stores and trust
stores in PEM format:

   -
   
https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key


Thank you...

Regards,

Rajini

Re: [VOTE] KIP-632: Add DirectoryConfigProvider

[Tom Bentley]

With 3 binding +1 (Manikumar, Mickael and Rajini) and 2 non-binding +1
(David and William) the vote passes.

Thanks to all who voted.

Tom

On Thu, Aug 6, 2020 at 12:39 PM Rajini Sivaram 
wrote:

> Hi Tom,
>
> +1 (binding)
>
> Thanks for the KIP. Apart from being useful when using frameworks like
> Kubernetes, it also seems like a neater way of storing complex
> multi-line configs without embedding into a property file.
>
> Regards,
>
> Rajini
>
>
> On Thu, Aug 6, 2020 at 12:29 PM William Reynolds <
> william.reyno...@instaclustr.com> wrote:
>
> > +1 (non binding) Looks like a good addition!
> >
> > On 06/08/2020, Tom Bentley  wrote:
> > > This pretty minor change has 2 binding and 1 non-binding votes. It
> would
> > be
> > > great if more people could take a look and either vote or give feedback
> > > about how it should be improved.
> > >
> > > Many thanks,
> > >
> > > Tom
> > >
> > > On Wed, Jul 8, 2020 at 7:07 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> +1 (binding)
> > >> Thanks
> > >>
> > >> On Wed, Jul 8, 2020 at 11:31 AM Manikumar 
> > >> wrote:
> > >> >
> > >> > +1 (bindig)
> > >> >
> > >> > Thanks for the KIP.
> > >> >
> > >> > On Tue, Jul 7, 2020 at 10:30 PM David Jacot 
> > >> > wrote:
> > >> >
> > >> > > +1 (non-binding). Thanks for the KIP!
> > >> > >
> > >> > > On Tue, Jul 7, 2020 at 12:54 PM Tom Bentley 
> > >> wrote:
> > >> > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > I'd like to start a vote on KIP-632, which is about making the
> > >> > > > config
> > >> > > > provider mechanism more ergonomic on Kubernetes:
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+DirectoryConfigProvider
> > >> > > >
> > >> > > > Please take a look if you have time.
> > >> > > >
> > >> > > > Many thanks,
> > >> > > >
> > >> > > > Tom
> > >> > > >
> > >> > >
> > >>
> > >>
> > >
> >
> >
> > --
> >
> >
> >
> > *William Reynolds**Technical Operations Engineer*
> >
> >
> >    <
> https://twitter.com/instaclustr>
> > 
> >
> > Read our latest technical blog posts here
> > .
> >
> > This email has been sent on behalf of Instaclustr Pty. Limited
> (Australia)
> > and Instaclustr Inc (USA).
> >
> > This email and any attachments may contain confidential and legally
> > privileged information.  If you are not the intended recipient, do not
> copy
> > or disclose its content, but please reply to this email immediately and
> > highlight the error to the sender and then immediately delete the
> message.
> >
> > Instaclustr values your privacy. Our privacy policy can be found at
> > https://www.instaclustr.com/company/policies/privacy-policy
> >
>

Re: New Website Layout

[Tom Bentley]

Hi Ben,

The documentation for the configs (broker, producer etc) used to function
as links as well as anchors, which made the url fragments more
discoverable, because you could click on the link and then copy+paste the
browser URL:


  batch.size


What seems to have happened with the new layout is the  tags are empty,
and no longer enclose the config name,


  
  batch.size


meaning you can't click on the link to copy and paste the URL. Could the
old behaviour be restored?

Thanks,

Tom

On Wed, Aug 5, 2020 at 12:43 PM Luke Chen  wrote:

> When entering streams doc, it'll always show:
> *You're viewing documentation for an older version of Kafka - check out our
> current documentation here.*
>
>
>
> On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford  wrote:
>
> > Thanks for the PR and feedback Michael. Appreciated.
> >
> > On Wed, 5 Aug 2020 at 10:49, Mickael Maison 
> > wrote:
> >
> > > Thank you, it looks great!
> > >
> > > I found a couple of small issues:
> > > - It's not rendering correctly with http.
> > > - It's printing "called" to the console. I opened a PR to remove the
> > > console.log() call: https://github.com/apache/kafka-site/pull/278
> > >
> > > On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford  wrote:
> > > >
> > > > The new website layout has gone live as you may have seen. There are
> a
> > > > couple of rendering issues in the streams developer guide that we're
> > > > getting addressed. If anyone spots anything else could they please
> > reply
> > > to
> > > > this thread.
> > > >
> > > > Thanks
> > > >
> > > > Ben
> > > >
> > > > On Fri, 26 Jun 2020 at 11:48, Ben Stopford  wrote:
> > > >
> > > > > Hey folks
> > > > >
> > > > > We've made some updates to the website's look and feel. There is a
> > > staged
> > > > > version in the link below.
> > > > >
> > > > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > > > username: kafka
> > > > > password: streaming
> > > > >
> > > > > Comments welcomed.
> > > > >
> > > > > Ben
> > > > >
> > > > >
> > >
> >
> >
> > --
> >
> > Ben Stopford
> >
> > Lead Technologist, Office of the CTO
> >
> > 
> >
>

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #284: MINOR: Change the arrow direction based on the view state is expanded or not

[GitBox]

scott-confluent commented on a change in pull request #284:
URL: https://github.com/apache/kafka-site/pull/284#discussion_r466554238



##
File path: includes/_footer.htm
##
@@ -329,7 +329,13 @@
var docsHandle = 
document.querySelector('.toc-handle');
 
function toggleDocsWidth() {
-   
docsContainer.classList.toggle('expanded');
+   let isExpanded = 
docsContainer.classList.toggle('expanded');
+   // change the arrow direction based on 
the view state is expanded or not
+   if (isExpanded) {

Review comment:
   This is technically only working because JavaScript is lax on types and 
the `DOMTokenList` returned by  `classList.toggle` is empty resulting in a 
falsy value. We should probably do `isExapnded.contains('expanded')` as the 
condition to be more explicit.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Guozhang Wang]

Thank you so much for driving the release Randall!

Guozhang

On Thu, Aug 6, 2020 at 7:35 AM Ismael Juma  wrote:

> Thanks for driving the releas, Randall. Congratulations to all the
> contributors! :)
>
> Ismael
>
> On Thu, Aug 6, 2020, 7:21 AM Randall Hauch  wrote:
>
> > The Apache Kafka community is pleased to announce the release for Apache
> > Kafka 2.6.0
> >
> > * TLSv1.3 has been enabled by default for Java 11 or newer.
> > * Significant performance improvements, especially when the broker has
> > large numbers of partitions
> > * Smooth scaling out of Kafka Streams applications
> > * Kafka Streams support for emit on change
> > * New metrics for better operational insight
> > * Kafka Connect can automatically create topics for source connectors
> > * Improved error reporting options for sink connectors in Kafka Connect
> > * New Filter and conditional SMTs in Kafka Connect
> > * The default value for the `client.dns.lookup` configuration is
> > now `use_all_dns_ips`
> > * Upgrade Zookeeper to 3.5.8
> >
> > This release also includes other features, 74 improvements, 175 bug
> fixes,
> > plus other changes.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
> >
> >
> > You can download the source and binary release (Scala 2.12 and 2.13)
> from:
> > https://kafka.apache.org/downloads#2.6.0
> >
> >
> >
> ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records
> to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 127 contributors to this release!
> >
> > 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
> > Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> > Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston,
> avalsa,
> > Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen,
> Brian
> > Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai,
> Chris
> > Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
> > Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> > Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava,
> fantayeneh,
> > feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> > Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma,
> Jason
> > Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
> > Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge
> Esteban
> > Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis,
> Kowshik
> > Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> > Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
> > Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> > maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
> > Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> > Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
> > Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> > Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks,
> sebwills,
> > Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
> > Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
> > Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth
> chandar,
> > Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋
> >
> > We welcome your help and feedback. For more 

Build failed in Jenkins: kafka-trunk-jdk8 #4767

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10162; Use Token Bucket algorithm for controller mutation quota


--
[...truncated 1.42 MB...]
org.apache.kafka.common.network.SslSelectorTest > testCloseOldestConnection 
PASSED

org.apache.kafka.common.network.SslSelectorTest > testServerDisconnect STARTED

org.apache.kafka.common.network.SslSelectorTest > testServerDisconnect PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testMetricsCleanupOnSelectorClose STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testMetricsCleanupOnSelectorClose PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testPartialSendAndReceiveReflectedInMetrics STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testPartialSendAndReceiveReflectedInMetrics PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testWriteCompletesSendWithNoBytesWritten STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testWriteCompletesSendWithNoBytesWritten PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testIdleExpiryWithoutReadyKeys STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testIdleExpiryWithoutReadyKeys PASSED

org.apache.kafka.common.network.SslSelectorTest > testConnectionsByClientMetric 
STARTED

org.apache.kafka.common.network.SslSelectorTest > testConnectionsByClientMetric 
PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testInboundConnectionsCountInConnectionCreationMetric STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testInboundConnectionsCountInConnectionCreationMetric PASSED

org.apache.kafka.common.network.SslSelectorTest > testNoRouteToHost STARTED

org.apache.kafka.common.network.SslSelectorTest > testNoRouteToHost PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testClearCompletedSendsAndReceives STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testClearCompletedSendsAndReceives PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testPartialReceiveGracefulClose STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testPartialReceiveGracefulClose PASSED

org.apache.kafka.common.network.SslSelectorTest > testNormalOperation STARTED

org.apache.kafka.common.network.SslSelectorTest > testNormalOperation PASSED

org.apache.kafka.common.network.SslSelectorTest > testMuteOnOOM STARTED

org.apache.kafka.common.network.SslSelectorTest > testMuteOnOOM PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testCloseOldestConnectionWithMultiplePendingReceives STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testCloseOldestConnectionWithMultiplePendingReceives PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testExpireClosedConnectionWithPendingReceives STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testExpireClosedConnectionWithPendingReceives PASSED

org.apache.kafka.common.network.SslSelectorTest > testConnectionRefused STARTED

org.apache.kafka.common.network.SslSelectorTest > testConnectionRefused PASSED

org.apache.kafka.common.network.SslSelectorTest > 
testChannelCloseWhileProcessingReceives STARTED

org.apache.kafka.common.network.SslSelectorTest > 
testChannelCloseWhileProcessingReceives PASSED

org.apache.kafka.common.network.SslSelectorTest > testEmptyRequest STARTED

org.apache.kafka.common.network.SslSelectorTest > testEmptyRequest PASSED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteForTls12FailsForTls13 STARTED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteForTls12FailsForTls13 SKIPPED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteForTls12 STARTED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteForTls12 PASSED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteForTls13 STARTED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteForTls13 SKIPPED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteFailForServerTls12ClientTls13 STARTED

org.apache.kafka.common.network.SslTransportTls12Tls13Test > 
testCiphersSuiteFailForServerTls12ClientTls13 SKIPPED

org.apache.kafka.common.network.SaslChannelBuilderTest > 
testNativeGssapiCredentials STARTED

org.apache.kafka.common.network.SaslChannelBuilderTest > 
testNativeGssapiCredentials PASSED

org.apache.kafka.common.network.SaslChannelBuilderTest > 
testCloseBeforeConfigureIsIdempotent STARTED

org.apache.kafka.common.network.SaslChannelBuilderTest > 
testCloseBeforeConfigureIsIdempotent PASSED

org.apache.kafka.common.network.SaslChannelBuilderTest > 
testCloseAfterConfigIsIdempotent STARTED

org.apache.kafka.common.network.SaslChannelBuilderTest > 
testCloseAfterConfigIsIdempotent PASSED

org.apache.kafka.common.network.SaslChannelBuilderTest > 
testClientChannelBuilderWithBrokerConfigs STARTED

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[John Roesler]

Thanks, Yuriy,

That sounds good to me. 

-John

On Wed, Aug 5, 2020, at 00:24, Yuriy Badalyantc wrote:
> Hi John!
> 
> I understand your point. But, I'm still not sure that having a Unit serde
> is a good idea because it could be ambiguous for the user. In a KStream
> [Long, Unit] valid value could mean one the following:
> 1. Empty byte array
> 2. null
> 3. Any value (value ignored)
> 
> As a user of a library, I think all these options are more or less valid.
> That's because the Unit type in my opinion has no clear representation in
> terms of values and deserialization.
> 
> About Null serde - I'm ok with this. But, it should be a low priority
> implicit to not intersect will other implicits. I will add it.
> 
> -Yuriy
> 
> On Sat, Jul 11, 2020 at 9:18 PM John Roesler  wrote:
> 
> > Thanks, Yuriy,
> >
> > IIRC, I added VoidSerde because there are times when the key or value is
> > always null, but since ‘null’ has no type in Java, we had used the
> > ByteArraySerde or some other proxy. This is confusing, because then the
> > type is ‘byte[]’ although we intended it to always be null. It also delays
> > finding bugs because serdes themselves force runtime type checks, but the
> > ByteArraySerde accepts all data.
> >
> > Anyway, Void isn’t the type of null, but it’s close enough. Although
> > Scala’s equivalent to “void” method is Unit methods, and like Void, Unit is
> > uninstantiable, Unit has a value, also called Unit or “()”, whereas Void
> > has no value in Java. But because Void is still a reference type, it can be
> > null (I.e., it can only be null), so I’m some sense, it’s closer to the
> > Scala type Null.
> >
> > I guess Scala users would benefit from both a NullSerde and a UnitSerde,
> > but not a VoidSerde.
> >
> > Thanks for bringing it up, Matthias.
> >
> > -John
> >
> > On Fri, Jul 10, 2020, at 21:49, Yuriy Badalyantc wrote:
> > > Ok, I mentioned adding missing serdes in the Proposed Change paragraph.
> > >
> > > About VoidSerde. I didn't add it intentionally. The semantic of the Unit
> > > (scala's void) type is not clear in terms of the data. If kafka topic
> > > contains messages of type Unit, what does it actually means? That there
> > is
> > > always null? Well, for that we have a Null type. That it's an empty byte
> > > array? For that, we have an Array[Byte]. Empty string? No, it's a String.
> > > So, I decided to not include Unit serde in the built-in Serdes. And if a
> > > user will want to use the Unit type he can implement its own serde.
> > >
> > > -Yuriy
> > >
> > > On Fri, Jul 10, 2020 at 11:21 PM Matthias J. Sax 
> > wrote:
> > >
> > > > Thanks Yuriy!
> > > >
> > > > What about `VoidSerde` ? It's not listed.
> > > >
> > > > It might also be nice to add a short sentence and state that in
> > addition
> > > > to fixing the name collisions, the KIP will also close the gap of
> > > > out-of-the-box serdes and add missing Serdes that are offered in Java
> > to
> > > > Scala.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 7/10/20 7:51 AM, Yuriy Badalyantc wrote:
> > > > > Oh, ok. I have done that. Just didn't know that it was necessary.
> > > > >
> > > > > -Yuriy
> > > > >
> > > > > On Fri, Jul 10, 2020 at 9:30 PM John Roesler 
> > > > wrote:
> > > > >
> > > > >> Ah, thanks Yuriy,
> > > > >>
> > > > >> Sorry if this wasn't clear, but _all_ public API changes have to
> > > > >> be explicitly included in the KIP. Can you just enumerate all
> > > > >> the contents of the new API?
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
> > > > >>> Hi, Matthias,
> > > > >>>
> > > > >>> It's not directly mentioned in the KIP, but I added all missing
> > Java
> > > > >>> serdes. I mentioned it in the pull request description:
> > > > >>> https://github.com/apache/kafka/pull/8955
> > > > >>>
> > > > >>> And also, this KIP originally was based on a pull request where I
> > added
> > > > >>> missing java serdes :) https://github.com/apache/kafka/pull/8049
> > > > >>>
> > > > >>> -Yuriy
> > > > >>>
> > > > >>> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax 
> > > > >> wrote:
> > > > >>>
> > > >  Yuriy,
> > > > 
> > > >  thanks for the KIP update. I have one follow up thought: I checked
> > > > what
> > > >  default Serdes we offer in the Java class
> > > > 
> > > >   `org.apache.kafka.common.serialization.Serdes`
> > > > 
> > > >  and I think it would be good if we could close the gap between the
> > > > Java
> > > >  and Scala code and add the missing Java Serdes in Scala, too.
> > > > 
> > > >  It seems we are missing `Short` (Java and Scala), `Void`, `UUID`,
> > and
> > > >  `ByterBuffer`.
> > > > 
> > > >  Can we add those in addition?
> > > > 
> > > > 
> > > >  -Matthias
> > > > 
> > > >  On 7/8/20 6:45 AM, John Roesler wrote:
> > > > > Hi Yuriy,
> > > > >
> > > > > Once it seems like there’s general agreement in the 

[GitHub] [kafka-site] guozhangwang merged pull request #281: Remove mention of Twitter as a way to get added to "Powered By"

[GitBox]

guozhangwang merged pull request #281:
URL: https://github.com/apache/kafka-site/pull/281


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

[David Jacot]

Hi all,

I would like to inform you that we have slightly changed our thoughts about
the implementation
of the Token Bucket algorithm. Our initial idea was to change our existing
Rate to behave like
a Token Bucket. That works as we expected but we have realized that the
value of the Rate is
not really a sampled Rate anymore. From an observability perspective, it is
not good to change
the behavior of our Rate. Instead, we propose to implement a new
MeasurableStat that implements
the Token Bucket algorithm and to use it alongside the Rate in the Sensor.
Which this, the Rate
remains there for observability purposes and the Token Bucket is used for
enforcing the quota.
The Token Bucket metrics is exposed via a new metric named "credits" that
represents the
remaining amount of credits in the bucket. As a first step, this will be
only used for the controller
quota.

I have updated the KIP to reflect this change.

We hope this change is fine for everyone. Please, raise your concerns if
not.

Best,
David

On Tue, Jul 28, 2020 at 1:48 PM David Jacot  wrote:

> Hi all,
>
> Just a quick update. We have made good progress regarding the
> implementation
> of this KIP. The major parts are already in trunk modulo the new "rate"
> implementation
> which is still under development.
>
> I would like to change the type of the `controller_mutations_rate` from a
> Long to
> a Double. I have made various experiments and the rate can be quite low,
> especially
> in clusters with a lot of tenants. Using a Long is quite limiting here to
> fine tune small
> rates. I hope that this change is fine for everyone.
>
> Best,
> David
>
> On Mon, Jun 15, 2020 at 9:26 AM David Jacot  wrote:
>
>> Hi all,
>>
>> The vote has passed with 5 binding votes (Gwen, Rajini, Mickael, Jun and
>> Colin)
>> and 2 non-binding votes (Tom, Anna).
>>
>> Thank you all for the fruitful discussion! I'd like to particularly thank
>> Anna who has
>> heavily contributed to the design of this KIP.
>>
>> Regards,
>> David
>>
>> On Fri, Jun 12, 2020 at 10:08 PM Colin McCabe  wrote:
>>
>>> +1.  Thanks, David!
>>>
>>> best,
>>> Colin
>>>
>>> On Thu, Jun 11, 2020, at 23:51, David Jacot wrote:
>>> > Colin, Jun,
>>> >
>>> > Do the proposed error code and the updated KIP look good to you guys?
>>> I’d
>>> > like to wrap up and close the vote.
>>> >
>>> > Thanks,
>>> > David
>>> >
>>> > Le mer. 10 juin 2020 à 14:50, David Jacot  a
>>> écrit :
>>> >
>>> > > Hi Colin and Jun,
>>> > >
>>> > > I have no problem if we have to rewrite part of it when the new
>>> controller
>>> > > comes
>>> > > out. I will be more than happy to help out.
>>> > >
>>> > > Regarding KIP-590, I think that we can cope with a principal as a
>>> string
>>> > > when the
>>> > > time comes. The user entity name is defined with a string already.
>>> > >
>>> > > Regarding the name of the error, you have made a good point. I do
>>> agree
>>> > > that it
>>> > > is important to differentiate the two cases. I propose the following
>>> two
>>> > > errors:
>>> > > - THROTTLING_QUOTA_EXCEEDED - Throttling is slightly better than
>>> rate as
>>> > > we have quotas which are not rate (e.g. request quota). This one is
>>> > > retryable
>>> > > once the throttle time is passed.
>>> > > - LIMIT_QUOTA_EXCEEDED - This one would indicate that the limit has
>>> been
>>> > > reached and is a final error.
>>> > > We only need the former in this KIP. What do you think?
>>> > >
>>> > > Jun, I have added a few examples in the KIP. The new name works
>>> exactly
>>> > > like
>>> > > the existing one once it is added to the accepted dynamic configs
>>> for the
>>> > > user
>>> > > and the client entities. I have added a "Kafka Config Command"
>>> chapter in
>>> > > the
>>> > > KIP. I will also open a Jira to not forget updating the AK
>>> documentation
>>> > > once
>>> > > the KIP gets merged.
>>> > >
>>> > > Thanks,
>>> > > David
>>> > >
>>> > > On Wed, Jun 10, 2020 at 3:03 AM Jun Rao  wrote:
>>> > >
>>> > >> Hi, Colin,
>>> > >>
>>> > >> Good point. Maybe sth like THROTTLING_QUOTA_VIOLATED will make this
>>> clear.
>>> > >>
>>> > >> Hi, David,
>>> > >>
>>> > >> We added a new quota name in the KIP. You chose not to bump up the
>>> version
>>> > >> of DESCRIBE_CLIENT_QUOTAS and ALTER_CLIENT_QUOTAS, which seems ok
>>> since
>>> > >> the
>>> > >> quota name is represented as a string. However, the new quota name
>>> can be
>>> > >> used in client tools for setting and listing the quota (
>>> > >> https://kafka.apache.org/documentation/#quotas). Could you
>>> document how
>>> > >> the
>>> > >> new name will be used in those tools?
>>> > >>
>>> > >> Thanks,
>>> > >>
>>> > >> Jun
>>> > >>
>>> > >> On Tue, Jun 9, 2020 at 3:37 PM Colin McCabe 
>>> wrote:
>>> > >>
>>> > >> > On Tue, Jun 9, 2020, at 05:06, David Jacot wrote:
>>> > >> > > Hi Colin,
>>> > >> > >
>>> > >> > > Thank you for your feedback.
>>> > >> > >
>>> > >> > > Jun has summarized the situation pretty well. Thanks Jun! I
>>> would
>>> > >> like to
>>> > >> > > complement 

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Ismael Juma]

Thanks for driving the releas, Randall. Congratulations to all the
contributors! :)

Ismael

On Thu, Aug 6, 2020, 7:21 AM Randall Hauch  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.6.0
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Significant performance improvements, especially when the broker has
> large numbers of partitions
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes other features, 74 improvements, 175 bug fixes,
> plus other changes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.6.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 127 contributors to this release!
>
> 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
> Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
> Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
> Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
> Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
> Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
> feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
> Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
> Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
> Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
> Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
> Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
> Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
> Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
> Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
> Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
> Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
> Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> Randall Hauch
>

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Stanislav Kozlovski]

Thanks for driving the release Randall!
Congratulations to everybody involved - awesome work!

On Thu, Aug 6, 2020 at 5:21 PM Randall Hauch  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.6.0
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Significant performance improvements, especially when the broker has
> large numbers of partitions
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes other features, 74 improvements, 175 bug fixes,
> plus other changes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.6.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 127 contributors to this release!
>
> 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
> Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
> Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
> Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
> Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
> Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
> feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
> Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
> Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
> Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
> Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
> Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
> Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
> Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
> Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
> Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
> Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
> Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> Randall Hauch
>


-- 
Best,
Stanislav

[ANNOUNCE] Apache Kafka 2.6.0

[Randall Hauch]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.6.0

* TLSv1.3 has been enabled by default for Java 11 or newer.
* Significant performance improvements, especially when the broker has
large numbers of partitions
* Smooth scaling out of Kafka Streams applications
* Kafka Streams support for emit on change
* New metrics for better operational insight
* Kafka Connect can automatically create topics for source connectors
* Improved error reporting options for sink connectors in Kafka Connect
* New Filter and conditional SMTs in Kafka Connect
* The default value for the `client.dns.lookup` configuration is
now `use_all_dns_ips`
* Upgrade Zookeeper to 3.5.8

This release also includes other features, 74 improvements, 175 bug fixes,
plus other changes.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#2.6.0

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 127 contributors to this release!

17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Randall Hauch

[jira] [Created] (KAFKA-10368) Rack aware assignment results not balanced

[hanmingze (Jira)]

hanmingze created KAFKA-10368:
-

 Summary: Rack aware assignment results not balanced
 Key: KAFKA-10368
 URL: https://issues.apache.org/jira/browse/KAFKA-10368
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Affects Versions: 2.4.0, 1.1.1, 1.1.0, 0.11.0.0
Reporter: hanmingze


When the number of broker in two rack is inconsistent, the assignment results 
is not balanced in the broker within a rack。



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [kafka-site] rhauch merged pull request #277: 2.6.0 release

[GitBox]

rhauch merged pull request #277:
URL: https://github.com/apache/kafka-site/pull/277


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-632: Add DirectoryConfigProvider

[Rajini Sivaram]

Hi Tom,

+1 (binding)

Thanks for the KIP. Apart from being useful when using frameworks like
Kubernetes, it also seems like a neater way of storing complex
multi-line configs without embedding into a property file.

Regards,

Rajini


On Thu, Aug 6, 2020 at 12:29 PM William Reynolds <
william.reyno...@instaclustr.com> wrote:

> +1 (non binding) Looks like a good addition!
>
> On 06/08/2020, Tom Bentley  wrote:
> > This pretty minor change has 2 binding and 1 non-binding votes. It would
> be
> > great if more people could take a look and either vote or give feedback
> > about how it should be improved.
> >
> > Many thanks,
> >
> > Tom
> >
> > On Wed, Jul 8, 2020 at 7:07 PM Mickael Maison 
> > wrote:
> >
> >> +1 (binding)
> >> Thanks
> >>
> >> On Wed, Jul 8, 2020 at 11:31 AM Manikumar 
> >> wrote:
> >> >
> >> > +1 (bindig)
> >> >
> >> > Thanks for the KIP.
> >> >
> >> > On Tue, Jul 7, 2020 at 10:30 PM David Jacot 
> >> > wrote:
> >> >
> >> > > +1 (non-binding). Thanks for the KIP!
> >> > >
> >> > > On Tue, Jul 7, 2020 at 12:54 PM Tom Bentley 
> >> wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > I'd like to start a vote on KIP-632, which is about making the
> >> > > > config
> >> > > > provider mechanism more ergonomic on Kubernetes:
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+DirectoryConfigProvider
> >> > > >
> >> > > > Please take a look if you have time.
> >> > > >
> >> > > > Many thanks,
> >> > > >
> >> > > > Tom
> >> > > >
> >> > >
> >>
> >>
> >
>
>
> --
>
>
>
> *William Reynolds**Technical Operations Engineer*
>
>
>    
> 
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
> Instaclustr values your privacy. Our privacy policy can be found at
> https://www.instaclustr.com/company/policies/privacy-policy
>

Re: [VOTE] KIP-632: Add DirectoryConfigProvider

[William Reynolds]

+1 (non binding) Looks like a good addition!

On 06/08/2020, Tom Bentley  wrote:
> This pretty minor change has 2 binding and 1 non-binding votes. It would be
> great if more people could take a look and either vote or give feedback
> about how it should be improved.
>
> Many thanks,
>
> Tom
>
> On Wed, Jul 8, 2020 at 7:07 PM Mickael Maison 
> wrote:
>
>> +1 (binding)
>> Thanks
>>
>> On Wed, Jul 8, 2020 at 11:31 AM Manikumar 
>> wrote:
>> >
>> > +1 (bindig)
>> >
>> > Thanks for the KIP.
>> >
>> > On Tue, Jul 7, 2020 at 10:30 PM David Jacot 
>> > wrote:
>> >
>> > > +1 (non-binding). Thanks for the KIP!
>> > >
>> > > On Tue, Jul 7, 2020 at 12:54 PM Tom Bentley 
>> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I'd like to start a vote on KIP-632, which is about making the
>> > > > config
>> > > > provider mechanism more ergonomic on Kubernetes:
>> > > >
>> > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+DirectoryConfigProvider
>> > > >
>> > > > Please take a look if you have time.
>> > > >
>> > > > Many thanks,
>> > > >
>> > > > Tom
>> > > >
>> > >
>>
>>
>


-- 



*William Reynolds**Technical Operations Engineer*


   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Instaclustr values your privacy. Our privacy policy can be found at
https://www.instaclustr.com/company/policies/privacy-policy

Re: [VOTE] KIP-632: Add DirectoryConfigProvider

[Tom Bentley]

This pretty minor change has 2 binding and 1 non-binding votes. It would be
great if more people could take a look and either vote or give feedback
about how it should be improved.

Many thanks,

Tom

On Wed, Jul 8, 2020 at 7:07 PM Mickael Maison 
wrote:

> +1 (binding)
> Thanks
>
> On Wed, Jul 8, 2020 at 11:31 AM Manikumar 
> wrote:
> >
> > +1 (bindig)
> >
> > Thanks for the KIP.
> >
> > On Tue, Jul 7, 2020 at 10:30 PM David Jacot  wrote:
> >
> > > +1 (non-binding). Thanks for the KIP!
> > >
> > > On Tue, Jul 7, 2020 at 12:54 PM Tom Bentley 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to start a vote on KIP-632, which is about making the config
> > > > provider mechanism more ergonomic on Kubernetes:
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+DirectoryConfigProvider
> > > >
> > > > Please take a look if you have time.
> > > >
> > > > Many thanks,
> > > >
> > > > Tom
> > > >
> > >
>
>

[jira] [Created] (KAFKA-10367) Allow running the Streams demo app with a config file

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-10367:
--

 Summary: Allow running the Streams demo app with a config file
 Key: KAFKA-10367
 URL: https://issues.apache.org/jira/browse/KAFKA-10367
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Mickael Maison
Assignee: Mickael Maison


In the quickstart, we recommend users to play with the Kafka Streams 
WordCountDemo app.

This sample uses hardcoded configuration so it can only be run against a local 
cluster. It would be nice to be allowed to provide a configuration file, like 
for all the tools to allow targetting any Kafka environment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [kafka-site] mimaison merged pull request #278: Remove console.log()

[GitBox]

mimaison merged pull request #278:
URL: https://github.com/apache/kafka-site/pull/278


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and private key

[Rajini Sivaram]

Hi Andrew, Ron & Manikumar,

Thanks for reviewing the KIP!

If there are no concerns, I will start vote later today.


Regards,

Rajini


On Thu, Aug 6, 2020 at 5:55 AM Manikumar  wrote:

> Thanks for the KIP. LGTM.
>
> On Wed, Aug 5, 2020 at 3:43 AM Ron Dagostino  wrote:
>
> > Yes, Rajinit, it looks good -- an excellently-written KIP.  Thanks!
> >
> > Ron
> >
> > On Mon, Aug 3, 2020 at 9:17 AM Andrew Otto  wrote:
> >
> > > Yes please!  I had to build custom tooling to automate generation and
> > > distribution of java truststore and keystore files at Wikimedia for
> Kafka
> > > brokers and clients.  PEM is much more standard outside of the Java
> world
> > > and it would be simpler to not have to maintain custom tooling for
> > > keystores.
> > >
> > > - Andrew Otto
> > >
> > > On Mon, Aug 3, 2020 at 7:29 AM Rajini Sivaram  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have submitted KIP-651 to support PEM format for SSL key and trust
> > > > stores:
> > > >
> > > >-
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> > > >
> > > > This enables better integration with other Kafka features like
> dynamic
> > > > config update and password protection.
> > > >
> > > > Feedback and suggestions are welcome.
> > > >
> > > > Thank you...
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
>

[GitHub] [kafka-site] ijuma commented on pull request #281: Remove mention of Twitter as a way to get added to "Powered By"

[GitBox]

ijuma commented on pull request #281:
URL: https://github.com/apache/kafka-site/pull/281#issuecomment-669756104


   @guozhangwang I updated the PR to mention pull request first.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is back to normal : kafka-trunk-jdk8 #4766

[Apache Jenkins Server]

See