Jenkins build is back to normal : kafka-trunk-jdk11 #1635

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-10267) [Documentation] | Correction in kafka-console-producer command

[Hemant Girase (Jira)]

Hemant Girase created KAFKA-10267:
-

 Summary: [Documentation] | Correction in kafka-console-producer 
command
 Key: KAFKA-10267
 URL: https://issues.apache.org/jira/browse/KAFKA-10267
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Reporter: Hemant Girase


Hi Team,

[https://kafka.apache.org/documentation/]

In the below command, it should be "--broker-list" instead of 
"--bootstrap-server".



> bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic 
> my-replicated-topic

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk8 #4708

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[Yuriy Badalyantc]

Ok, I mentioned adding missing serdes in the Proposed Change paragraph.

About VoidSerde. I didn't add it intentionally. The semantic of the Unit
(scala's void) type is not clear in terms of the data. If kafka topic
contains messages of type Unit, what does it actually means? That there is
always null? Well, for that we have a Null type. That it's an empty byte
array? For that, we have an Array[Byte]. Empty string? No, it's a String.
So, I decided to not include Unit serde in the built-in Serdes. And if a
user will want to use the Unit type he can implement its own serde.

-Yuriy

On Fri, Jul 10, 2020 at 11:21 PM Matthias J. Sax  wrote:

> Thanks Yuriy!
>
> What about `VoidSerde` ? It's not listed.
>
> It might also be nice to add a short sentence and state that in addition
> to fixing the name collisions, the KIP will also close the gap of
> out-of-the-box serdes and add missing Serdes that are offered in Java to
> Scala.
>
>
> -Matthias
>
> On 7/10/20 7:51 AM, Yuriy Badalyantc wrote:
> > Oh, ok. I have done that. Just didn't know that it was necessary.
> >
> > -Yuriy
> >
> > On Fri, Jul 10, 2020 at 9:30 PM John Roesler 
> wrote:
> >
> >> Ah, thanks Yuriy,
> >>
> >> Sorry if this wasn't clear, but _all_ public API changes have to
> >> be explicitly included in the KIP. Can you just enumerate all
> >> the contents of the new API?
> >>
> >> Thanks,
> >> John
> >>
> >> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
> >>> Hi, Matthias,
> >>>
> >>> It's not directly mentioned in the KIP, but I added all missing Java
> >>> serdes. I mentioned it in the pull request description:
> >>> https://github.com/apache/kafka/pull/8955
> >>>
> >>> And also, this KIP originally was based on a pull request where I added
> >>> missing java serdes :) https://github.com/apache/kafka/pull/8049
> >>>
> >>> -Yuriy
> >>>
> >>> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax 
> >> wrote:
> >>>
>  Yuriy,
> 
>  thanks for the KIP update. I have one follow up thought: I checked
> what
>  default Serdes we offer in the Java class
> 
>   `org.apache.kafka.common.serialization.Serdes`
> 
>  and I think it would be good if we could close the gap between the
> Java
>  and Scala code and add the missing Java Serdes in Scala, too.
> 
>  It seems we are missing `Short` (Java and Scala), `Void`, `UUID`, and
>  `ByterBuffer`.
> 
>  Can we add those in addition?
> 
> 
>  -Matthias
> 
>  On 7/8/20 6:45 AM, John Roesler wrote:
> > Hi Yuriy,
> >
> > Once it seems like there’s general agreement in the discussion, you
> >> can
>  start a voting thread. You can find examples on the mailing list of
> >> what to
>  say in the first message. It’s basically just a message with the
> >> subject
>  line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating that
>  you’d like to start the vote. It’s nice to link to the kip document
> >> again.
> >
> > The rules for the vote are at the top of the “Kafka Improvement
> >> Process”
>  page, but you basically need 3 binding +1 votes and no binding -1
> >> votes.
>  You also need to wait at least three days from when you start the vote
>  before you can declare it accepted. There’s no upper time limit.
> >
> > If you’re unsure of who has a binding vote, it’s just the people
> >> listed
>  on the Apache Kafka Committers page.
> >
> > If people are slow to vote, feel free to keep bumping the thread,
> >> just
>  like with the discussion.
> >
> > Thanks again for getting involved!
> > -John
> >
> > On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote:
> >> So, what's next? It's my first KIP and I'm not familiar with all
>  processes.
> >>
> >> -Yuriy
> >>
> >> On Mon, Jul 6, 2020 at 1:32 AM John Roesler 
>  wrote:
> >>
> >>> Hi Yuriy,
> >>>
> >>> Thanks for the update! It looks good to me.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote:
>  Hi John.
> 
>  I updated the KIP. An old proposed implementation is now in the
>  rejected
>  alternatives.
> 
>  - Yuriy
> 
>  On Sun, Jul 5, 2020 at 12:03 AM John Roesler  >>>
> >>> wrote:
> 
> > Hi Yuriy,
> >
> > I agree, we can keep them separate. I just wanted to make you
> >> aware
>  of
> >>> it.
> >
> > Thanks for the PR, it looks the way I expected.
> >
> > I just read over the KIP document again. I think it needs to be
> >>> updated to
> > the current proposal, and then we’ll be able to start the vote.
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote:
> >> Hi everybody!
> >>
> >> Looks like a discussion about KIP-513 could 

Build failed in Jenkins: kafka-2.6-jdk8 #77

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-10263: Do not assign standby for revoking stateless tasks 
(#9005)


--
[...truncated 3.15 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord 

Build failed in Jenkins: kafka-trunk-jdk14 #284

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10263: Do not assign standby for revoking stateless tasks (#9005)


--
[...truncated 3.19 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

[Guozhang Wang]

Hello Colin,

Thanks for the nice written KIP. A few meta comments:

1) We need to talk a bit about broker failure detection: is that piggy
backed with fencing? i.e. should the controller immediately migrate leader
partitions from the fenced brokers? On one side, when a broker is fenced it
cannot take any client requests including produce / fetch, so it seems we'd
need to let other brokers to take those hosted partitions asap; on the
other side, if the fenced broker is just due to stale metadata which is
soon to be caught up then maybe we only want it to stop handling metadata
request from clients, and if we treat it as a failed broker then we may
have back-and-forth partition migrations.

2) I'm a bit confused about ControllerHeartbeat v.s. BrokerHeartbeat RPC:
in the Broker Registration and State Management section it is stated that
"each broker sends a ControllerHeartbeat request to the active controller
every few seconds" but later in RPC it seems that brokers are
sending BrokerHeartbeat instead. If that's the case, what
are ControllerHeartbeat used for?


Guozhang


On Fri, Jul 10, 2020 at 4:12 AM Unmesh Joshi  wrote:

> I was thinking that we might need something like multi-operation
>  record in zookeeper
> to atomically create topic and partition records when this multi record is
> committed.  This way metadata will have both the TopicRecord and
> PartitionRecord together always, and in no situation we can have
> TopicRecord without PartitionRecord. Not sure if there are other situations
> where multi-operation is needed.
> 
>
> Thanks,
> Unmesh
>
> On Fri, Jul 10, 2020 at 11:32 AM Colin McCabe  wrote:
>
> > Hi Unmesh,
> >
> > Yes, once the last stable offset advanced, we would consider the topic
> > creation to be done, and then we could return success to the client.
> >
> > best,
> > Colin
> >
> > On Thu, Jul 9, 2020, at 19:44, Unmesh Joshi wrote:
> > > It still needs HighWaterMark / LastStableOffset to be advanced by two
> > > records? Something like following?
> > >
> > >
> > >||
> > > <--||   HighWaterMark
> > >Response|PartitionRecord |
> > >||
> > >-|
> > >| TopicRecord|  -
> > >||
> > > --->   --   Previous HighWaterMark
> > >CreateTopic ||
> > >||
> > >||
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Jul 10, 2020 at 1:30 AM Colin McCabe 
> wrote:
> > >
> > > > On Thu, Jul 9, 2020, at 04:37, Unmesh Joshi wrote:
> > > > > I see that, when a new topic is created, two metadata records, a
> > > > > TopicRecord (just the name and id of the topic) and a
> PartitionRecord
> > > > (more
> > > > > like LeaderAndIsr, with leader id and replica ids for the
> partition)
> > are
> > > > > created.
> > > > > While creating the topic, log entries for both the records need to
> be
> > > > > committed in RAFT core. Will it need something like a
> > > > MultiOperationRecord
> > > > > in zookeeper. Then, we can have a single log entry with both the
> > records,
> > > > > and  the create topic request can be fulfilled atomically when both
> > the
> > > > > records are committed?
> > > > >
> > > >
> > > > Hi Unmesh,
> > > >
> > > > Since the active controller is the only node writing to the log,
> there
> > is
> > > > no need for any kind of synchronization or access control at the log
> > level.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > > Unmesh
> > > > >
> > > > > On Wed, Jul 8, 2020 at 6:57 AM Ron Dagostino 
> > wrote:
> > > > >
> > > > > > HI Colin.  Thanks for the KIP.  Here is some feedback and various
> > > > > > questions.
> > > > > >
> > > > > > "*Controller processes will listen on a separate port from
> brokers.
> > > > This
> > > > > > will be true even when the broker and controller are co-located
> in
> > the
> > > > same
> > > > > > JVM*". I assume it is possible that the port numbers could be the
> > same
> > > > when
> > > > > > using separate JVMs (i.e. broker uses port 9192 and controller
> also
> > > > uses
> > > > > > port 9192).  I think it would be clearer to state this along
> these
> > > > > > lines: "Controller
> > > > > > nodes will listen on a port, and the controller port must differ
> > from
> > > > any
> > > > > > port that a broker in the same JVM is listening on.  In other
> > words, a
> > > > > > controller and a broker node, when in the same JVM, do not share
> > ports"
> > > > > >
> > > > > > I think the sentence "*In the realm of ACLs, this translates to
> > > > controllers
> > > > > > requiring 

Re: [DISCUSS] Apache Kafka 2.6.0 release

[Matthias J. Sax]

Randall,

we found another blocker: https://issues.apache.org/jira/browse/KAFKA-10262

Luckily, we have already a PR for it.


-Matthias


On 7/8/20 3:05 PM, Sophie Blee-Goldman wrote:
> Hey Randall,
> 
> We just discovered another regression in 2.6:
> https://issues.apache.org/jira/browse/KAFKA-10249
> 
> The fix is extremely straightforward -- only about two lines of actual
> code -- and low risk. It is a new regression introduced in 2.6 and affects
> all Streams apps with any suppression or other in-memory state.
> 
> The PR is already ready here: https://github.com/apache/kafka/pull/8996
> 
> Best,
> Sophie
> 
> On Wed, Jul 8, 2020 at 10:59 AM John Roesler  wrote:
> 
>> Hi Randall,
>>
>> While developing system tests, I've just unearthed a new 2.6 regression:
>> https://issues.apache.org/jira/browse/KAFKA-10247
>>
>> I've got a PR in progress. Hoping to finish it up today:
>> https://github.com/apache/kafka/pull/8994
>>
>> Sorry for the trouble,
>> -John
>>
>> On Mon, Jun 29, 2020, at 09:29, Randall Hauch wrote:
>>> Thanks for raising this, David. I agree it makes sense to include this
>> fix
>>> in 2.6, so I've adjusted the "Fix Version(s)" field to include '2.6.0'.
>>>
>>> Best regards,
>>>
>>> Randall
>>>
>>> On Mon, Jun 29, 2020 at 8:25 AM David Jacot  wrote:
>>>
 Hi Randall,

 We have discovered an annoying issue that we introduced in 2.5:

 Describing topics with the command line tool fails if the user does not
 have the
 privileges to access the ListPartitionReassignments API. I believe that
 this is the
 case for most non-admin users.

 I propose to include the fix in 2.6. The fix is trivial so low risk.
>> What
 do you think?

 JIRA: https://issues.apache.org/jira/browse/KAFKA-10212
 PR: https://github.com/apache/kafka/pull/8947

 Best,
 David

 On Sat, Jun 27, 2020 at 4:39 AM John Roesler 
>> wrote:

> Hi Randall,
>
> I neglected to notify this thread when I merged the fix for
> https://issues.apache.org/jira/browse/KAFKA-10185
> on June 19th. I'm sorry about that oversight. It is marked with
> a fix version of 2.6.0.
>
> On a side node, I have a fix for KAFKA-10173, which I'm merging
> and backporting right now.
>
> Thanks for managing the release,
> -John
>
> On Thu, Jun 25, 2020, at 10:23, Randall Hauch wrote:
>> Thanks for the update, folks!
>>
>> Based upon Jira [1], we currently have 4 issues that are considered
>> blockers for the 2.6.0 release and production of RCs:
>>
>>- https://issues.apache.org/jira/browse/KAFKA-10134 - High CPU
 issue
>>during rebalance in Kafka consumer after upgrading to 2.5
 (unassigned)
>>- https://issues.apache.org/jira/browse/KAFKA-10143 - Can no
>> longer
>>change replication throttle with reassignment tool (Jason G)
>>- https://issues.apache.org/jira/browse/KAFKA-10166 - Excessive
>>TaskCorruptedException seen in testing (Sophie, Bruno)
>>- https://issues.apache.org/jira/browse/KAFKA-10173
>>- BufferUnderflowException during Kafka Streams Upgrade (John R)
>>
>> and one critical issue that may be a regression that at this time
>> will
> not
>> block production of RCs:
>>
>>- https://issues.apache.org/jira/browse/KAFKA-10017 - Flaky
>> Test
>>EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta
> (Matthias)
>>
>> and one build/release issue we'd like to fix if possible but will
>> not
> block
>> RCs or the release:
>>
>>- https://issues.apache.org/jira/browse/KAFKA-9381
>>- kafka-streams-scala: Javadocs + Scaladocs not published on
>> maven
> central
>>(me)
>>
>> I'm working with the assignees and reporters of these issues (via
> comments
>> on the issues) to identify an ETA and to track progress. Anyone is
> welcome
>> to chime in on those issues.
>>
>> At this time, no other changes (other than PRs that only
>> fix/improve
> tests)
>> should be merged to the `2.6` branch. If you think you've
>> identified a
> new
>> blocker issue or believe another existing issue should be treated
>> as a
>> blocker for 2.6.0, please mark the issue's `fix version` as `2.6.0`
 _and_
>> respond to this thread with details, and I will work with you to
> determine
>> whether it is indeed a blocker.
>>
>> As always, let me know here if you have any questions/concerns.
>>
>> Best regards,
>>
>> Randall
>>
>> [1]
>> https://issues.apache.org/jira/projects/KAFKA/versions/12346918
>>
>> On Thu, Jun 25, 2020 at 8:27 AM Mario Molina 
 wrote:
>>
>>> Hi Randal,
>>>
>>> Ticket https://issues.apache.org/jira/browse/KAFKA-9018 is not a
> blocker
>>> so
>>> it can be moved to the 2.7.0 version.
>>>
>>> Mario
>>>

Re: [VOTE] KIP-623: Add "internal-topics" option to streams application reset tool

[Boyang Chen]

Thanks for the update!

On Fri, Jul 3, 2020 at 3:18 PM Joel Wee  wrote:

> Thanks all for voting!
>
> I am closing the vote as accepted with three binding +1 votes (Boyang,
> Guozhang, John).
>
> Thanks John for the suggestion. I think that makes sense. I have updated
> the KIP to say that only topics flagged as internal by the tool can be
> deleted, and have also rephrased the option description to make this
> clearer:
>
> > --internal-topics   Comma-separated list of internal
> topics
> > to delete. Must be a subset of
> the
> > internal topics marked for
> deletion by
> > the default behaviour. To view
> these
> > topics, do a dry-run without
> this
> > option.
>
> Hmm, could you explain why we couldn't run with both --internal-topics and
--dry-run? To me we could either display the exact set of internal topics
specified to be deleted, or reject the same way as we are doing an actual
reset.

>
> Thanks Guozhang for the suggestion. The updated option description should
> address your first point. By the “topology description” script are you
> referring to bin/kafka-topics.sh? It currently has the option to display
> all topics (including internal topics). Were you thinking about adding
> something to view just internal topics?
>
> Thanks Bruno for the suggestion. I will close this vote for now, and we
> can continue the discussion on another thread. (P.S. apologies for
> misspelling your name previously)
>
> - Joel
>
> > On 1 Jul 2020, at 3:04 AM, Boyang Chen 
> wrote:
> >
> > Hey Bruno,
> >
> > I agree adding a prompt would be a nice precaution, but it is not
> backward
> > compatible as you suggested and could make the automation harder to
> > achieve.
> >
> > If you want, we may consider starting a separate ticket to discuss
> whether
> > adding a prompt to let users be aware of the topics that are about to
> > delete. However, this is also inverting the assumptions we made about
> > `--dry-run` mode, which would become useless to me once we are adding a
> > prompt asking users whether they want to remove these topics completely,
> or
> > do nothing at all.
> >
> > Back to KIP-623, I think this discussion could be held in orthogonal,
> which
> > applies to more general considerations of reducing human errors, etc.
> >
> > Boyang
> >
> > On Tue, Jun 30, 2020 at 12:55 AM Bruno Cadonna 
> wrote:
> >
> >> Hi,
> >>
> >> I have already brought this up in the discussion thread.
> >>
> >> Should we not run a dry-run in any case to avoid inadvertently
> >> deleting topics of other applications?
> >>
> >> I know it is a backward incompatible change if users use it in
> >> scripts, but I think it is still worth discussing it. I would to hear
> >> what committers think about it.
> >>
> >> Best,
> >> Bruno
> >>
> >> On Tue, Jun 30, 2020 at 5:33 AM Boyang Chen  >
> >> wrote:
> >>>
> >>> Thanks John for the great suggestion. +1 for enforcing the prefix check
> >> for
> >>> the `--internal-topics` list.
> >>>
> >>> On Mon, Jun 29, 2020 at 5:11 PM John Roesler 
> >> wrote:
> >>>
>  Oh, I meant to say, if that’s the case, then I’m +1 (binding) also :)
> 
>  Thanks again,
>  John
> 
>  On Mon, Jun 29, 2020, at 19:09, John Roesler wrote:
> > Thanks for the KIP, Joel!
> >
> > It seems like a good pattern to document would be to first run with
> > —dry-run and without —internal-topics to list all potential topics,
> >> and
> > then to use —internal-topics if needed to limit the internal topics
> >> to
> > delete.
> >
> > Just to make sure, would we have a sanity check to forbid including
> > arbitrary topics? I.e., it seems like —internal-topics should require
> > all the topics to be prefixed with the app id. Is that right?
> >
> > Thanks,
> > John
> >
> > On Mon, Jun 29, 2020, at 18:25, Guozhang Wang wrote:
> >> Thanks Joel, the KIP lgtm.
> >>
> >> A minor suggestion is to explain where users can get the list of
>  internal
> >> topics of a given application, and maybe also add it as part of the
>  helper
> >> scripts, for example via topology description.
> >>
> >> Overall, I'm +1 as well (binding).
> >>
> >> Guozhang
> >>
> >>
> >> On Sat, Jun 27, 2020 at 4:33 AM Joel Wee 
> >> wrote:
> >>
> >>> Thanks Boyang, I think what you’ve said makes sense. I’ve made
> >> the
> >>> motivation clearer now:
> >>>
> >>>
> >>> Users may want to specify which internal topics should be
> >> deleted. At
> >>> present, the streams reset tool deletes all topics that start
> >> with "<
> >>> application.id>-" and there are no
> >> options to
> >>> control it.
> >>>
> >>> The `--internal-topics` option is especially useful when there
> 

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

[Boyang Chen]

Thanks for the update. One nit for the KIP is to format the signature
indentation for all code templates, like:

public static Properties addSerializerToConfig(Properties properties,

Serializer keySerializer,

Serializer valueSerializer)

Other than that, +1 (binding) from me.

On Fri, Jul 10, 2020 at 9:37 AM Chia-Ping Tsai  wrote:

> > I don't think my question gets answered,
>
> Sorry for incorrect response :(
>
> > why would deprecating the map
> > based `addSerializerToConfig` break user's recompilation? If you worry
> > about warnings, we could refactor out the content and create a
> > package-private `attachSerializersToConfig` or something similar.
>
> you are right. We can add more deprecation for this KIP. Both
> ProducerConfig.addSerializerToConfig(Map ...) and
> ConsumerConfig.addDeserializerToConfig(Map ...) can be
> deprecated and we add package-private variety of them.
>
> I will update KIP !
>
> On 2020/07/07 16:12:49, Boyang Chen  wrote:
> > Ok, after a second thought, keeping a function which still has production
> > reference is ok. We probably should not make it public in the first
> place,
> > but this is not high priority either.
> >
> > On Tue, Jul 7, 2020 at 9:03 AM Chia-Ping Tsai 
> wrote:
> >
> > > > do we just suggest they no longer have any production use case?
> > >
> > > yep
> > >
> > > > KafkaProducer internal only. Do we also want to deprecate this public
> > > API as well?
> > >
> > > We have to make sure users' code can keep working beyond recompilation
> > > when migrating to "next" release. Hence, deprecation cycle is
> necessary.
> > >
> > > I don't think my question gets answered, why would deprecating the map
> > based `addSerializerToConfig` break user's recompilation? If you worry
> > about warnings, we could refactor out the content and create a
> > package-private `attachSerializersToConfig` or something similar.
> >
> > On 2020/07/07 06:52:25, Boyang Chen  wrote:
> > > > Thanks for the KIP. One question I have is that when we refer to the
> two
> > > > methods as useless, do we just suggest they no longer have any
> production
> > > > use case? If this is the case,
> Producer#addSerializerToConfig(Map > > > Object> configs, keySerializer, valueSerializer) is only used in
> > > > KafkaProducer internal only. Do we also want to deprecate this
> public API
> > > > as well?
> > > >
> > > > Boyang
> > > >
> > > >
> > > > On Mon, Jul 6, 2020 at 11:36 PM Manikumar  >
> > > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > On Wed, Jun 10, 2020 at 11:43 PM Matthias J. Sax  >
> > > wrote:
> > > > >
> > > > > > Yes, it does.
> > > > > >
> > > > > > I guess many people are busy wrapping up 2.6 release. Today is
> code
> > > > > freeze.
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > >
> > > > > > On 6/10/20 12:11 AM, Chia-Ping Tsai wrote:
> > > > > > > hi Matthias,
> > > > > > >
> > > > > > > Does this straightforward KIP still need 3 votes?
> > > > > > >
> > > > > > > On 2020/06/05 21:27:52, "Matthias J. Sax" 
> > > wrote:
> > > > > > >> +1 (binding)
> > > > > > >>
> > > > > > >> Thanks for the KIP!
> > > > > > >>
> > > > > > >>
> > > > > > >> -Matthias
> > > > > > >>
> > > > > > >> On 6/4/20 11:25 PM, Chia-Ping Tsai wrote:
> > > > > > >>> hi All,
> > > > > > >>>
> > > > > > >>> I would like to start the vote on KIP-620:
> > > > > > >>>
> > > > > > >>>
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=155749118
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> Chia-Ping
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-10266) Fix connector configs in docs to mention the correct default value inherited from worker configs

[Konstantine Karantasis (Jira)]

Konstantine Karantasis created KAFKA-10266:
--

 Summary: Fix connector configs in docs to mention the correct 
default value inherited from worker configs
 Key: KAFKA-10266
 URL: https://issues.apache.org/jira/browse/KAFKA-10266
 Project: Kafka
  Issue Type: Bug
Reporter: Konstantine Karantasis


 

Example: 
[https://kafka.apache.org/documentation/#header.converter]

has the correct default when it is mentioned as a worker property. 
But under the section of source connector configs, it's default value is said 
to be `null`. 

Though that is correct in terms of implementation, it's confusing for users. 
We should surface the correct defaults for configs that inherit (or otherwise 
override) worker configs. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

[Badai Aqrandista]

Hi all

The vote for KIP-431 has passed with 3 binding and 1 non-binding +1s, and
no objections.

Thanks everyone for reviews and feedback.

Regards,
Badai

On Fri, Jul 10, 2020 at 5:22 PM Badai Aqrandista  wrote:
>
> Hu
>
> I would make it more readable and move the key next to the value like this:
>
> CreateTime:1592475472398|Partition:3|Offset:0|Headers:h1=v1,h2=v2|key1|value1
>
> What do you think?
>
> In future KIP, we'll add some formatting like David Jacot suggested.
>
> Regards
> Badai
>
> On Fri, Jul 10, 2020 at 2:57 PM huxi_2b  wrote:
> >
> > Badai,
> >
> > Replying in this thread. Two possible ways come up into my head:
> >
> > 1. CreateTime:1592475472398|key1|3|<0>|h1=v1,h2=v2|value1
> > 2. CreateTime:1592475472398|key1|3|offset=0|h1=v1,h2=v2|value1
> >
> > I prefer the option #1 which could be accomplished by a complementary usage 
> > description. What do you think?
> >
> > On 2020/06/21 13:39:36, Badai Aqrandista  wrote:
> > > Excellent.>
> > >
> > > Would like to hear more feedback from others.>
> > >
> > > On Sat, Jun 20, 2020 at 1:27 AM David Jacot  wrote:>
> > > >>
> > > > Hi Badai,>
> > > >>
> > > > Thanks for your reply.>
> > > >>
> > > > 2. Yes, that makes sense.>
> > > >>
> > > > Best,>
> > > > David>
> > > >>
> > > > On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista  
> > > > wrote:>
> > > >>
> > > > > David>
> > > > >>
> > > > > Thank you for replying>
> > > > >>
> > > > > 1. It seems that `print.partition` is already implemented. Do you 
> > > > > confirm?>
> > > > > BADAI: Yes, you are correct. I have removed it from the KIP.>
> > > > >>
> > > > > 2. Will `null.literal` be only used when the value of the message>
> > > > > is NULL or for any fields? Also, it seems that we print out "null">
> > > > > today when the key or the value is empty. Shall we use "null" as>
> > > > > a default instead of ""?>
> > > > > BADAI: For any fields. Do you think this is useful?>
> > > > >>
> > > > > 3. Could we add a small example of the output in the KIP?>
> > > > > BADAI: Yes, I have updated the KIP to add a couple of example.>
> > > > >>
> > > > > 4. When there are no headers, are we going to print something>
> > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP>
> > > > > where there is no timestamp.>
> > > > > BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS.>
> > > > >>
> > > > > Thanks>
> > > > > Badai>
> > > > >>
> > > > >>
> > > > > On Thu, Jun 18, 2020 at 7:25 PM David Jacot  
> > > > > wrote:>
> > > > > >>
> > > > > > Hi Badai,>
> > > > > >>
> > > > > > Thanks for resuming this. I have few small comments:>
> > > > > >>
> > > > > > 1. It seems that `print.partition` is already implemented. Do you>
> > > > > confirm?>
> > > > > >>
> > > > > > 2. Will `null.literal` be only used when the value of the message>
> > > > > > is NULL or for any fields? Also, it seems that we print out "null">
> > > > > > today when the key or the value is empty. Shall we use "null" as>
> > > > > > a default instead of ""?>
> > > > > >>
> > > > > > 3. Could we add a small example of the output in the KIP?>
> > > > > >>
> > > > > > 4. When there are no headers, are we going to print something>
> > > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP>
> > > > > > where there is no timestamp.>
> > > > > >>
> > > > > > Best,>
> > > > > > David>
> > > > > >>
> > > > > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista 
> > > > > > >
> > > > > wrote:>
> > > > > >>
> > > > > > > Hi all,>
> > > > > > >>
> > > > > > > I have contacted Mateusz separately and he is ok for me to take 
> > > > > > > over>
> > > > > > > KIP-431:>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter>
> > > > > > >>
> > > > > > > I have updated it a bit. Can anyone give a quick look at it again 
> > > > > > > and>
> > > > > > > give me some feedback?>
> > > > > > >>
> > > > > > > This feature will be very helpful for people supporting Kafka in>
> > > > > > > operations.>
> > > > > > >>
> > > > > > > If it is ready for a vote, please let me know.>
> > > > > > >>
> > > > > > > Thanks>
> > > > > > > Badai>
> > > > > > >>
> > > > > > > On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista 
> > > > > > > >
> > > > > > > wrote:>
> > > > > > > >>
> > > > > > > > Mateusz>
> > > > > > > >>
> > > > > > > > This KIP would be very useful for debugging. But the last 
> > > > > > > > discussion>
> > > > > > > > is in Feb 2019.>
> > > > > > > >>
> > > > > > > > Are you ok if I take over this KIP?>
> > > > > > > >>
> > > > > > > > -->
> > > > > > > > Thanks,>
> > > > > > > > Badai>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > > -->
> > > > > > > Thanks,>
> > > > > > > Badai>
> > > > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > > -->
> > > > > Thanks,>
> > > > > Badai>
> > > > >>
> > >
> > >
> > >
> > > -- >
> > > Thanks,>
> > > Badai>
> > >
>
>
>

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Colin McCabe]

Hi all,

Thanks for the KIP.

I took a look and one thing that stood out to me is that the more metadata we 
have, the more storage we will need on local disk for the rocksDB database.  
This seems like it contradicts some of the goals of the project.  Ideally the 
space we need on local disk should be related only to the size of the hot set, 
not the size of the cold set.  It also seems like it could lead to extremely 
long rocksdb rebuild times if we somehow lose a broker's local storage and have 
to rebuild it.

Instead, I think it would be more reasonable to store cold metadata in the 
"remote" storage (HDFS, s3, etc.).  Not only does this free up space on the 
local and avoid long rebuild times, but it also gives us more control over the 
management of our cache.  With rocksDB we are delegating cache management to an 
external library that doesn't really understand our use-case.

To give a concrete example of how this is bad, imagine that we have 10 worker 
threads and we get  10 requests for something that requires us to fetch cold 
tiered storage metadata.  Now every worker thread is blocked inside rocksDB and 
the broker can do nothing until it finishes fetching from disk.  When accessing 
a remote service like HDFS or S3, in contrast, we would be able to check if the 
data was in our local cache first.  If it wasn't, we could put the request in a 
purgatory and activate a background thread to fetch the needed data, and then 
release the worker thread to be used by some other request.  Having control of 
our own caching strategy increases observability, maintainability, and 
performance.

I can anticipate a possible counter-argument here: the size of the metadata 
should be small and usually fully resident in memory anyway.  While this is 
true today, I don't think it will always be true.  The current low limit of a 
few thousand partitions is not competitive in the long term and needs to be 
lifted.  We'd like to get to at least a million partitions with KIP-500, and 
much more later.  Also, when you give people the ability to have unlimited 
retention, they will want to make use of it.  That means lots of historical log 
segments to track.  This scenario is by no means hypothetical.  Even with the 
current software, it's easy to think of cases where someone misconfigured the 
log segment roll settings and overwhelmed the system with segments.  So 
overall, I like to understand why we want to store metadata on local disk 
rather than remote, and what the options are for the future.

best,
Colin


On Thu, Jul 9, 2020, at 09:55, Harsha Chintalapani wrote:
> Hi Jun,
>   Thanks for the replies and feedback on design and giving input.
> We are coming close to finish the implementation.
> We also did several perf tests as well at our peak production loads and
> with tiered storage we didn't see any degradation on write throughputs and
> latencies.
> Ying already added some of the perf tests results in the KIP itself.
>  It will be great if we can get design and code reviews from you
> and others in the community as we make progress.
> Thanks,
> Harsha
> 
> On Tue, Jul 7, 2020 at 10:34 AM Jun Rao  wrote:
> 
> > Hi, Ying,
> >
> > Thanks for the update. It's good to see the progress on this. Please let
> > us know when you are done updating the KIP wiki.
> >
> > Jun
> >
> > On Tue, Jul 7, 2020 at 10:13 AM Ying Zheng  wrote:
> >
> >> Hi Jun,
> >>
> >> Satish and I have added more design details in the KIP, including how to
> >> keep consistency between replicas (especially when there is leadership
> >> changes / log truncations) and new metrics. We also made some other minor
> >> changes in the doc. We will finish the KIP changes in the next couple of
> >> days. We will let you know when we are done. Most of the changes are
> >> already updated to the wiki KIP. You can take a look. But it's not the
> >> final version yet.
> >>
> >> As for the implementation, the code is mostly done and we already had some
> >> feature tests / system tests. I have added the performance test results in
> >> the KIP. However the recent design changes (e.g. leader epoch info
> >> management / log truncation / some of the new metrics) have not been
> >> implemented yet. It will take about 2 weeks for us to implement after you
> >> review and agree with those design changes.
> >>
> >>
> >>
> >> On Tue, Jul 7, 2020 at 9:23 AM Jun Rao  wrote:
> >>
> >> > Hi, Satish, Harsha,
> >> >
> >> > Any new updates on the KIP? This feature is one of the most important
> >> and
> >> > most requested features in Apache Kafka right now. It would be helpful
> >> if
> >> > we can make sustained progress on this. Could you share how far along is
> >> > the design/implementation right now? Is there anything that other people
> >> > can help to get it across the line?
> >> >
> >> > As for "transactional support" and "follower requests/replication", no
> >> > further comments from me as long as the producer state and leader epoch
> >> 

Re: [DISCUSS] KIP-450: Sliding Windows

[Boyang Chen]

Thanks Leah and Sophie for the KIP.

1. I'm a bit surprised that we don't have an advance time. Could we
elaborate how the storage layer is structured?

2. IIUC, there will be extra cost in terms of fetching aggregation results,
since we couldn't pre-aggregate until the user asks for it. Would be good
to also discuss it.

3. We haven't discussed the possibility of supporting sliding windows
inherently. For a user who actually uses a hopping window, Streams could
detect such an inefficiency doing a window_size/advance_time ratio to reach
a conclusion on whether the write amplification is too high compared with
some configured threshold. The benefit of doing so is that existing Streams
users don't need to change their code, learn a new API, but only to upgrade
Streams library to get benefits for their inefficient hopping window
implementation. There might be some compatibility issues for sure, but
worth listing them out for trade-off.

Boyang

On Fri, Jul 10, 2020 at 12:40 PM Leah Thomas  wrote:

> Hey Matthias,
>
> Thanks for pointing that out. I added the following to the Propose Changes
> section of the KIP:
>
> "Records that come out of order will be processed the same way as in-order
> records, as long as they fall within the grace period. Any new windows
> created by the late record will still be created, and the existing windows
> that are changed by the late record will be updated. Any record that falls
> outside of the grace period (either user defined or default) will be
> discarded. "
>
> All the best,
> Leah
>
> On Thu, Jul 9, 2020 at 9:47 PM Matthias J. Sax  wrote:
>
> > Leah,
> >
> > thanks a lot for the KIP. Very well written.
> >
> > The KIP does not talk about the handling of out-of-order data though.
> > How do you propose to address this?
> >
> >
> > -Matthias
> >
> > On 7/8/20 5:33 PM, Leah Thomas wrote:
> > > Hi all,
> > > I'd like to kick-off the discussion for KIP-450, adding sliding window
> > > aggregation support to Kafka Streams.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL
> > >
> > > Let me know what you think,
> > > Leah
> > >
> >
> >
>

Build failed in Jenkins: kafka-2.6-jdk8 #76

[Apache Jenkins Server]

See 


Changes:

[vvcephei] KAFKA-10191 fix flaky StreamsOptimizedTest (#8913)


--
[...truncated 3.15 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

Re: [DISCUSS] KIP-450: Sliding Windows

[Leah Thomas]

Hey Matthias,

Thanks for pointing that out. I added the following to the Propose Changes
section of the KIP:

"Records that come out of order will be processed the same way as in-order
records, as long as they fall within the grace period. Any new windows
created by the late record will still be created, and the existing windows
that are changed by the late record will be updated. Any record that falls
outside of the grace period (either user defined or default) will be
discarded. "

All the best,
Leah

On Thu, Jul 9, 2020 at 9:47 PM Matthias J. Sax  wrote:

> Leah,
>
> thanks a lot for the KIP. Very well written.
>
> The KIP does not talk about the handling of out-of-order data though.
> How do you propose to address this?
>
>
> -Matthias
>
> On 7/8/20 5:33 PM, Leah Thomas wrote:
> > Hi all,
> > I'd like to kick-off the discussion for KIP-450, adding sliding window
> > aggregation support to Kafka Streams.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL
> >
> > Let me know what you think,
> > Leah
> >
>
>

[GitHub] [kafka-site] vvcephei merged pull request #274: MINOR: Add vvcephei to KEYS

[GitBox]

vvcephei merged pull request #274:
URL: https://github.com/apache/kafka-site/pull/274


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] vvcephei opened a new pull request #274: MINOR: Add vvcephei to KEYS

[GitBox]

vvcephei opened a new pull request #274:
URL: https://github.com/apache/kafka-site/pull/274


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-450: Sliding Windows

[Sophie Blee-Goldman]

Thanks Leah!

This kind of assumes an implicit answer to Matthias's question, but I was
wondering
if we should take this opportunity to choose a better default value for the
grace period.
Note that the default of -1 in the TimeWindows class, for example,
ultimately gets
translated into a default value of 24 hours. Which is kind of a long time.

This has hit users of suppression especially hard, since it means you won't
see *any*
output for 24 hours. It's pretty frustrating when you're trying to unit
test your topology
and nothing happens just because you didn't explicitly override the default
grace period.

Unfortunately, we're stuck with the 24hr grace period for our existing
operators for
compatibility reasons. But since this is a new kind of aggregation, we have
the opportunity
to consider alternatives and try to improve on this pain point for users.

Of course, the obvious question now is: what would be a good grace period?
We've
discussed this a number of times before and as far as I know were never
came up with
a good answer. It might also be somewhat confusing for users if different
kinds of
windowed aggregations had different default grace periods, although that's
not a
great reason to keep doing something that's obviously causing problems.

On the other hand, someone recently brought up what I thought was a good
suggestion:
just make the grace period a required parameter. This seems to solve the
existing
problem while dodging the question of what a "good" universal default would
be.

WDYT?

Cheers,
Sophie

On Thu, Jul 9, 2020 at 9:47 PM Matthias J. Sax  wrote:

> Leah,
>
> thanks a lot for the KIP. Very well written.
>
> The KIP does not talk about the handling of out-of-order data though.
> How do you propose to address this?
>
>
> -Matthias
>
> On 7/8/20 5:33 PM, Leah Thomas wrote:
> > Hi all,
> > I'd like to kick-off the discussion for KIP-450, adding sliding window
> > aggregation support to Kafka Streams.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL
> >
> > Let me know what you think,
> > Leah
> >
>
>

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

[Boyang Chen]

Hey Colin, thanks for the KIP. One question I have about AlterScramUsers
RPC is whether we could consolidate the deletion list and alteration list,
since in response we only have a single list of results. The further
benefit is to reduce unintentional duplicate entries for both deletion and
alteration, which makes the broker side handling logic easier. Another
alternative is to add DeleteScramUsers RPC to align what we currently have
with other user provided data such as delegation tokens (create, change,
delete).

For my own education, the salt will be automatically generated by the admin
client when we send the SCRAM requests correct?

Best,
Boyang

On Fri, Jul 10, 2020 at 8:10 AM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Thanks for the KIP, Colin!
>
> Regards,
>
> Rajini
>
>
> On Thu, Jul 9, 2020 at 8:49 PM Colin McCabe  wrote:
>
> > Hi all,
> >
> > I'd like to call a vote for KIP-554: Add a broker-side SCRAM
> configuration
> > API.  The KIP is here: https://cwiki.apache.org/confluence/x/ihERCQ
> >
> > The previous discussion thread is here:
> >
> >
> https://lists.apache.org/thread.html/r69bdc65bdf58f5576944a551ff249d759073ecbf5daa441cff680ab0%40%3Cdev.kafka.apache.org%3E
> >
> > best,
> > Colin
> >
>

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

[Chia-Ping Tsai]

> I don't think my question gets answered, 

Sorry for incorrect response :(

> why would deprecating the map
> based `addSerializerToConfig` break user's recompilation? If you worry
> about warnings, we could refactor out the content and create a
> package-private `attachSerializersToConfig` or something similar.

you are right. We can add more deprecation for this KIP. Both 
ProducerConfig.addSerializerToConfig(Map ...) and 
ConsumerConfig.addDeserializerToConfig(Map ...) can be 
deprecated and we add package-private variety of them.

I will update KIP !

On 2020/07/07 16:12:49, Boyang Chen  wrote: 
> Ok, after a second thought, keeping a function which still has production
> reference is ok. We probably should not make it public in the first place,
> but this is not high priority either.
> 
> On Tue, Jul 7, 2020 at 9:03 AM Chia-Ping Tsai  wrote:
> 
> > > do we just suggest they no longer have any production use case?
> >
> > yep
> >
> > > KafkaProducer internal only. Do we also want to deprecate this public
> > API as well?
> >
> > We have to make sure users' code can keep working beyond recompilation
> > when migrating to "next" release. Hence, deprecation cycle is necessary.
> >
> > I don't think my question gets answered, why would deprecating the map
> based `addSerializerToConfig` break user's recompilation? If you worry
> about warnings, we could refactor out the content and create a
> package-private `attachSerializersToConfig` or something similar.
> 
> On 2020/07/07 06:52:25, Boyang Chen  wrote:
> > > Thanks for the KIP. One question I have is that when we refer to the two
> > > methods as useless, do we just suggest they no longer have any production
> > > use case? If this is the case, Producer#addSerializerToConfig(Map > > Object> configs, keySerializer, valueSerializer) is only used in
> > > KafkaProducer internal only. Do we also want to deprecate this public API
> > > as well?
> > >
> > > Boyang
> > >
> > >
> > > On Mon, Jul 6, 2020 at 11:36 PM Manikumar 
> > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > On Wed, Jun 10, 2020 at 11:43 PM Matthias J. Sax 
> > wrote:
> > > >
> > > > > Yes, it does.
> > > > >
> > > > > I guess many people are busy wrapping up 2.6 release. Today is code
> > > > freeze.
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > >
> > > > > On 6/10/20 12:11 AM, Chia-Ping Tsai wrote:
> > > > > > hi Matthias,
> > > > > >
> > > > > > Does this straightforward KIP still need 3 votes?
> > > > > >
> > > > > > On 2020/06/05 21:27:52, "Matthias J. Sax" 
> > wrote:
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> Thanks for the KIP!
> > > > > >>
> > > > > >>
> > > > > >> -Matthias
> > > > > >>
> > > > > >> On 6/4/20 11:25 PM, Chia-Ping Tsai wrote:
> > > > > >>> hi All,
> > > > > >>>
> > > > > >>> I would like to start the vote on KIP-620:
> > > > > >>>
> > > > > >>>
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=155749118
> > > > > >>>
> > > > > >>> --
> > > > > >>> Chia-Ping
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[Matthias J. Sax]

Thanks Yuriy!

What about `VoidSerde` ? It's not listed.

It might also be nice to add a short sentence and state that in addition
to fixing the name collisions, the KIP will also close the gap of
out-of-the-box serdes and add missing Serdes that are offered in Java to
Scala.


-Matthias

On 7/10/20 7:51 AM, Yuriy Badalyantc wrote:
> Oh, ok. I have done that. Just didn't know that it was necessary.
> 
> -Yuriy
> 
> On Fri, Jul 10, 2020 at 9:30 PM John Roesler  wrote:
> 
>> Ah, thanks Yuriy,
>>
>> Sorry if this wasn't clear, but _all_ public API changes have to
>> be explicitly included in the KIP. Can you just enumerate all
>> the contents of the new API?
>>
>> Thanks,
>> John
>>
>> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
>>> Hi, Matthias,
>>>
>>> It's not directly mentioned in the KIP, but I added all missing Java
>>> serdes. I mentioned it in the pull request description:
>>> https://github.com/apache/kafka/pull/8955
>>>
>>> And also, this KIP originally was based on a pull request where I added
>>> missing java serdes :) https://github.com/apache/kafka/pull/8049
>>>
>>> -Yuriy
>>>
>>> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax 
>> wrote:
>>>
 Yuriy,

 thanks for the KIP update. I have one follow up thought: I checked what
 default Serdes we offer in the Java class

  `org.apache.kafka.common.serialization.Serdes`

 and I think it would be good if we could close the gap between the Java
 and Scala code and add the missing Java Serdes in Scala, too.

 It seems we are missing `Short` (Java and Scala), `Void`, `UUID`, and
 `ByterBuffer`.

 Can we add those in addition?


 -Matthias

 On 7/8/20 6:45 AM, John Roesler wrote:
> Hi Yuriy,
>
> Once it seems like there’s general agreement in the discussion, you
>> can
 start a voting thread. You can find examples on the mailing list of
>> what to
 say in the first message. It’s basically just a message with the
>> subject
 line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating that
 you’d like to start the vote. It’s nice to link to the kip document
>> again.
>
> The rules for the vote are at the top of the “Kafka Improvement
>> Process”
 page, but you basically need 3 binding +1 votes and no binding -1
>> votes.
 You also need to wait at least three days from when you start the vote
 before you can declare it accepted. There’s no upper time limit.
>
> If you’re unsure of who has a binding vote, it’s just the people
>> listed
 on the Apache Kafka Committers page.
>
> If people are slow to vote, feel free to keep bumping the thread,
>> just
 like with the discussion.
>
> Thanks again for getting involved!
> -John
>
> On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote:
>> So, what's next? It's my first KIP and I'm not familiar with all
 processes.
>>
>> -Yuriy
>>
>> On Mon, Jul 6, 2020 at 1:32 AM John Roesler 
 wrote:
>>
>>> Hi Yuriy,
>>>
>>> Thanks for the update! It looks good to me.
>>>
>>> Thanks,
>>> John
>>>
>>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote:
 Hi John.

 I updated the KIP. An old proposed implementation is now in the
 rejected
 alternatives.

 - Yuriy

 On Sun, Jul 5, 2020 at 12:03 AM John Roesler >>
>>> wrote:

> Hi Yuriy,
>
> I agree, we can keep them separate. I just wanted to make you
>> aware
 of
>>> it.
>
> Thanks for the PR, it looks the way I expected.
>
> I just read over the KIP document again. I think it needs to be
>>> updated to
> the current proposal, and then we’ll be able to start the vote.
>
> Thanks,
> John
>
> On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote:
>> Hi everybody!
>>
>> Looks like a discussion about KIP-513 could take a while. I
>> think we
> should
>> move forward with KIP-616 without waiting for KIP-513.
>>
>> I created a new pull request for KIP-616:
>> https://github.com/apache/kafka/pull/8955. It contains a new
>> `org.apache.kafka.streams.scala.serialization.Serdes` object
>> without
>>> name
>> clash. An old one was marked as deprecated. This change is
>> backward
>> compatible and it could be merged in any further release.
>>
>> On Wed, Jun 3, 2020 at 12:41 PM Yuriy Badalyantc <
>> lmne...@gmail.com
>
> wrote:
>>
>>> Hi, John
>>>
>>> Thanks for pointing that out. I expressed my thoughts about
>>> KIP-513 and
>>> its connection to KIP-616 in the KIP-513 mail list.
>>>
>>> - Yuriy
>>>
>>> On Sun, May 31, 2020 at 1:26 AM John Roesler <
>> 

[jira] [Resolved] (KAFKA-10265) FetchRequest and FetchResponse should use the generated message classes

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-10265.
--
Resolution: Duplicate

> FetchRequest and FetchResponse should use the generated message classes
> ---
>
> Key: KAFKA-10265
> URL: https://issues.apache.org/jira/browse/KAFKA-10265
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Minor
>
> We should use the generated message classes for all our APIs. This ticket is 
> to track this improvements for the Fetch APIs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10265) FetchRequest and FetchResponse should use the generated message classes

[David Arthur (Jira)]

David Arthur created KAFKA-10265:


 Summary: FetchRequest and FetchResponse should use the generated 
message classes
 Key: KAFKA-10265
 URL: https://issues.apache.org/jira/browse/KAFKA-10265
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: David Arthur
Assignee: David Arthur


We should use the generated message classes for all our APIs. This ticket is to 
track this improvements for the Fetch APIs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

[Rajini Sivaram]

+1 (binding)

Thanks for the KIP, Colin!

Regards,

Rajini


On Thu, Jul 9, 2020 at 8:49 PM Colin McCabe  wrote:

> Hi all,
>
> I'd like to call a vote for KIP-554: Add a broker-side SCRAM configuration
> API.  The KIP is here: https://cwiki.apache.org/confluence/x/ihERCQ
>
> The previous discussion thread is here:
>
> https://lists.apache.org/thread.html/r69bdc65bdf58f5576944a551ff249d759073ecbf5daa441cff680ab0%40%3Cdev.kafka.apache.org%3E
>
> best,
> Colin
>

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[Yuriy Badalyantc]

Oh, ok. I have done that. Just didn't know that it was necessary.

-Yuriy

On Fri, Jul 10, 2020 at 9:30 PM John Roesler  wrote:

> Ah, thanks Yuriy,
>
> Sorry if this wasn't clear, but _all_ public API changes have to
> be explicitly included in the KIP. Can you just enumerate all
> the contents of the new API?
>
> Thanks,
> John
>
> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
> > Hi, Matthias,
> >
> > It's not directly mentioned in the KIP, but I added all missing Java
> > serdes. I mentioned it in the pull request description:
> > https://github.com/apache/kafka/pull/8955
> >
> > And also, this KIP originally was based on a pull request where I added
> > missing java serdes :) https://github.com/apache/kafka/pull/8049
> >
> > -Yuriy
> >
> > On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax 
> wrote:
> >
> > > Yuriy,
> > >
> > > thanks for the KIP update. I have one follow up thought: I checked what
> > > default Serdes we offer in the Java class
> > >
> > >  `org.apache.kafka.common.serialization.Serdes`
> > >
> > > and I think it would be good if we could close the gap between the Java
> > > and Scala code and add the missing Java Serdes in Scala, too.
> > >
> > > It seems we are missing `Short` (Java and Scala), `Void`, `UUID`, and
> > > `ByterBuffer`.
> > >
> > > Can we add those in addition?
> > >
> > >
> > > -Matthias
> > >
> > > On 7/8/20 6:45 AM, John Roesler wrote:
> > > > Hi Yuriy,
> > > >
> > > > Once it seems like there’s general agreement in the discussion, you
> can
> > > start a voting thread. You can find examples on the mailing list of
> what to
> > > say in the first message. It’s basically just a message with the
> subject
> > > line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating that
> > > you’d like to start the vote. It’s nice to link to the kip document
> again.
> > > >
> > > > The rules for the vote are at the top of the “Kafka Improvement
> Process”
> > > page, but you basically need 3 binding +1 votes and no binding -1
> votes.
> > > You also need to wait at least three days from when you start the vote
> > > before you can declare it accepted. There’s no upper time limit.
> > > >
> > > > If you’re unsure of who has a binding vote, it’s just the people
> listed
> > > on the Apache Kafka Committers page.
> > > >
> > > > If people are slow to vote, feel free to keep bumping the thread,
> just
> > > like with the discussion.
> > > >
> > > > Thanks again for getting involved!
> > > > -John
> > > >
> > > > On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote:
> > > >> So, what's next? It's my first KIP and I'm not familiar with all
> > > processes.
> > > >>
> > > >> -Yuriy
> > > >>
> > > >> On Mon, Jul 6, 2020 at 1:32 AM John Roesler 
> > > wrote:
> > > >>
> > > >>> Hi Yuriy,
> > > >>>
> > > >>> Thanks for the update! It looks good to me.
> > > >>>
> > > >>> Thanks,
> > > >>> John
> > > >>>
> > > >>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote:
> > >  Hi John.
> > > 
> > >  I updated the KIP. An old proposed implementation is now in the
> > > rejected
> > >  alternatives.
> > > 
> > >  - Yuriy
> > > 
> > >  On Sun, Jul 5, 2020 at 12:03 AM John Roesler  >
> > > >>> wrote:
> > > 
> > > > Hi Yuriy,
> > > >
> > > > I agree, we can keep them separate. I just wanted to make you
> aware
> > > of
> > > >>> it.
> > > >
> > > > Thanks for the PR, it looks the way I expected.
> > > >
> > > > I just read over the KIP document again. I think it needs to be
> > > >>> updated to
> > > > the current proposal, and then we’ll be able to start the vote.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote:
> > > >> Hi everybody!
> > > >>
> > > >> Looks like a discussion about KIP-513 could take a while. I
> think we
> > > > should
> > > >> move forward with KIP-616 without waiting for KIP-513.
> > > >>
> > > >> I created a new pull request for KIP-616:
> > > >> https://github.com/apache/kafka/pull/8955. It contains a new
> > > >> `org.apache.kafka.streams.scala.serialization.Serdes` object
> without
> > > >>> name
> > > >> clash. An old one was marked as deprecated. This change is
> backward
> > > >> compatible and it could be merged in any further release.
> > > >>
> > > >> On Wed, Jun 3, 2020 at 12:41 PM Yuriy Badalyantc <
> lmne...@gmail.com
> > > >
> > > > wrote:
> > > >>
> > > >>> Hi, John
> > > >>>
> > > >>> Thanks for pointing that out. I expressed my thoughts about
> > > >>> KIP-513 and
> > > >>> its connection to KIP-616 in the KIP-513 mail list.
> > > >>>
> > > >>> - Yuriy
> > > >>>
> > > >>> On Sun, May 31, 2020 at 1:26 AM John Roesler <
> vvcep...@apache.org>
> > > > wrote:
> > > >>>
> > >  Hi Yuriy,
> > > 
> > >  I was just looking back at KIP-513, and I’m wondering if
> there’s
> > > >>> any

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[John Roesler]

Ah, thanks Yuriy,

Sorry if this wasn't clear, but _all_ public API changes have to
be explicitly included in the KIP. Can you just enumerate all
the contents of the new API?

Thanks,
John

On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
> Hi, Matthias,
> 
> It's not directly mentioned in the KIP, but I added all missing Java
> serdes. I mentioned it in the pull request description:
> https://github.com/apache/kafka/pull/8955
> 
> And also, this KIP originally was based on a pull request where I added
> missing java serdes :) https://github.com/apache/kafka/pull/8049
> 
> -Yuriy
> 
> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax  wrote:
> 
> > Yuriy,
> >
> > thanks for the KIP update. I have one follow up thought: I checked what
> > default Serdes we offer in the Java class
> >
> >  `org.apache.kafka.common.serialization.Serdes`
> >
> > and I think it would be good if we could close the gap between the Java
> > and Scala code and add the missing Java Serdes in Scala, too.
> >
> > It seems we are missing `Short` (Java and Scala), `Void`, `UUID`, and
> > `ByterBuffer`.
> >
> > Can we add those in addition?
> >
> >
> > -Matthias
> >
> > On 7/8/20 6:45 AM, John Roesler wrote:
> > > Hi Yuriy,
> > >
> > > Once it seems like there’s general agreement in the discussion, you can
> > start a voting thread. You can find examples on the mailing list of what to
> > say in the first message. It’s basically just a message with the subject
> > line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating that
> > you’d like to start the vote. It’s nice to link to the kip document again.
> > >
> > > The rules for the vote are at the top of the “Kafka Improvement Process”
> > page, but you basically need 3 binding +1 votes and no binding -1 votes.
> > You also need to wait at least three days from when you start the vote
> > before you can declare it accepted. There’s no upper time limit.
> > >
> > > If you’re unsure of who has a binding vote, it’s just the people listed
> > on the Apache Kafka Committers page.
> > >
> > > If people are slow to vote, feel free to keep bumping the thread, just
> > like with the discussion.
> > >
> > > Thanks again for getting involved!
> > > -John
> > >
> > > On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote:
> > >> So, what's next? It's my first KIP and I'm not familiar with all
> > processes.
> > >>
> > >> -Yuriy
> > >>
> > >> On Mon, Jul 6, 2020 at 1:32 AM John Roesler 
> > wrote:
> > >>
> > >>> Hi Yuriy,
> > >>>
> > >>> Thanks for the update! It looks good to me.
> > >>>
> > >>> Thanks,
> > >>> John
> > >>>
> > >>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote:
> >  Hi John.
> > 
> >  I updated the KIP. An old proposed implementation is now in the
> > rejected
> >  alternatives.
> > 
> >  - Yuriy
> > 
> >  On Sun, Jul 5, 2020 at 12:03 AM John Roesler 
> > >>> wrote:
> > 
> > > Hi Yuriy,
> > >
> > > I agree, we can keep them separate. I just wanted to make you aware
> > of
> > >>> it.
> > >
> > > Thanks for the PR, it looks the way I expected.
> > >
> > > I just read over the KIP document again. I think it needs to be
> > >>> updated to
> > > the current proposal, and then we’ll be able to start the vote.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote:
> > >> Hi everybody!
> > >>
> > >> Looks like a discussion about KIP-513 could take a while. I think we
> > > should
> > >> move forward with KIP-616 without waiting for KIP-513.
> > >>
> > >> I created a new pull request for KIP-616:
> > >> https://github.com/apache/kafka/pull/8955. It contains a new
> > >> `org.apache.kafka.streams.scala.serialization.Serdes` object without
> > >>> name
> > >> clash. An old one was marked as deprecated. This change is backward
> > >> compatible and it could be merged in any further release.
> > >>
> > >> On Wed, Jun 3, 2020 at 12:41 PM Yuriy Badalyantc  > >
> > > wrote:
> > >>
> > >>> Hi, John
> > >>>
> > >>> Thanks for pointing that out. I expressed my thoughts about
> > >>> KIP-513 and
> > >>> its connection to KIP-616 in the KIP-513 mail list.
> > >>>
> > >>> - Yuriy
> > >>>
> > >>> On Sun, May 31, 2020 at 1:26 AM John Roesler 
> > > wrote:
> > >>>
> >  Hi Yuriy,
> > 
> >  I was just looking back at KIP-513, and I’m wondering if there’s
> > >>> any
> >  overlap we should consider here, or if they are just orthogonal.
> > 
> >  Thanks,
> >  -John
> > 
> >  On Thu, May 28, 2020, at 21:36, Yuriy Badalyantc wrote:
> > > At the current moment, I think John's plan is better than the
> > > original
> >  plan
> > > described in the KIP. I think we should create a new `Serdes` in
> > > another
> > > package. The old one will be deprecated.

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

[Tom Bentley]

Thanks Dongjin, I was hoping to close the vote, but was waiting till we had
consensus around the points you raised on the discussion thread.

On Fri, Jul 10, 2020 at 2:39 PM Dongjin Lee  wrote:

> +1. (non-binding)
>
> As of present, it has 3 bindings (Colin, Manikumar, and Mickael) and 2
> non-bindings (David, Dongjin).
>
> Since it has 3 binding +1 votes and more binding +1 votes than -1 votes,
> this KIP is accepted.
>
> Thanks everyone for the votes.
>
> On Thu, Jul 9, 2020 at 6:19 PM Mickael Maison 
> wrote:
>
> > +1 (binding)
> > Thanks
> >
> > On Thu, Jul 9, 2020 at 9:11 AM David Jacot  wrote:
> > >
> > > I couldn't think of a better alternative either. Thanks for the KIP!
> > >
> > > +1 (non-binding)
> > >
> > > On Wed, Jul 8, 2020 at 12:42 PM Manikumar 
> > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > On Tue, Jul 7, 2020 at 8:19 PM Colin McCabe 
> > wrote:
> > > >
> > > > > Thanks, Tom.
> > > > >
> > > > > I tried to think of a better way to do this, but I think you're
> right
> > > > that
> > > > > we probably just need different methods.
> > > > >
> > > > > +1 (binding).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Mon, Jul 6, 2020, at 01:14, Tom Bentley wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'd like to start a vote on KIP-621 which is about deprecating
> > methods
> > > > in
> > > > > > DescribeLogDirsResult which leak internal classes, replacing them
> > with
> > > > > > public API classes.
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158862109
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Tom
> > > > > >
> > > > >
> > > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

[Dongjin Lee]

+1. (non-binding)

As of present, it has 3 bindings (Colin, Manikumar, and Mickael) and 2
non-bindings (David, Dongjin).

Since it has 3 binding +1 votes and more binding +1 votes than -1 votes,
this KIP is accepted.

Thanks everyone for the votes.

On Thu, Jul 9, 2020 at 6:19 PM Mickael Maison 
wrote:

> +1 (binding)
> Thanks
>
> On Thu, Jul 9, 2020 at 9:11 AM David Jacot  wrote:
> >
> > I couldn't think of a better alternative either. Thanks for the KIP!
> >
> > +1 (non-binding)
> >
> > On Wed, Jul 8, 2020 at 12:42 PM Manikumar 
> wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks for the KIP.
> > >
> > > On Tue, Jul 7, 2020 at 8:19 PM Colin McCabe 
> wrote:
> > >
> > > > Thanks, Tom.
> > > >
> > > > I tried to think of a better way to do this, but I think you're right
> > > that
> > > > we probably just need different methods.
> > > >
> > > > +1 (binding).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Mon, Jul 6, 2020, at 01:14, Tom Bentley wrote:
> > > > > Hi,
> > > > >
> > > > > I'd like to start a vote on KIP-621 which is about deprecating
> methods
> > > in
> > > > > DescribeLogDirsResult which leak internal classes, replacing them
> with
> > > > > public API classes.
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158862109
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Tom
> > > > >
> > > >
> > >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*




*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

[Dongjin Lee]

Hi Tom and Colin,

I see. Thanks for the comprehensive explanation. I learned a lot.

Although my approach includes a new member field
`DescribeLogDirsResult.LogDirInfo.tpToReplicaInfos` to keep binary
compatibility, but anyway, you are right - it does not worth except small
consistency gain.

Best,
Dongjin

On Fri, Jul 10, 2020 at 3:07 PM Colin McCabe  wrote:

> Incidentally, whenever we do a hard compatibility break with this API, we
> should probably get rid of the function that iterates over every replica on
> the broker.  That's not a scalable API for the future.  We also probably
> should have an API to list just the directories that are moving, which is
> what most tools care about anyway.
>
> best,
> Colin
>
>
> On Thu, Jul 9, 2020, at 23:04, Colin McCabe wrote:
> > Yeah.  The issue with subclassing is that it's a source compatibility
> > break, although not (I think) a binary compatibility break.  I don't
> > think it's really worth it given that it leaves us with a weird class
> > hierarchy, and we still need to do a hard compatibility break later to
> > fix the real problem with either solution.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Jul 9, 2020, at 08:39, Tom Bentley wrote:
> > > Hi Dongjin,
> > >
> > > Thanks for updating the link and sorry that you put effort into writing
> > > your own KIP for this. I was aware of KAFKA-8794, but since the PR was
> > > about javadocing the internal classes I felt it wasn't quite the same
> > > thing. All the same, I should have pinged you on KAFKA-8794, sorry.
> > >
> > > I did consider subclassing too. Initially I thought it would be binary
> > > incompatible, but thinking about it more I realise that, perhaps, it's
> > > binary compatible due to erasure, (I'd have to test it to be sure, or
> > > perhaps you've already done so, and can confirm).
> > >
> > > But I think there are a few other reasons to be considered:
> > >
> > > * The change would not be source compatible (people would have to
> change
> > > their source code) because the signature of all() and values() use type
> > > arguments invariantly, i.e. KafkaFuture > > DescribeLogDirsResult .LogDirInfo>>> is not a subtype of
> > > KafkaFuture DescribeLogDirsResponse.LogDirInfo>>>
> > > so you'd get a compile time error as use sites where you were trying to
> > > assign the new result to the old type (and this applies to all the
> > > intermediate generic types too).
> > > * There's also the fact that `replicaInfos` and `error` is accessed
> via a
> > > field, rather than a method. To fix that you would need to:
> > > * Add methods to the new subclasses and deprecate the old fields
> (so
> > > that a user gets a warning if accessing the field, even though they've
> > > eliminated the old class name from the code).
> > > * As Colin has mentioned, you'd have to deprecate those classes and
> fields,
> > > even though they're not really deprecated from the internal PoV,
> they're
> > > only deprecated from the external PoV.
> > > * Using subclassing in this way would introduce a different kind of
> > > inconsistency: other APIs don't use internal classes at all, even via
> > > inheritance.
> > > * Using static member classes is inconsistent with other Admin APIs
> such as
> > > TopicDescription (though I think this is trivially fixed in your
> proposal).
> > >
> > > The loss of consistency wrt to the method names in the approach
> advocated
> > > by KIP-621 could eventually be removed by adding "new" values() and
> all()
> > > methods back with those names but the new types. Indeed, I _think_ this
> > > could be done in a major release without breaking SemVer.
> > >
> > > All considered, I think it's cleaner to deprecate the methods and
> create
> > > new classes completely independent of the internal ones.
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > > On Thu, Jul 9, 2020 at 4:26 PM Dongjin Lee  wrote:
> > >
> > > > Hi Colin,
> > > >
> > > >
> > > > Thanks for the quick response. Sure, the problem is that the internal
> > > > classes (DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo], or 'the
> old
> > > > ones') are exposed to the public API, not the class itself. You are
> right.
> > > >
> > > >
> > > > However, deprecating DescribeLogDirsResult#[all, values] and
> defining the
> > > > new methods causes another problem: consistency. As of 2.5, most of
> the
> > > > admin result classes (listed below) have 'all' and 'values' methods.
> Since
> > > > these methods are public APIs, I think keeping consistency would be
> better.
> > > >
> > > >
> > > >
> > > >-
> > > >
> > > >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/CreateTopicsResult.html
> > > >-
> > > >
> > > >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/AlterPartitionReassignmentsResult.html
> > > >-
> > > >
> > > >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/CreateAclsResult.html
> > > >-
> > > >
> > > >
> 

Build failed in Jenkins: kafka-trunk-jdk11 #1634

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10249: don't try to read un-checkpointed offsets of in-memory


--
[...truncated 1.91 MB...]

kafka.api.GroupAuthorizerIntegrationTest > testAuthorizedProduceAndConsume 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testAuthorizedProduceAndConsume 
PASSED

kafka.api.DescribeAuthorizedOperationsTest > testClusterAuthorizedOperations 
STARTED

kafka.api.DescribeAuthorizedOperationsTest > testClusterAuthorizedOperations 
PASSED

kafka.api.DescribeAuthorizedOperationsTest > testTopicAuthorizedOperations 
STARTED

kafka.api.DescribeAuthorizedOperationsTest > testTopicAuthorizedOperations 
PASSED

kafka.api.DescribeAuthorizedOperationsTest > 
testConsumerGroupAuthorizedOperations STARTED

kafka.api.DescribeAuthorizedOperationsTest > 
testConsumerGroupAuthorizedOperations PASSED

kafka.api.TransactionsTest > testBumpTransactionalEpoch STARTED

kafka.api.TransactionsTest > testBumpTransactionalEpoch PASSED

kafka.api.TransactionsTest > testSendOffsetsWithGroupMetadata STARTED

kafka.api.TransactionsTest > testSendOffsetsWithGroupMetadata PASSED

kafka.api.TransactionsTest > testBasicTransactions STARTED

kafka.api.TransactionsTest > testBasicTransactions PASSED

kafka.api.TransactionsTest > testSendOffsetsWithGroupId STARTED

kafka.api.TransactionsTest > testSendOffsetsWithGroupId PASSED

kafka.api.TransactionsTest > testFencingOnSendOffsets STARTED

kafka.api.TransactionsTest > testFencingOnSendOffsets PASSED

kafka.api.TransactionsTest > testFencingOnAddPartitions STARTED

kafka.api.TransactionsTest > testFencingOnAddPartitions PASSED

kafka.api.TransactionsTest > testFencingOnTransactionExpiration STARTED

kafka.api.TransactionsTest > testFencingOnTransactionExpiration PASSED

kafka.api.TransactionsTest > testDelayedFetchIncludesAbortedTransaction STARTED

kafka.api.TransactionsTest > testDelayedFetchIncludesAbortedTransaction PASSED

kafka.api.TransactionsTest > testOffsetMetadataInSendOffsetsToTransaction 
STARTED

kafka.api.TransactionsTest > testOffsetMetadataInSendOffsetsToTransaction PASSED

kafka.api.TransactionsTest > testConsecutivelyRunInitTransactions STARTED

kafka.api.TransactionsTest > testConsecutivelyRunInitTransactions PASSED

kafka.api.TransactionsTest > testReadCommittedConsumerShouldNotSeeUndecidedData 
STARTED

kafka.api.TransactionsTest > testReadCommittedConsumerShouldNotSeeUndecidedData 
PASSED

kafka.api.TransactionsTest > testFencingOnSend STARTED

kafka.api.TransactionsTest > testFencingOnSend PASSED

kafka.api.TransactionsTest > testFencingOnCommit STARTED

kafka.api.TransactionsTest > testFencingOnCommit PASSED

kafka.api.TransactionsTest > testMultipleMarkersOneLeader STARTED

kafka.api.TransactionsTest > testMultipleMarkersOneLeader PASSED

kafka.api.TransactionsTest > testCommitTransactionTimeout STARTED

kafka.api.TransactionsTest > testCommitTransactionTimeout PASSED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideLowerQuota STARTED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideLowerQuota PASSED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled 
STARTED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.UserClientIdQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.UserClientIdQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.UserClientIdQuotaTest > testThrottledRequest STARTED

kafka.api.UserClientIdQuotaTest > testThrottledRequest PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED
ERROR: No tool found matching GRADLE_4_10_2_HOME
ERROR: Could not install GRADLE_4_10_3_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:873)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:484)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:693)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:658)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400)
at hudson.scm.SCM.poll(SCM.java:417)
at hudson.model.AbstractProject._poll(AbstractProject.java:1390)
at hudson.model.AbstractProject.poll(AbstractProject.java:1293)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:603)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:649)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at 

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

[Unmesh Joshi]

I was thinking that we might need something like multi-operation
 record in zookeeper
to atomically create topic and partition records when this multi record is
committed.  This way metadata will have both the TopicRecord and
PartitionRecord together always, and in no situation we can have
TopicRecord without PartitionRecord. Not sure if there are other situations
where multi-operation is needed.


Thanks,
Unmesh

On Fri, Jul 10, 2020 at 11:32 AM Colin McCabe  wrote:

> Hi Unmesh,
>
> Yes, once the last stable offset advanced, we would consider the topic
> creation to be done, and then we could return success to the client.
>
> best,
> Colin
>
> On Thu, Jul 9, 2020, at 19:44, Unmesh Joshi wrote:
> > It still needs HighWaterMark / LastStableOffset to be advanced by two
> > records? Something like following?
> >
> >
> >||
> > <--||   HighWaterMark
> >Response|PartitionRecord |
> >||
> >-|
> >| TopicRecord|  -
> >||
> > --->   --   Previous HighWaterMark
> >CreateTopic ||
> >||
> >||
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Jul 10, 2020 at 1:30 AM Colin McCabe  wrote:
> >
> > > On Thu, Jul 9, 2020, at 04:37, Unmesh Joshi wrote:
> > > > I see that, when a new topic is created, two metadata records, a
> > > > TopicRecord (just the name and id of the topic) and a PartitionRecord
> > > (more
> > > > like LeaderAndIsr, with leader id and replica ids for the partition)
> are
> > > > created.
> > > > While creating the topic, log entries for both the records need to be
> > > > committed in RAFT core. Will it need something like a
> > > MultiOperationRecord
> > > > in zookeeper. Then, we can have a single log entry with both the
> records,
> > > > and  the create topic request can be fulfilled atomically when both
> the
> > > > records are committed?
> > > >
> > >
> > > Hi Unmesh,
> > >
> > > Since the active controller is the only node writing to the log, there
> is
> > > no need for any kind of synchronization or access control at the log
> level.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > > Unmesh
> > > >
> > > > On Wed, Jul 8, 2020 at 6:57 AM Ron Dagostino 
> wrote:
> > > >
> > > > > HI Colin.  Thanks for the KIP.  Here is some feedback and various
> > > > > questions.
> > > > >
> > > > > "*Controller processes will listen on a separate port from brokers.
> > > This
> > > > > will be true even when the broker and controller are co-located in
> the
> > > same
> > > > > JVM*". I assume it is possible that the port numbers could be the
> same
> > > when
> > > > > using separate JVMs (i.e. broker uses port 9192 and controller also
> > > uses
> > > > > port 9192).  I think it would be clearer to state this along these
> > > > > lines: "Controller
> > > > > nodes will listen on a port, and the controller port must differ
> from
> > > any
> > > > > port that a broker in the same JVM is listening on.  In other
> words, a
> > > > > controller and a broker node, when in the same JVM, do not share
> ports"
> > > > >
> > > > > I think the sentence "*In the realm of ACLs, this translates to
> > > controllers
> > > > > requiring CLUSTERACTION on CLUSTER for all operations*" is
> confusing.
> > > It
> > > > > feels to me that you can just delete it.  Am I missing something
> here?
> > > > >
> > > > > The KIP states "*The metadata will be stored in memory on all the
> > > active
> > > > > controllers.*"  Can there be multiple active controllers?  Should
> it
> > > > > instead read "The metadata will be stored in memory on all
> potential
> > > > > controllers." (or something like that)?
> > > > >
> > > > > KIP-595 states "*we have assumed the name __cluster_metadata for
> this
> > > > > topic, but this is not a formal part of this proposal*".  This
> KIP-631
> > > > > states "*Metadata changes need to be persisted to the __metadata
> log
> > > before
> > > > > we propagate them to the other nodes in the cluster.  This means
> > > waiting
> > > > > for the metadata log's last stable offset to advance to the offset
> of
> > > the
> > > > > change.*"  Are we here formally defining "__metadata" as the topic
> > > name,
> > > > > and should these sentences refer to "__metadata topic" rather than
> > > > > "__metadata log"?  What are the "other nodes in the cluster" that
> are
> > > > > referred to?  These are not controller nodes but brokers, right?
> If
> > > so,
> > > > > then should we say "before we propagate them to the brokers"?
> > > Technically
> > > > > we have a controller cluster and 

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[Yuriy Badalyantc]

Hi, Matthias,

It's not directly mentioned in the KIP, but I added all missing Java
serdes. I mentioned it in the pull request description:
https://github.com/apache/kafka/pull/8955

And also, this KIP originally was based on a pull request where I added
missing java serdes :) https://github.com/apache/kafka/pull/8049

-Yuriy

On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax  wrote:

> Yuriy,
>
> thanks for the KIP update. I have one follow up thought: I checked what
> default Serdes we offer in the Java class
>
>  `org.apache.kafka.common.serialization.Serdes`
>
> and I think it would be good if we could close the gap between the Java
> and Scala code and add the missing Java Serdes in Scala, too.
>
> It seems we are missing `Short` (Java and Scala), `Void`, `UUID`, and
> `ByterBuffer`.
>
> Can we add those in addition?
>
>
> -Matthias
>
> On 7/8/20 6:45 AM, John Roesler wrote:
> > Hi Yuriy,
> >
> > Once it seems like there’s general agreement in the discussion, you can
> start a voting thread. You can find examples on the mailing list of what to
> say in the first message. It’s basically just a message with the subject
> line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating that
> you’d like to start the vote. It’s nice to link to the kip document again.
> >
> > The rules for the vote are at the top of the “Kafka Improvement Process”
> page, but you basically need 3 binding +1 votes and no binding -1 votes.
> You also need to wait at least three days from when you start the vote
> before you can declare it accepted. There’s no upper time limit.
> >
> > If you’re unsure of who has a binding vote, it’s just the people listed
> on the Apache Kafka Committers page.
> >
> > If people are slow to vote, feel free to keep bumping the thread, just
> like with the discussion.
> >
> > Thanks again for getting involved!
> > -John
> >
> > On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote:
> >> So, what's next? It's my first KIP and I'm not familiar with all
> processes.
> >>
> >> -Yuriy
> >>
> >> On Mon, Jul 6, 2020 at 1:32 AM John Roesler 
> wrote:
> >>
> >>> Hi Yuriy,
> >>>
> >>> Thanks for the update! It looks good to me.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote:
>  Hi John.
> 
>  I updated the KIP. An old proposed implementation is now in the
> rejected
>  alternatives.
> 
>  - Yuriy
> 
>  On Sun, Jul 5, 2020 at 12:03 AM John Roesler 
> >>> wrote:
> 
> > Hi Yuriy,
> >
> > I agree, we can keep them separate. I just wanted to make you aware
> of
> >>> it.
> >
> > Thanks for the PR, it looks the way I expected.
> >
> > I just read over the KIP document again. I think it needs to be
> >>> updated to
> > the current proposal, and then we’ll be able to start the vote.
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote:
> >> Hi everybody!
> >>
> >> Looks like a discussion about KIP-513 could take a while. I think we
> > should
> >> move forward with KIP-616 without waiting for KIP-513.
> >>
> >> I created a new pull request for KIP-616:
> >> https://github.com/apache/kafka/pull/8955. It contains a new
> >> `org.apache.kafka.streams.scala.serialization.Serdes` object without
> >>> name
> >> clash. An old one was marked as deprecated. This change is backward
> >> compatible and it could be merged in any further release.
> >>
> >> On Wed, Jun 3, 2020 at 12:41 PM Yuriy Badalyantc  >
> > wrote:
> >>
> >>> Hi, John
> >>>
> >>> Thanks for pointing that out. I expressed my thoughts about
> >>> KIP-513 and
> >>> its connection to KIP-616 in the KIP-513 mail list.
> >>>
> >>> - Yuriy
> >>>
> >>> On Sun, May 31, 2020 at 1:26 AM John Roesler 
> > wrote:
> >>>
>  Hi Yuriy,
> 
>  I was just looking back at KIP-513, and I’m wondering if there’s
> >>> any
>  overlap we should consider here, or if they are just orthogonal.
> 
>  Thanks,
>  -John
> 
>  On Thu, May 28, 2020, at 21:36, Yuriy Badalyantc wrote:
> > At the current moment, I think John's plan is better than the
> > original
>  plan
> > described in the KIP. I think we should create a new `Serdes` in
> > another
> > package. The old one will be deprecated.
> >
> > - Yuriy
> >
> > On Fri, May 29, 2020 at 8:58 AM John Roesler <
> >>> vvcep...@apache.org>
>  wrote:
> >
> >> Thanks, Matthias,
> >>
> >> If we go with the approach Yuriy and I agreed on, to
> >>> deprecate and
>  replace
> >> the whole class and not just a few of the methods, then the
> > timeline
>  is
> >> less of a concern. Under that plan, Yuriy can just write the
> >>> new
> > class
> >> 

Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

[Badai Aqrandista]

Hu

I would make it more readable and move the key next to the value like this:

CreateTime:1592475472398|Partition:3|Offset:0|Headers:h1=v1,h2=v2|key1|value1

What do you think?

In future KIP, we'll add some formatting like David Jacot suggested.

Regards
Badai

On Fri, Jul 10, 2020 at 2:57 PM huxi_2b  wrote:
>
> Badai,
>
> Replying in this thread. Two possible ways come up into my head:
>
> 1. CreateTime:1592475472398|key1|3|<0>|h1=v1,h2=v2|value1
> 2. CreateTime:1592475472398|key1|3|offset=0|h1=v1,h2=v2|value1
>
> I prefer the option #1 which could be accomplished by a complementary usage 
> description. What do you think?
>
> On 2020/06/21 13:39:36, Badai Aqrandista  wrote:
> > Excellent.>
> >
> > Would like to hear more feedback from others.>
> >
> > On Sat, Jun 20, 2020 at 1:27 AM David Jacot  wrote:>
> > >>
> > > Hi Badai,>
> > >>
> > > Thanks for your reply.>
> > >>
> > > 2. Yes, that makes sense.>
> > >>
> > > Best,>
> > > David>
> > >>
> > > On Thu, Jun 18, 2020 at 2:08 PM Badai Aqrandista  
> > > wrote:>
> > >>
> > > > David>
> > > >>
> > > > Thank you for replying>
> > > >>
> > > > 1. It seems that `print.partition` is already implemented. Do you 
> > > > confirm?>
> > > > BADAI: Yes, you are correct. I have removed it from the KIP.>
> > > >>
> > > > 2. Will `null.literal` be only used when the value of the message>
> > > > is NULL or for any fields? Also, it seems that we print out "null">
> > > > today when the key or the value is empty. Shall we use "null" as>
> > > > a default instead of ""?>
> > > > BADAI: For any fields. Do you think this is useful?>
> > > >>
> > > > 3. Could we add a small example of the output in the KIP?>
> > > > BADAI: Yes, I have updated the KIP to add a couple of example.>
> > > >>
> > > > 4. When there are no headers, are we going to print something>
> > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP>
> > > > where there is no timestamp.>
> > > > BADAI: Yes, good idea. I have updated the KIP to print NO_HEADERS.>
> > > >>
> > > > Thanks>
> > > > Badai>
> > > >>
> > > >>
> > > > On Thu, Jun 18, 2020 at 7:25 PM David Jacot  wrote:>
> > > > >>
> > > > > Hi Badai,>
> > > > >>
> > > > > Thanks for resuming this. I have few small comments:>
> > > > >>
> > > > > 1. It seems that `print.partition` is already implemented. Do you>
> > > > confirm?>
> > > > >>
> > > > > 2. Will `null.literal` be only used when the value of the message>
> > > > > is NULL or for any fields? Also, it seems that we print out "null">
> > > > > today when the key or the value is empty. Shall we use "null" as>
> > > > > a default instead of ""?>
> > > > >>
> > > > > 3. Could we add a small example of the output in the KIP?>
> > > > >>
> > > > > 4. When there are no headers, are we going to print something>
> > > > > to indicate it to the user? For instance, we print out NO_TIMESTAMP>
> > > > > where there is no timestamp.>
> > > > >>
> > > > > Best,>
> > > > > David>
> > > > >>
> > > > > On Wed, Jun 17, 2020 at 4:53 PM Badai Aqrandista >
> > > > wrote:>
> > > > >>
> > > > > > Hi all,>
> > > > > >>
> > > > > > I have contacted Mateusz separately and he is ok for me to take 
> > > > > > over>
> > > > > > KIP-431:>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter>
> > > > > >>
> > > > > > I have updated it a bit. Can anyone give a quick look at it again 
> > > > > > and>
> > > > > > give me some feedback?>
> > > > > >>
> > > > > > This feature will be very helpful for people supporting Kafka in>
> > > > > > operations.>
> > > > > >>
> > > > > > If it is ready for a vote, please let me know.>
> > > > > >>
> > > > > > Thanks>
> > > > > > Badai>
> > > > > >>
> > > > > > On Sat, Jun 13, 2020 at 10:59 PM Badai Aqrandista 
> > > > > > >
> > > > > > wrote:>
> > > > > > >>
> > > > > > > Mateusz>
> > > > > > >>
> > > > > > > This KIP would be very useful for debugging. But the last 
> > > > > > > discussion>
> > > > > > > is in Feb 2019.>
> > > > > > >>
> > > > > > > Are you ok if I take over this KIP?>
> > > > > > >>
> > > > > > > -->
> > > > > > > Thanks,>
> > > > > > > Badai>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > > -->
> > > > > > Thanks,>
> > > > > > Badai>
> > > > > >>
> > > >>
> > > >>
> > > >>
> > > > -->
> > > > Thanks,>
> > > > Badai>
> > > >>
> >
> >
> >
> > -- >
> > Thanks,>
> > Badai>
> >



-- 
Thanks,
Badai

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

[Colin McCabe]

Incidentally, whenever we do a hard compatibility break with this API, we 
should probably get rid of the function that iterates over every replica on the 
broker.  That's not a scalable API for the future.  We also probably should 
have an API to list just the directories that are moving, which is what most 
tools care about anyway.

best,
Colin


On Thu, Jul 9, 2020, at 23:04, Colin McCabe wrote:
> Yeah.  The issue with subclassing is that it's a source compatibility 
> break, although not (I think) a binary compatibility break.  I don't 
> think it's really worth it given that it leaves us with a weird class 
> hierarchy, and we still need to do a hard compatibility break later to 
> fix the real problem with either solution.
> 
> best,
> Colin
> 
> 
> On Thu, Jul 9, 2020, at 08:39, Tom Bentley wrote:
> > Hi Dongjin,
> > 
> > Thanks for updating the link and sorry that you put effort into writing
> > your own KIP for this. I was aware of KAFKA-8794, but since the PR was
> > about javadocing the internal classes I felt it wasn't quite the same
> > thing. All the same, I should have pinged you on KAFKA-8794, sorry.
> > 
> > I did consider subclassing too. Initially I thought it would be binary
> > incompatible, but thinking about it more I realise that, perhaps, it's
> > binary compatible due to erasure, (I'd have to test it to be sure, or
> > perhaps you've already done so, and can confirm).
> > 
> > But I think there are a few other reasons to be considered:
> > 
> > * The change would not be source compatible (people would have to change
> > their source code) because the signature of all() and values() use type
> > arguments invariantly, i.e. KafkaFuture > DescribeLogDirsResult .LogDirInfo>>> is not a subtype of
> > KafkaFuture>>
> > so you'd get a compile time error as use sites where you were trying to
> > assign the new result to the old type (and this applies to all the
> > intermediate generic types too).
> > * There's also the fact that `replicaInfos` and `error` is accessed via a
> > field, rather than a method. To fix that you would need to:
> > * Add methods to the new subclasses and deprecate the old fields (so
> > that a user gets a warning if accessing the field, even though they've
> > eliminated the old class name from the code).
> > * As Colin has mentioned, you'd have to deprecate those classes and fields,
> > even though they're not really deprecated from the internal PoV, they're
> > only deprecated from the external PoV.
> > * Using subclassing in this way would introduce a different kind of
> > inconsistency: other APIs don't use internal classes at all, even via
> > inheritance.
> > * Using static member classes is inconsistent with other Admin APIs such as
> > TopicDescription (though I think this is trivially fixed in your proposal).
> > 
> > The loss of consistency wrt to the method names in the approach advocated
> > by KIP-621 could eventually be removed by adding "new" values() and all()
> > methods back with those names but the new types. Indeed, I _think_ this
> > could be done in a major release without breaking SemVer.
> > 
> > All considered, I think it's cleaner to deprecate the methods and create
> > new classes completely independent of the internal ones.
> > 
> > Kind regards,
> > 
> > Tom
> > 
> > On Thu, Jul 9, 2020 at 4:26 PM Dongjin Lee  wrote:
> > 
> > > Hi Colin,
> > >
> > >
> > > Thanks for the quick response. Sure, the problem is that the internal
> > > classes (DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo], or 'the old
> > > ones') are exposed to the public API, not the class itself. You are right.
> > >
> > >
> > > However, deprecating DescribeLogDirsResult#[all, values] and defining the
> > > new methods causes another problem: consistency. As of 2.5, most of the
> > > admin result classes (listed below) have 'all' and 'values' methods. Since
> > > these methods are public APIs, I think keeping consistency would be 
> > > better.
> > >
> > >
> > >
> > >-
> > >
> > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/CreateTopicsResult.html
> > >-
> > >
> > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/AlterPartitionReassignmentsResult.html
> > >-
> > >
> > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/CreateAclsResult.html
> > >-
> > >
> > > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/AlterConfigsResult.html
> > >
> > >
> > > In contrast, my approach - deprecating the old ones and defining the new
> > > ones (DescribeLogDirsResult.[LogDirInfo, ReplicaInfo]) that extends the 
> > > old
> > > counterpart - not only not breaking the consistency but also gives the
> > > users the message that they have to use the new ones.
> > >
> > >
> > > So, the overall process would be:
> > >
> > >
> > >
> > >1. Deprecate the old ones and make DescribeLogDirsResult#[all, values]
> > >to return the new ones, without changing the return type. Since the new
> 

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

[Colin McCabe]

Yeah.  The issue with subclassing is that it's a source compatibility break, 
although not (I think) a binary compatibility break.  I don't think it's really 
worth it given that it leaves us with a weird class hierarchy, and we still 
need to do a hard compatibility break later to fix the real problem with either 
solution.

best,
Colin


On Thu, Jul 9, 2020, at 08:39, Tom Bentley wrote:
> Hi Dongjin,
> 
> Thanks for updating the link and sorry that you put effort into writing
> your own KIP for this. I was aware of KAFKA-8794, but since the PR was
> about javadocing the internal classes I felt it wasn't quite the same
> thing. All the same, I should have pinged you on KAFKA-8794, sorry.
> 
> I did consider subclassing too. Initially I thought it would be binary
> incompatible, but thinking about it more I realise that, perhaps, it's
> binary compatible due to erasure, (I'd have to test it to be sure, or
> perhaps you've already done so, and can confirm).
> 
> But I think there are a few other reasons to be considered:
> 
> * The change would not be source compatible (people would have to change
> their source code) because the signature of all() and values() use type
> arguments invariantly, i.e. KafkaFuture DescribeLogDirsResult .LogDirInfo>>> is not a subtype of
> KafkaFuture>>
> so you'd get a compile time error as use sites where you were trying to
> assign the new result to the old type (and this applies to all the
> intermediate generic types too).
> * There's also the fact that `replicaInfos` and `error` is accessed via a
> field, rather than a method. To fix that you would need to:
> * Add methods to the new subclasses and deprecate the old fields (so
> that a user gets a warning if accessing the field, even though they've
> eliminated the old class name from the code).
> * As Colin has mentioned, you'd have to deprecate those classes and fields,
> even though they're not really deprecated from the internal PoV, they're
> only deprecated from the external PoV.
> * Using subclassing in this way would introduce a different kind of
> inconsistency: other APIs don't use internal classes at all, even via
> inheritance.
> * Using static member classes is inconsistent with other Admin APIs such as
> TopicDescription (though I think this is trivially fixed in your proposal).
> 
> The loss of consistency wrt to the method names in the approach advocated
> by KIP-621 could eventually be removed by adding "new" values() and all()
> methods back with those names but the new types. Indeed, I _think_ this
> could be done in a major release without breaking SemVer.
> 
> All considered, I think it's cleaner to deprecate the methods and create
> new classes completely independent of the internal ones.
> 
> Kind regards,
> 
> Tom
> 
> On Thu, Jul 9, 2020 at 4:26 PM Dongjin Lee  wrote:
> 
> > Hi Colin,
> >
> >
> > Thanks for the quick response. Sure, the problem is that the internal
> > classes (DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo], or 'the old
> > ones') are exposed to the public API, not the class itself. You are right.
> >
> >
> > However, deprecating DescribeLogDirsResult#[all, values] and defining the
> > new methods causes another problem: consistency. As of 2.5, most of the
> > admin result classes (listed below) have 'all' and 'values' methods. Since
> > these methods are public APIs, I think keeping consistency would be better.
> >
> >
> >
> >-
> >
> > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/CreateTopicsResult.html
> >-
> >
> > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/AlterPartitionReassignmentsResult.html
> >-
> >
> > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/CreateAclsResult.html
> >-
> >
> > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/admin/AlterConfigsResult.html
> >
> >
> > In contrast, my approach - deprecating the old ones and defining the new
> > ones (DescribeLogDirsResult.[LogDirInfo, ReplicaInfo]) that extends the old
> > counterpart - not only not breaking the consistency but also gives the
> > users the message that they have to use the new ones.
> >
> >
> > So, the overall process would be:
> >
> >
> >
> >1. Deprecate the old ones and make DescribeLogDirsResult#[all, values]
> >to return the new ones, without changing the return type. Since the new
> >ones extend the old ones, it does not cause any problems.
> >2. Since deprecation message guides to move to the new ones, the uses
> >will migrate to use the new classes.
> >3. After a time, do the following:
> >   1. Change the return type of DescribeLogDirsResult#[all, values] from
> >   old ones to the new ones, with a notice. Since we already deprecated
> > the
> >   old ones, most users would already be moved into the new ones. So it
> > does
> >   not occur any problems.
> >   2. *Hide the old ones from the public, with removing the deprecated
> >   annotation.* 

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

[Colin McCabe]

Hi Unmesh,

Yes, once the last stable offset advanced, we would consider the topic creation 
to be done, and then we could return success to the client.

best,
Colin

On Thu, Jul 9, 2020, at 19:44, Unmesh Joshi wrote:
> It still needs HighWaterMark / LastStableOffset to be advanced by two
> records? Something like following?
> 
> 
>||
> <--||   HighWaterMark
>Response|PartitionRecord |
>||
>-|
>| TopicRecord|  -
>||
> --->   --   Previous HighWaterMark
>CreateTopic ||
>||
>||
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Fri, Jul 10, 2020 at 1:30 AM Colin McCabe  wrote:
> 
> > On Thu, Jul 9, 2020, at 04:37, Unmesh Joshi wrote:
> > > I see that, when a new topic is created, two metadata records, a
> > > TopicRecord (just the name and id of the topic) and a PartitionRecord
> > (more
> > > like LeaderAndIsr, with leader id and replica ids for the partition) are
> > > created.
> > > While creating the topic, log entries for both the records need to be
> > > committed in RAFT core. Will it need something like a
> > MultiOperationRecord
> > > in zookeeper. Then, we can have a single log entry with both the records,
> > > and  the create topic request can be fulfilled atomically when both the
> > > records are committed?
> > >
> >
> > Hi Unmesh,
> >
> > Since the active controller is the only node writing to the log, there is
> > no need for any kind of synchronization or access control at the log level.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > > Unmesh
> > >
> > > On Wed, Jul 8, 2020 at 6:57 AM Ron Dagostino  wrote:
> > >
> > > > HI Colin.  Thanks for the KIP.  Here is some feedback and various
> > > > questions.
> > > >
> > > > "*Controller processes will listen on a separate port from brokers.
> > This
> > > > will be true even when the broker and controller are co-located in the
> > same
> > > > JVM*". I assume it is possible that the port numbers could be the same
> > when
> > > > using separate JVMs (i.e. broker uses port 9192 and controller also
> > uses
> > > > port 9192).  I think it would be clearer to state this along these
> > > > lines: "Controller
> > > > nodes will listen on a port, and the controller port must differ from
> > any
> > > > port that a broker in the same JVM is listening on.  In other words, a
> > > > controller and a broker node, when in the same JVM, do not share ports"
> > > >
> > > > I think the sentence "*In the realm of ACLs, this translates to
> > controllers
> > > > requiring CLUSTERACTION on CLUSTER for all operations*" is confusing.
> > It
> > > > feels to me that you can just delete it.  Am I missing something here?
> > > >
> > > > The KIP states "*The metadata will be stored in memory on all the
> > active
> > > > controllers.*"  Can there be multiple active controllers?  Should it
> > > > instead read "The metadata will be stored in memory on all potential
> > > > controllers." (or something like that)?
> > > >
> > > > KIP-595 states "*we have assumed the name __cluster_metadata for this
> > > > topic, but this is not a formal part of this proposal*".  This KIP-631
> > > > states "*Metadata changes need to be persisted to the __metadata log
> > before
> > > > we propagate them to the other nodes in the cluster.  This means
> > waiting
> > > > for the metadata log's last stable offset to advance to the offset of
> > the
> > > > change.*"  Are we here formally defining "__metadata" as the topic
> > name,
> > > > and should these sentences refer to "__metadata topic" rather than
> > > > "__metadata log"?  What are the "other nodes in the cluster" that are
> > > > referred to?  These are not controller nodes but brokers, right?  If
> > so,
> > > > then should we say "before we propagate them to the brokers"?
> > Technically
> > > > we have a controller cluster and a broker cluster -- two separate
> > clusters,
> > > > correct?  (Even though we could potentially share JVMs and therefore
> > > > require no additional processes.). If the statement is referring to
> > nodes
> > > > in both clusters then maybe we should state "before we propagate them
> > to
> > > > the other nodes in the controller cluster or to brokers."
> > > >
> > > > "*The controller may have several of these uncommitted changes in
> > flight at
> > > > any given time.  In essence, the controller's in-memory state is
> > always a
> > > > little bit in the future compared to the current state.  This allows
> > the
> > > > controller to continue doing things while it waits for the previous
> > changes
> > > > to be committed to the Raft log.*"  Should the three references above
> > be to
> > > > the active