[jira] [Created] (KAFKA-12977) Eliminate temporary ProducerStateManager in Log recovery logic

[Kowshik Prakasam (Jira)]

Kowshik Prakasam created KAFKA-12977:


 Summary: Eliminate temporary ProducerStateManager in Log recovery 
logic
 Key: KAFKA-12977
 URL: https://issues.apache.org/jira/browse/KAFKA-12977
 Project: Kafka
  Issue Type: Improvement
Reporter: Kowshik Prakasam


The temporary ProducerStateManager (PSM) instance created in the Log recovery 
logic (inside LogLoader) is a source of complexity and confusion. For example, 
when fixing KAFKA-12964 (see [PR# 
10896|https://github.com/apache/kafka/pull/10896]) we figured that there are 
cases where the temporary PSM instance's state goes out of sync with the real 
PSM instance (within LoadLogParams). And we need to adjust the code suitably to 
handle for the consequences of these 2 instances being out of sync. To fix 
this, we should just get rid of the temporary PSM instance which is used in the 
following places:
 # In LogLoader.recoverLog(), we could just pass in the real PSM.
 # In LogLoader.completeSwapOperations(), we try to avoid recovering segment 
here in  [PR #10763|https://github.com/apache/kafka/pull/10763].
 # In LogLoader.loadSegmentFiles(), we probably need to clean this part of the 
logic a bit. If we are missing index file or the index file is corrupted, 
typically we can just rebuild the index without changing PSM. If the segment is 
truncated while rebuilding the index, we actually want to follow the process in 
step 1, by just removing the rest of the segments. So, we could also get rid of 
the temporary PSM in this case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12976) Remove UNSUPPORTED_VERSION error from delete and describe topics calls

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-12976:
--

 Summary: Remove UNSUPPORTED_VERSION error from delete and describe 
topics calls
 Key: KAFKA-12976
 URL: https://issues.apache.org/jira/browse/KAFKA-12976
 Project: Kafka
  Issue Type: Task
Reporter: Justine Olshan
Assignee: Justine Olshan


Originally I thought it would be useful to have an unsupported version error 
returned when the broker's IBP did not support the operation. However, this 
error is transient and in the case of describe topics, it may not be accurate. 
Additionally, unsupported version is not retriable when the scenarios that see 
this likely should be. 

I propose always returning UNKNOWN_TOPIC_ID error when the topic ID is not 
found on the broker. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12975) Consider how Topic IDs can improve consume experience

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-12975:
--

 Summary: Consider how Topic IDs can improve consume experience
 Key: KAFKA-12975
 URL: https://issues.apache.org/jira/browse/KAFKA-12975
 Project: Kafka
  Issue Type: Improvement
Reporter: Justine Olshan
Assignee: Justine Olshan


Currently when a consumer subscribes to a topic, it will continue to consume 
from this topic across topic deletions and recreations with the same name. By 
adding topic IDs to the consumer, we will have more insight for these events. 
We should figure out if we want to change consumer logic now that we have this 
information.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

[Ryanne Dolan]

Omnia, I agree with you that allowing users to specify the whole topic name
via configuration is likely to create problems. MM2 must distinguish
between internal topics from different clusters, and pushing that
complexity into configuration sounds really complicated.

I like the ReplicationPolicy approach.

Ryanne

On Mon, Jun 21, 2021, 2:04 PM Omnia Ibrahim  wrote:

> Any thoughts on this KIP?
>
>
> On Thu, Jun 17, 2021 at 1:38 PM Omnia Ibrahim 
> wrote:
>
> > Another reason why I think adding a configuration for each internal
> topic is not a good solution is how MM2 is naming these topics at the
> moment.
> > Right now MM2 sets the name of the offset-syncs topic to
> mm2-offset-syncs..internal and for checkpoints is
> .checkpoints.internal so the name has a pattern to link it
> back to the herder of source -> target mirror link so having this in
> configuration will lead to
> > 1. having a method that determines the final name of internal topics for
> backward compatibility and have this method to be the default of the
> configuration values. The challenge here is that we need to load first the
> clusters alias to calculate the default value for offset-syncs.topic.name
> and checkpoints.topic.name.
> > 2. Consider use cases where MM2 is used to mirror between multiple
> clusters, for example:
> > source1 -> target.enabled = true
> > source2 -> target.enabled = true
> > For this use-case the current behaviour will create the following
> offset-syncs and checkpoints on each cluster:
> > source1 cluster
> > - mm2-offset-syncs.target.internal
> > source2 cluster
> > - mm2-offset-syncs.target.internal
> > target cluster
> > - source1.checkpoints.internal
> > - source2.checkpoints.internal
> > As MM2 design in the original KIP-382 is spliting internal topics bsed
> on mirroring links. Now if we let MM2 users set the full name of these
> topics as configuration, how will we detect if the user has a wrong
> configuration where they used the same name for checkpoints topic for both
> source1 and source2. How this will work if both source1 and source2
> clusters have consumer groups with same ids as checkpoints topic messages
> contains consumer group id? Should we warn the MM2 user that this topic has
> been used before for another source cluster? If not how will the MM2 user
> notice that?
> >
> >
> > On Mon, Jun 14, 2021 at 5:54 PM Omnia Ibrahim 
> > wrote:
> >
> >> Hi folks, let me try to clarify some of your concerns and questions.
> >>
> >> Mickael: Have you considered making names changeable via configurations?
> >>>
> >>
> >> Gwen: may be missing something, but we are looking at 3 new configs (one
> >>> for each topic). And this rejected alternative is basically identical
> to
> >>> what Connect already does (you can choose names for internal topics
> using
> >>> configs).
> >>>
> >>> These are valid points. The reasons why we should prefer an interface
> >> (the current proposal is using the ReplicationPolicy interface which
> >> already exists in MM2) instead are
> >>
> >> 1. the number of configurations that MM2 has. Right now MM2 has its own
> >> set of configuration in addition to configuration for admin, consumer
> and
> >> producer clients and Connect API. And these configurations in some
> >> use-cases could be different based on the herder.
> >>
> >> Consider a use case where MM2 is used to mirror between a set of
> clusters
> >> running by different teams and have different naming policies. So if we
> are
> >> using 3 configurations for internal topics for a use case like below the
> >> configuration will be like this. If the number of policies grows, the
> >> amount of configuration can get unwieldy.
> >>
> >> clusters = newCenterCluster, teamACluster, teamBCluster, ...
> >>
> >> //newCenterCluster policy is .
> >> //teamACluster naming policy is _ when move to
> newCenterCluster it will be teamA._
> >> //teamBCluster naming policy is . when move to
> newCenterCluster it will be teamB._
> >>
> >> //The goal is to move all topics from team-specific cluster to one new
> cluster
> >> // where the org can unify resource management and naming conventions
> >>
> >> replication.policy.class=MyCustomReplicationPolicy
> >>
> >> teamACluster.heartbeat.topic=mm2_heartbeat_topic // created on source
> cluster
> >>
> teamACluster->newCenterCluster.offsets-sync.topic=mm2_my_offset_sync_topic
> //created on source cluster at the moment
> >>
> teamACluster->newCenterCluster.checkpoints.topic=teamA.mm2_checkpoint_topic
> //created on target cluster which newCenterCluster
> >>
> >> teamBCluster.heartbeat.topic=mm2.heartbeat_topic // created on source
> cluster
> >>
> teamBCluster->newCenterCluster.offsets-sync.topic=mm2.my_offset_sync_topic
> //created on source cluster at the moment
> >>
> teamBCluster->newCenterCluster.checkpoints.topic=teamB.mm2_checkpoint_topic
> //created on target cluster which newCenterCluster
> >>
> >> teamACluster.config.storage.topic=...

[GitHub] [kafka-site] kkonstantine merged pull request #360: Add Konstantine Karantasis as PMC member

[GitBox]

kkonstantine merged pull request #360:
URL: https://github.com/apache/kafka-site/pull/360


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] kkonstantine opened a new pull request #360: Add Konstantine Karantasis as PMC member

[GitBox]

kkonstantine opened a new pull request #360:
URL: https://github.com/apache/kafka-site/pull/360


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Konstantine Karantasis]

Thanks for the update Bruno.
I've moved KIP-698 to the list of postponed KIPs in the plan.

Konstantine

On Mon, Jun 21, 2021 at 2:30 AM Bruno Cadonna  wrote:

> Hi Konstantine,
>
> The implementation of
>
> KIP-698: Add Explicit User Initialization of Broker-side State to Kafka
> Streams
>
> will not be ready for 3.0, so you can remove it from the list.
>
> Best,
> Bruno
>
> On 15.06.21 07:33, Konstantine Karantasis wrote:
> > Done. Moved it into the table of Adopted KIPs targeting 3.0.0 and to the
> > release plan of course.
> > Thanks for catching this Israel.
> >
> > Best,
> > Konstantine
> >
> > On Mon, Jun 14, 2021 at 7:40 PM Israel Ekpo 
> wrote:
> >
> >> Konstantine,
> >>
> >> One of mine is missing from this list
> >>
> >> KIP-633: Drop 24 hour default of grace period in Streams
> >> Please could you include it?
> >>
> >> Voting has already concluded a long time ago
> >>
> >>
> >>
> >> On Mon, Jun 14, 2021 at 6:08 PM Konstantine Karantasis
> >>  wrote:
> >>
> >>> Hi all.
> >>>
> >>> KIP Freeze for the next major release of Apache Kafka was reached last
> >>> week.
> >>>
> >>> As of now, 36 KIPs have concluded their voting process and have been
> >>> adopted.
> >>> These KIPs are targeting 3.0 (unless it's noted otherwise in the
> release
> >>> plan) and their inclusion as new features will be finalized right after
> >>> Feature Freeze.
> >>>
> >>> At the high level, out of these 36 KIPs, 11 have been implemented
> already
> >>> and 25 are open or in progress.
> >>> Here is the full list of adopted KIPs:
> >>>
> >>> KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)
> >>> KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)
> >>> KIP-746: Revise KRaft Metadata Records
> >>> KIP-745: Connect API to restart connector and tasks
> >>> KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with
> >>> internal implementation
> >>> KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics
> >> version
> >>> config
> >>> KIP-741: Change default serde to be null
> >>> KIP-740: Clean up public API in TaskId and fix TaskMetadata#taskId()
> >>> KIP-738: Removal of Connect's internal converter properties
> >>> KIP-734: Improve AdminClient.listOffsets to return timestamp and offset
> >> for
> >>> the record with the largest timestamp
> >>> KIP-733: Change Kafka Streams default replication factor config
> >>> KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> >>> KIP-730: Producer ID generation in KRaft mode
> >>> KIP-726: Make the "cooperative-sticky, range" as the default assignor
> >>> KIP-725: Streamlining configurations for WindowedSerializer and
> >>> WindowedDeserializer.
> >>> KIP-724: Drop support for message formats v0 and v1
> >>> KIP-722: Enable connector client overrides by default
> >>> KIP-721: Enable connector log contexts in Connect Log4j configuration
> >>> KIP-720: Deprecate MirrorMaker v1
> >>> KIP-716: Allow configuring the location of the offset-syncs topic with
> >>> MirrorMaker2
> >>> KIP-715: Expose Committed offset in streams
> >>> KIP-709: Extend OffsetFetch requests to accept multiple group ids.
> >>> KIP-708: Rack awareness for Kafka Streams
> >>> KIP-707: The future of KafkaFuture
> >>> KIP-699: Update FindCoordinator to resolve multiple Coordinators at a
> >> time
> >>> KIP-698: Add Explicit User Initialization of Broker-side State to Kafka
> >>> Streams
> >>> KIP-695: Further Improve Kafka Streams Timestamp Synchronization
> >>> KIP-691: Enhance Transactional Producer Exception Handling
> >>> KIP-679: Producer will enable the strongest delivery guarantee by
> default
> >>> KIP-666: Add Instant-based methods to ReadOnlySessionStore
> >>> KIP-653: Upgrade log4j to log4j2
> >>> KIP-623: Add "internal-topics" option to streams application reset tool
> >>> KIP-622: Add currentSystemTimeMs and currentStreamTimeMs to
> >>> ProcessorContext
> >>> KIP-466: Add support for List serialization and deserialization
> >>> KIP-405: Kafka Tiered Storage
> >>> KIP-390: Support Compression Level
> >>>
> >>> If you notice that a KIP is missing from the list above and should be
> >> part
> >>> of the release plan for 3.0, please reply below.
> >>> The single source of truth remains the official release plan for 3.0,
> >> which
> >>> you may read at any time here:
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> >>>
> >>> Kind reminder that for all the adopted KIPs any required changes to the
> >>> documentation are also part of their respective feature.
> >>>
> >>> For the KIPs that are still in progress, please work closely with your
> >>> reviewers to make sure that the features are stable and land on time
> for
> >>> Feature Freeze.
> >>>
> >>> The next milestone for the Apache Kafka 3.0 release is Feature Freeze
> on
> >>> June 30th, 2021.
> >>>
> >>> Best regards,
> >>> Konstantine
> >>>
> >>> On Fri, Jun 4, 2021 at 3:47 PM Konstantine Karantasis <
> >>> konstant...@confluent.io> wrote:
> >>>
> 
> 

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Konstantine Karantasis]

Thank you Mickael and everyone for the kind replies. I'm honored and
excited to help the project from this role as well.

Best,
Konstantine

On Mon, Jun 21, 2021 at 1:35 PM Bruno Cadonna  wrote:

> Congrats Konstantine!
>
> Best,
> Bruno
>
> On 21.06.21 20:10, Randall Hauch wrote:
> > Congratulations, Konstantine!
> >
> > On Mon, Jun 21, 2021 at 12:39 PM Walker Carlson
> >  wrote:
> >
> >> Congratulations!
> >>
> >> On Mon, Jun 21, 2021 at 12:25 PM Dhruvil Shah
>  >>>
> >> wrote:
> >>
> >>> Congratulations Konstantine! Well deserved!
> >>>
> >>> On Mon, Jun 21, 2021 at 10:20 AM Boyang Chen <
> reluctanthero...@gmail.com
> >>>
> >>> wrote:
> >>>
>  Congratulations Konstantine!
> 
>  On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax 
> >>> wrote:
> 
> > Congrats!
> >
> > On 6/21/21 12:57 PM, Raymond Ng wrote:
> >> Congrats Konstantine!
> >>
> >> /Ray
> >>
> >> On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang 
> > wrote:
> >>
> >>> Congratulations Konstantine!
> >>>
> >>> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley 
> > wrote:
> >>>
>  Congratulations Konstantine!
> 
>  On Mon, Jun 21, 2021 at 5:33 PM David Jacot
> >  
>  wrote:
> 
> > Congrats, Konstantine. Well deserved!
> >
> > Best,
> > David
> >
> > On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
> >>> ramesh.154...@gmail.com
> >
> > wrote:
> >
> >> Congrats Konstantine
> >>
> >> On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison <
>  mimai...@apache.org>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> It's my pleasure to announce that Konstantine Karantasis is
> >> now
> >>> a
> >>> member of the Kafka PMC.
> >>>
> >>> Konstantine has been a Kafka committer since Feb 2020. He has
>  remained
> >>> active in the community since becoming a committer.
> >>>
> >>> Congratulations Konstantine!
> >>>
> >>> Mickael, on behalf of the Apache Kafka PMC
> >>>
> >>
> >
> 
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >
> 
> >>>
> >>
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Bruno Cadonna]

Congrats Konstantine!

Best,
Bruno

On 21.06.21 20:10, Randall Hauch wrote:

Congratulations, Konstantine!

On Mon, Jun 21, 2021 at 12:39 PM Walker Carlson
 wrote:


Congratulations!

On Mon, Jun 21, 2021 at 12:25 PM Dhruvil Shah 


wrote:


Congratulations Konstantine! Well deserved!

On Mon, Jun 21, 2021 at 10:20 AM Boyang Chen 
Congratulations Konstantine!

On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax 

wrote:



Congrats!

On 6/21/21 12:57 PM, Raymond Ng wrote:

Congrats Konstantine!

/Ray

On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang 

wrote:



Congratulations Konstantine!

On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley 

wrote:



Congratulations Konstantine!

On Mon, Jun 21, 2021 at 5:33 PM David Jacot



wrote:


Congrats, Konstantine. Well deserved!

Best,
David

On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <

ramesh.154...@gmail.com


wrote:


Congrats Konstantine

On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison <

mimai...@apache.org>

wrote:


Hi,

It's my pleasure to announce that Konstantine Karantasis is

now

a

member of the Kafka PMC.

Konstantine has been a Kafka committer since Feb 2020. He has

remained

active in the community since becoming a committer.

Congratulations Konstantine!

Mickael, on behalf of the Apache Kafka PMC










--
-- Guozhang

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #242

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-714: Client metrics and observability

[Tom Bentley]

Hi Magnus,

Thanks for the KIP.

1. Did you consider using a `default ClientTelemetryReceiver
clientReceiver() { return null; }` method on the existing MetricsReporter
interface, avoiding the need for the ClientTelemetry trait?
2. On the metrics naming and format, I wasn't really clear about what's
being proposed. I assume we're taking a subset of the existing client
metrics and representing them as OpenTelemetry metrics, but it didn't
really explain how the existing metric names would be mapped to meter and
instrument names. Or did I misunderstand?
3. In the client behaviour section it doesn't explicitly say whether the
client uses a dedicated thread for this work (I assume it does).
4. The description of the FunctionalityNotEnabled error code suggests that
PushTelemetryRequest would only be included in an ApiVersions response if
the broker was configured with a plugin. I think the ApiVersionsResponse is
normally a constant response (not dependent on broker config), so I wonder
whether this is really a precedent we want to set here? Surely in a broker
without a plugin configured it could just return an empty set of
RequestedMetrics and a maxint NextPushMs in the PushTelemetryResponse?
5. Maybe the AcceptedContentTypes should be documented to be in priority
order. That would simplify the action for UnsupportedCompressionType.
6. """As the client will not know the broker id of its bootstrap servers
the broker_id label should be set to “bootstrap”.""" Maybe using the same
convention as is used in the NetworkClient, where bootstrap servers are the
id of the negative of their index in the list?
7. Maybe call it "client.process.rss.bytes" rather than
"client.process.memory.bytes",
to be explicit?
8. It's a little confusing that --id option to kafka-client-metrics.sh can
be a prefix or an exact match. Perhaps --id and --id-prefix would be
clearer.
9. Maybe I missed it, but does the client continue to push metrics to the
same broker as it randomly picked initially? If it gets disconnected from
that broker what happens, does it just randomly pick another?
10. To subscribe to all metrics I assume I can just do
`kafka-client-metrics.sh ... --metric ''`? It might be worth saying this
explicitly. AFAICS this is the only way to find out all the metrics
supported by a client if you don't already know from the client's software
version.

Kind regards,

Tom

On Fri, Jun 18, 2021 at 9:39 PM Travis Bischel 
wrote:

> H Colin (and Magnus),
>
> Thanks for the replies!
>
> I think the biggest concern I have is the cardinality bits. I'm
> sympathetic to the aspect of this making it easier for Kafka brokers to
> understand *every* aspect of the kafka ecoystem. I am not sure this will
> 100% solve the need there, though: if a client is unable to connect to a
> broker, visibility disappears immediately, no?
>
> I do still think that the problem of difficulty of monitoring within an
> organization results from issues within organizations themselves: orgs
> should have proper processes in place such that anything talking to Kafka
> has the org's plug-in monitoring libraries. Kafka operators can define
> those libraries, such that all clients in the org have the libraries the
> operators require. This satisfies the same goals this KIP aims to provide,
> albeit with the increased org cost of not just having something defined to
> be plugged in.
>
> If Kafka operators themselves can which metrics they want, so that the
> broker can tell the client "only send these metrics", then my biggest
> concern is removed.
>
> I do still think that hooks can be a cleaner abstraction to this same
> goal, and then pre-provided libraries (say, "this library provides X,Y,Z
> and sends to prometheus from your client") could exist that more exactly
> satisfy what this KIP aims to provide. This would also avoid the
> kitchen-sink vs. not-comprehensive-enough issue I brought up previously.
> This would also avoid require KIPs for any supported metrics.
>
> On 2021/06/16 22:27:55, "Colin McCabe"  wrote:
> > On Sun, Jun 13, 2021, at 21:51, Travis Bischel wrote:
> > > Hi! I have a few thoughts on this KIP. First, I'd like to thank you
> for
> > > the writeup,
> > > clearly a lot of thought has gone into it and it is very thorough.
> > > However, I'm not
> > > convinced it's the right approach from a fundamental level.
> > >
> > > Fundamentally, this KIP seems like somewhat of a solution to an
> organizational
> > > problem. Metrics are organizational concerns, not Kafka operator
> concerns.
> >
> > Hi Travis,
> >
> > Metrics are certainly Kafka operator concerns. It is very important for
> cluster operators to know things like how many clients there are, what they
> clients are doing, and so forth. This information is needed to administer
> Kafka. Therefore it certainly falls in the domain of the Kafka operations
> team (and the Kafka development team.)
> >
> > We have added many metrics in the past to make it easier to monitor
> clients. I think this is jus

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Bill Bejeck]

Congrats Konstantine!

-Bill

On Mon, Jun 21, 2021 at 12:44 PM Tom Bentley  wrote:

> Congratulations Konstantine!
>
> On Mon, Jun 21, 2021 at 5:33 PM David Jacot 
> wrote:
>
> > Congrats, Konstantine. Well deserved!
> >
> > Best,
> > David
> >
> > On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan  >
> > wrote:
> >
> > > Congrats Konstantine
> > >
> > > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > It's my pleasure to announce that Konstantine Karantasis is now a
> > > > member of the Kafka PMC.
> > > >
> > > > Konstantine has been a Kafka committer since Feb 2020. He has
> remained
> > > > active in the community since becoming a committer.
> > > >
> > > > Congratulations Konstantine!
> > > >
> > > > Mickael, on behalf of the Apache Kafka PMC
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

[Omnia Ibrahim]

Any thoughts on this KIP?


On Thu, Jun 17, 2021 at 1:38 PM Omnia Ibrahim 
wrote:

> Another reason why I think adding a configuration for each internal topic is 
> not a good solution is how MM2 is naming these topics at the moment.
> Right now MM2 sets the name of the offset-syncs topic to 
> mm2-offset-syncs..internal and for checkpoints is 
> .checkpoints.internal so the name has a pattern to link it 
> back to the herder of source -> target mirror link so having this in 
> configuration will lead to
> 1. having a method that determines the final name of internal topics for 
> backward compatibility and have this method to be the default of the 
> configuration values. The challenge here is that we need to load first the 
> clusters alias to calculate the default value for offset-syncs.topic.name and 
> checkpoints.topic.name.
> 2. Consider use cases where MM2 is used to mirror between multiple clusters, 
> for example:
> source1 -> target.enabled = true
> source2 -> target.enabled = true
> For this use-case the current behaviour will create the following 
> offset-syncs and checkpoints on each cluster:
> source1 cluster
> - mm2-offset-syncs.target.internal
> source2 cluster
> - mm2-offset-syncs.target.internal
> target cluster
> - source1.checkpoints.internal
> - source2.checkpoints.internal
> As MM2 design in the original KIP-382 is spliting internal topics bsed on 
> mirroring links. Now if we let MM2 users set the full name of these topics as 
> configuration, how will we detect if the user has a wrong configuration where 
> they used the same name for checkpoints topic for both source1 and source2. 
> How this will work if both source1 and source2 clusters have consumer groups 
> with same ids as checkpoints topic messages contains consumer group id? 
> Should we warn the MM2 user that this topic has been used before for another 
> source cluster? If not how will the MM2 user notice that?
>
>
> On Mon, Jun 14, 2021 at 5:54 PM Omnia Ibrahim 
> wrote:
>
>> Hi folks, let me try to clarify some of your concerns and questions.
>>
>> Mickael: Have you considered making names changeable via configurations?
>>>
>>
>> Gwen: may be missing something, but we are looking at 3 new configs (one
>>> for each topic). And this rejected alternative is basically identical to
>>> what Connect already does (you can choose names for internal topics using
>>> configs).
>>>
>>> These are valid points. The reasons why we should prefer an interface
>> (the current proposal is using the ReplicationPolicy interface which
>> already exists in MM2) instead are
>>
>> 1. the number of configurations that MM2 has. Right now MM2 has its own
>> set of configuration in addition to configuration for admin, consumer and
>> producer clients and Connect API. And these configurations in some
>> use-cases could be different based on the herder.
>>
>> Consider a use case where MM2 is used to mirror between a set of clusters
>> running by different teams and have different naming policies. So if we are
>> using 3 configurations for internal topics for a use case like below the
>> configuration will be like this. If the number of policies grows, the
>> amount of configuration can get unwieldy.
>>
>> clusters = newCenterCluster, teamACluster, teamBCluster, ...
>>
>> //newCenterCluster policy is .
>> //teamACluster naming policy is _ when move to 
>> newCenterCluster it will be teamA._
>> //teamBCluster naming policy is . when move to 
>> newCenterCluster it will be teamB._
>>
>> //The goal is to move all topics from team-specific cluster to one new 
>> cluster
>> // where the org can unify resource management and naming conventions
>>
>> replication.policy.class=MyCustomReplicationPolicy
>>
>> teamACluster.heartbeat.topic=mm2_heartbeat_topic // created on source cluster
>> teamACluster->newCenterCluster.offsets-sync.topic=mm2_my_offset_sync_topic 
>> //created on source cluster at the moment
>> teamACluster->newCenterCluster.checkpoints.topic=teamA.mm2_checkpoint_topic 
>> //created on target cluster which newCenterCluster
>>
>> teamBCluster.heartbeat.topic=mm2.heartbeat_topic // created on source cluster
>> teamBCluster->newCenterCluster.offsets-sync.topic=mm2.my_offset_sync_topic 
>> //created on source cluster at the moment
>> teamBCluster->newCenterCluster.checkpoints.topic=teamB.mm2_checkpoint_topic 
>> //created on target cluster which newCenterCluster
>>
>> teamACluster.config.storage.topic=...
>> teamACluster.offset.storage.topic=...
>> teamACluster.status.storage.topic=...
>>
>> teamBCluster.config.storage.topic=...
>> teamBCluster.offset.storage.topic=...
>> teamBCluster.status.storage.topic=...
>>
>>
>> teamACluster→newCenterCluster.enabled=true
>> teamACluster→newCenterCluster.enabled=true
>>
>> 2. The other reason is what Mickael mentioned regards a future KIP to
>> move offset-syncs on the target cluster.
>>
>>> Mickael: I'm about to open a KIP to specify where the offset-syncs top

Re: [DISCUSS] KIP-747 Add support for basic aggregation APIs

[Mohan Parthasarathy]

Alex,


On Wed, Jun 16, 2021 at 8:07 AM Alexandre Brasil 
wrote:

> Mohan / Mathias,
>
> > > I think extending min/max to non-numeric types makes sense. Wondering
> > > why we should require a `Comparator` or if we should require that the
> > > types implement `Comparable` instead?
> > >
> > Good question. This is what it would look like:
> >
> > KTable min_comparable()
> > KTable min_comparator(Comparator comp)
>
> Not sure if I understood Mathias' proposal correctly, but I think that
> instead of going with
> your original proposal ( KTable min(Function VR> func...)
> or mine (KTable min(Comparator comparator...), we could simplify
> it a
> bit by using a function to extract a Comparable property from the original
> value:
>
> KTable min(Function> func...)
>
> I will let Matthias clarify. I am not sure why it is simpler than the
comparator one. Comparable is implemented by the type and not sure exposing
it this way makes it any better.

> I also think, that min/max should not change the value type. Using
> > `Long` for sum() make sense though, and also to require a ` > Number>`.
>
> Are there any reasons to limit the sum() to integers? Why not use a Double
> instead?
>
> Yeah, if the precision is important, then we should stick with Double.

-mohan

Best regards,
> Alexandre
>
> On Wed, Jun 16, 2021 at 1:01 AM Mohan Parthasarathy 
> wrote:
>
> > Matthias,
> >
> > On Mon, Jun 14, 2021 at 9:18 PM Matthias J. Sax
>  > >
> > wrote:
> >
> > > Hi,
> > >
> > > I think extending min/max to non-numeric types makes sense. Wondering
> > > why we should require a `Comparator` or if we should require that the
> > > types implement `Comparable` instead?
> > >
> > > Good question. This is what it would look like:
> >
> > KTable min_comparable()
> > KTable min_comparator(Comparator comp)
> >
> > For min_comparable to work, you still need the bounds "V extends
> > Comparable<
> > V>". AFAICT, to avoid the "type parameter V hiding the type V" warning,
> it
> > has to be at the interface level like this:
> >
> >  KStream>
> >
> > which is a little messy unless there is a different way to do the same.
> The
> > comparator gives a simple way to define an anonymous function.
> >
> > What do you think ?
> >
> >
> > > I also think, that min/max should not change the value type. Using
> > > `Long` for sum() make sense though, and also to require a ` > > Number>`.
> > >
> > > I guess these are the two possibilities:
> >
> >  Long sum(Function func)
> > Long sum(Function func)
> >
> > Both should work. "func" can return any subtypes of Number and I don't
> see
> > any advantages with the first version. Can you clarify ?
> >
> > Thanks
> > Mohan
> >
> >
> > >
> > > -Matthias
> > >
> > > On 6/8/21 5:00 PM, Mohan Parthasarathy wrote:
> > > > Hi Alex,
> > > >
> > > > On Tue, Jun 8, 2021 at 2:44 PM Alexandre Brasil <
> > > alexandre.bra...@gmail.com>
> > > > wrote:
> > > >
> > > >>
> > > >> My point here is that, when we're only interested in a max/min
> numeric
> > > >> value, it doesn't
> > > >> matter when we have repeated values, since we'd be only forwarding
> the
> > > >> number downstream,
> > > >> so I could disregard when the Comparator returns a zero value
> (meaning
> > > >> equals) and min/max
> > > >> would still be semantically correct. But when we're forwarding the
> > > original
> > > >> object downstream
> > > >> instead of its numeric property, it could mean different things
> > > >> semantically depending on how
> > > >> we handle the repeated values.
> > > >>
> > > >> As an example, if I were using max() on a stream of Biddings for
> > > products
> > > >> in an auction, the
> > > >> order of the biddings would probably influence the winner if two
> > clients
> > > >> send Biddings with the
> > > >> same value. If we're only forwarding the Bidding value downstream (a
> > > double
> > > >> value of 100, for
> > > >> example), it doesn't matter how repeated values are handled, since
> the
> > > max
> > > >> price for this
> > > >> auction would still be 100.00, no matter what Bidding got selected
> in
> > > the
> > > >> end. But if we're
> > > >> forwarding the Biddings downstream instead, then it matters whether
> > the
> > > >> winning Bidding sent
> > > >> downstream was originally posted by Client A or Client B.
> > > >>
> > > >> I'm not saying that an overloaded method to handle different options
> > for
> > > >> how repeated values
> > > >> should be handled by min/max is mandatory, but it should be clear on
> > the
> > > >> methods' docs
> > > >> what would happen when Comparator.compare() == 0. My preferred
> option
> > > for
> > > >> the default
> > > >> behaviour is to only forward a new value is smaller/bigger than the
> > > >> previous min/max value
> > > >> (ignoring compare() == 0), since it would emit less values
> downstream
> > > and
> > > >> would be easier
> > > >> to read ("I only send a value downstream if it's bigger/smaller than
> > the
> > > >> previously selected
> > > >> value").
> > > >>
> > 

[jira] [Resolved] (KAFKA-12515) ApiVersionManager should create response based on request version

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-12515.
-
Fix Version/s: 3.0.0
   Resolution: Fixed

This issue was fixed by making feature support ignorable in the ApiVersion 
response.

> ApiVersionManager should create response based on request version
> -
>
> Key: KAFKA-12515
> URL: https://issues.apache.org/jira/browse/KAFKA-12515
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Haoran Xuan
>Assignee: Haoran Xuan
>Priority: Minor
> Fix For: 3.0.0
>
>
> *Background:*
> Currently, `ApiVersionManager` generates `ApiVersionResponse` without 
> considering the version of `ApiVersionRequest`, it will return non-empty 
> `SupportedFeatures`, 
> `FinalizedFeaturesEpoch` and `FinalizedFeatures` if 
> `BrokerFeatures.createDefault` is not empty for older client who can not 
> understand it, which causes problem.
> To reproduce this probelm, we can simple modify the 
> `BrokerFeatures.createDefault` as:
> {code:java}
> def createDefault(): BrokerFeatures =
> { val featureMap = Map("Sample" -> new SupportedVersionRange(1,3)).asJava val 
> supportedFeatures = Features.supportedFeatures(featureMap) // The arguments 
> are currently empty, but, in the future as we define features we should // 
> populate the required values here. new BrokerFeatures(supportedFeatures) }
> {code}
>  
> And the `ApiVersionsRequestTest.testApiVersionsRequestValidationV0` will fail 
> with as:
> {code:java}
>  
> [2021-03-21 11:05:53,726] ERROR [KafkaApi-0] Unexpected error handling 
> request RequestHeader(apiKey=API_VERSIONS, apiVersion=0, clientId=client-id, 
> correlationId=2) – ApiVersionsRequestData(clientSoftwareName='', 
> clientSoftwareVersion='') with context 
> RequestContext(header=RequestHeader(apiKey=API_VERSIONS, apiVersion=0, 
> clientId=client-id, correlationId=2), 
> connectionId='127.0.0.1:59159-127.0.0.1:59161-0', clientAddress=/127.0.0.1, 
> principal=User:ANONYMOUS, listenerName=ListenerName(PLAINTEXT), 
> securityProtocol=PLAINTEXT, 
> clientInformation=ClientInformation(softwareName=unknown, 
> softwareVersion=unknown), fromPrivilegedListener=false, 
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@78db65f])
>  (kafka.server.KafkaApis:76)[2021-03-21 11:05:53,726] ERROR [KafkaApi-0] 
> Unexpected error handling request RequestHeader(apiKey=API_VERSIONS, 
> apiVersion=0, clientId=client-id, correlationId=2) – 
> ApiVersionsRequestData(clientSoftwareName='', clientSoftwareVersion='') with 
> context RequestContext(header=RequestHeader(apiKey=API_VERSIONS, 
> apiVersion=0, clientId=client-id, correlationId=2), 
> connectionId='127.0.0.1:59159-127.0.0.1:59161-0', clientAddress=/127.0.0.1, 
> principal=User:ANONYMOUS, listenerName=ListenerName(PLAINTEXT), 
> securityProtocol=PLAINTEXT, 
> clientInformation=ClientInformation(softwareName=unknown, 
> softwareVersion=unknown), fromPrivilegedListener=false, 
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@78db65f])
>  
> (kafka.server.KafkaApis:76)org.apache.kafka.common.errors.UnsupportedVersionException:
>  Attempted to write a non-default supportedFeatures at version 0
>  org.opentest4j.AssertionFailedError: API keys in ApiVersionsResponse must 
> match API keys supported by broker. ==> Expected :58Actual   :1 difference>
> at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at 
> org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62) at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150) at 
> org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:542) at 
> kafka.server.AbstractApiVersionsRequestTest.validateApiVersionsResponse(AbstractApiVersionsRequestTest.scala:61)
>  at 
> kafka.server.ApiVersionsRequestTest.testApiVersionsRequestValidationV0(ApiVersionsRequestTest.scala:69)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
>  at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>  at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
>  at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercep

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Randall Hauch]

Congratulations, Konstantine!

On Mon, Jun 21, 2021 at 12:39 PM Walker Carlson
 wrote:

> Congratulations!
>
> On Mon, Jun 21, 2021 at 12:25 PM Dhruvil Shah  >
> wrote:
>
> > Congratulations Konstantine! Well deserved!
> >
> > On Mon, Jun 21, 2021 at 10:20 AM Boyang Chen  >
> > wrote:
> >
> > > Congratulations Konstantine!
> > >
> > > On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax 
> > wrote:
> > >
> > > > Congrats!
> > > >
> > > > On 6/21/21 12:57 PM, Raymond Ng wrote:
> > > > > Congrats Konstantine!
> > > > >
> > > > > /Ray
> > > > >
> > > > > On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang 
> > > > wrote:
> > > > >
> > > > >> Congratulations Konstantine!
> > > > >>
> > > > >> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley 
> > > > wrote:
> > > > >>
> > > > >>> Congratulations Konstantine!
> > > > >>>
> > > > >>> On Mon, Jun 21, 2021 at 5:33 PM David Jacot
> > > >  > > > >>>
> > > > >>> wrote:
> > > > >>>
> > > >  Congrats, Konstantine. Well deserved!
> > > > 
> > > >  Best,
> > > >  David
> > > > 
> > > >  On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
> > > > >> ramesh.154...@gmail.com
> > > > 
> > > >  wrote:
> > > > 
> > > > > Congrats Konstantine
> > > > >
> > > > > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison <
> > > mimai...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> It's my pleasure to announce that Konstantine Karantasis is
> now
> > a
> > > > >> member of the Kafka PMC.
> > > > >>
> > > > >> Konstantine has been a Kafka committer since Feb 2020. He has
> > > > >>> remained
> > > > >> active in the community since becoming a committer.
> > > > >>
> > > > >> Congratulations Konstantine!
> > > > >>
> > > > >> Mickael, on behalf of the Apache Kafka PMC
> > > > >>
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> -- Guozhang
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Walker Carlson]

Congratulations!

On Mon, Jun 21, 2021 at 12:25 PM Dhruvil Shah 
wrote:

> Congratulations Konstantine! Well deserved!
>
> On Mon, Jun 21, 2021 at 10:20 AM Boyang Chen 
> wrote:
>
> > Congratulations Konstantine!
> >
> > On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax 
> wrote:
> >
> > > Congrats!
> > >
> > > On 6/21/21 12:57 PM, Raymond Ng wrote:
> > > > Congrats Konstantine!
> > > >
> > > > /Ray
> > > >
> > > > On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang 
> > > wrote:
> > > >
> > > >> Congratulations Konstantine!
> > > >>
> > > >> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley 
> > > wrote:
> > > >>
> > > >>> Congratulations Konstantine!
> > > >>>
> > > >>> On Mon, Jun 21, 2021 at 5:33 PM David Jacot
> > >  > > >>>
> > > >>> wrote:
> > > >>>
> > >  Congrats, Konstantine. Well deserved!
> > > 
> > >  Best,
> > >  David
> > > 
> > >  On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
> > > >> ramesh.154...@gmail.com
> > > 
> > >  wrote:
> > > 
> > > > Congrats Konstantine
> > > >
> > > > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison <
> > mimai...@apache.org>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> It's my pleasure to announce that Konstantine Karantasis is now
> a
> > > >> member of the Kafka PMC.
> > > >>
> > > >> Konstantine has been a Kafka committer since Feb 2020. He has
> > > >>> remained
> > > >> active in the community since becoming a committer.
> > > >>
> > > >> Congratulations Konstantine!
> > > >>
> > > >> Mickael, on behalf of the Apache Kafka PMC
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> -- Guozhang
> > > >>
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-12974) Change the default for snapshot generation configuration

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-12974:
--

 Summary: Change the default for snapshot generation configuration
 Key: KAFKA-12974
 URL: https://issues.apache.org/jira/browse/KAFKA-12974
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jose Armando Garcia Sancio


In PR https://github.com/apache/kafka/pull/10812 the default for the 
{metadata.log.snapshot.min.new_record.bytes} is set to {Int.MaxValue}. This was 
done to disable snapshot generation by default since snapshot loading is not 
implemented on the broker.

This value should be changed to something much smaller.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Dhruvil Shah]

Congratulations Konstantine! Well deserved!

On Mon, Jun 21, 2021 at 10:20 AM Boyang Chen 
wrote:

> Congratulations Konstantine!
>
> On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax  wrote:
>
> > Congrats!
> >
> > On 6/21/21 12:57 PM, Raymond Ng wrote:
> > > Congrats Konstantine!
> > >
> > > /Ray
> > >
> > > On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang 
> > wrote:
> > >
> > >> Congratulations Konstantine!
> > >>
> > >> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley 
> > wrote:
> > >>
> > >>> Congratulations Konstantine!
> > >>>
> > >>> On Mon, Jun 21, 2021 at 5:33 PM David Jacot
> >  > >>>
> > >>> wrote:
> > >>>
> >  Congrats, Konstantine. Well deserved!
> > 
> >  Best,
> >  David
> > 
> >  On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
> > >> ramesh.154...@gmail.com
> > 
> >  wrote:
> > 
> > > Congrats Konstantine
> > >
> > > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison <
> mimai...@apache.org>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> It's my pleasure to announce that Konstantine Karantasis is now a
> > >> member of the Kafka PMC.
> > >>
> > >> Konstantine has been a Kafka committer since Feb 2020. He has
> > >>> remained
> > >> active in the community since becoming a committer.
> > >>
> > >> Congratulations Konstantine!
> > >>
> > >> Mickael, on behalf of the Apache Kafka PMC
> > >>
> > >
> > 
> > >>>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Boyang Chen]

Congratulations Konstantine!

On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax  wrote:

> Congrats!
>
> On 6/21/21 12:57 PM, Raymond Ng wrote:
> > Congrats Konstantine!
> >
> > /Ray
> >
> > On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang 
> wrote:
> >
> >> Congratulations Konstantine!
> >>
> >> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley 
> wrote:
> >>
> >>> Congratulations Konstantine!
> >>>
> >>> On Mon, Jun 21, 2021 at 5:33 PM David Jacot
>  >>>
> >>> wrote:
> >>>
>  Congrats, Konstantine. Well deserved!
> 
>  Best,
>  David
> 
>  On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
> >> ramesh.154...@gmail.com
> 
>  wrote:
> 
> > Congrats Konstantine
> >
> > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison 
> > wrote:
> >
> >> Hi,
> >>
> >> It's my pleasure to announce that Konstantine Karantasis is now a
> >> member of the Kafka PMC.
> >>
> >> Konstantine has been a Kafka committer since Feb 2020. He has
> >>> remained
> >> active in the community since becoming a committer.
> >>
> >> Congratulations Konstantine!
> >>
> >> Mickael, on behalf of the Apache Kafka PMC
> >>
> >
> 
> >>>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Matthias J. Sax]

Congrats!

On 6/21/21 12:57 PM, Raymond Ng wrote:
> Congrats Konstantine!
> 
> /Ray
> 
> On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang  wrote:
> 
>> Congratulations Konstantine!
>>
>> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley  wrote:
>>
>>> Congratulations Konstantine!
>>>
>>> On Mon, Jun 21, 2021 at 5:33 PM David Jacot >>
>>> wrote:
>>>
 Congrats, Konstantine. Well deserved!

 Best,
 David

 On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
>> ramesh.154...@gmail.com

 wrote:

> Congrats Konstantine
>
> On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison 
> wrote:
>
>> Hi,
>>
>> It's my pleasure to announce that Konstantine Karantasis is now a
>> member of the Kafka PMC.
>>
>> Konstantine has been a Kafka committer since Feb 2020. He has
>>> remained
>> active in the community since becoming a committer.
>>
>> Congratulations Konstantine!
>>
>> Mickael, on behalf of the Apache Kafka PMC
>>
>

>>>
>>
>>
>> --
>> -- Guozhang
>>
> 

[jira] [Created] (KAFKA-12973) Update KIP and dev mailing list

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-12973:
--

 Summary: Update KIP and dev mailing list
 Key: KAFKA-12973
 URL: https://issues.apache.org/jira/browse/KAFKA-12973
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jose Armando Garcia Sancio


Update KIP-630 and the Kafka mailing list based on the small implementation 
deviations from what is documented in the KIP.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Raymond Ng]

Congrats Konstantine!

/Ray

On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang  wrote:

> Congratulations Konstantine!
>
> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley  wrote:
>
> > Congratulations Konstantine!
> >
> > On Mon, Jun 21, 2021 at 5:33 PM David Jacot  >
> > wrote:
> >
> > > Congrats, Konstantine. Well deserved!
> > >
> > > Best,
> > > David
> > >
> > > On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
> ramesh.154...@gmail.com
> > >
> > > wrote:
> > >
> > > > Congrats Konstantine
> > > >
> > > > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > It's my pleasure to announce that Konstantine Karantasis is now a
> > > > > member of the Kafka PMC.
> > > > >
> > > > > Konstantine has been a Kafka committer since Feb 2020. He has
> > remained
> > > > > active in the community since becoming a committer.
> > > > >
> > > > > Congratulations Konstantine!
> > > > >
> > > > > Mickael, on behalf of the Apache Kafka PMC
> > > > >
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Guozhang Wang]

Congratulations Konstantine!

On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley  wrote:

> Congratulations Konstantine!
>
> On Mon, Jun 21, 2021 at 5:33 PM David Jacot 
> wrote:
>
> > Congrats, Konstantine. Well deserved!
> >
> > Best,
> > David
> >
> > On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan  >
> > wrote:
> >
> > > Congrats Konstantine
> > >
> > > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > It's my pleasure to announce that Konstantine Karantasis is now a
> > > > member of the Kafka PMC.
> > > >
> > > > Konstantine has been a Kafka committer since Feb 2020. He has
> remained
> > > > active in the community since becoming a committer.
> > > >
> > > > Congratulations Konstantine!
> > > >
> > > > Mickael, on behalf of the Apache Kafka PMC
> > > >
> > >
> >
>


-- 
-- Guozhang

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Tom Bentley]

Congratulations Konstantine!

On Mon, Jun 21, 2021 at 5:33 PM David Jacot 
wrote:

> Congrats, Konstantine. Well deserved!
>
> Best,
> David
>
> On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan 
> wrote:
>
> > Congrats Konstantine
> >
> > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison 
> > wrote:
> >
> > > Hi,
> > >
> > > It's my pleasure to announce that Konstantine Karantasis is now a
> > > member of the Kafka PMC.
> > >
> > > Konstantine has been a Kafka committer since Feb 2020. He has remained
> > > active in the community since becoming a committer.
> > >
> > > Congratulations Konstantine!
> > >
> > > Mickael, on behalf of the Apache Kafka PMC
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[David Jacot]

Congrats, Konstantine. Well deserved!

Best,
David

On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan 
wrote:

> Congrats Konstantine
>
> On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison 
> wrote:
>
> > Hi,
> >
> > It's my pleasure to announce that Konstantine Karantasis is now a
> > member of the Kafka PMC.
> >
> > Konstantine has been a Kafka committer since Feb 2020. He has remained
> > active in the community since becoming a committer.
> >
> > Congratulations Konstantine!
> >
> > Mickael, on behalf of the Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Ramesh Krishnan]

Congrats Konstantine

On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison  wrote:

> Hi,
>
> It's my pleasure to announce that Konstantine Karantasis is now a
> member of the Kafka PMC.
>
> Konstantine has been a Kafka committer since Feb 2020. He has remained
> active in the community since becoming a committer.
>
> Congratulations Konstantine!
>
> Mickael, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Israel Ekpo]

Congratulations Konstantine and thank you for your service!

On Mon, Jun 21, 2021 at 11:28 AM Mickael Maison  wrote:

> Hi,
>
> It's my pleasure to announce that Konstantine Karantasis is now a
> member of the Kafka PMC.
>
> Konstantine has been a Kafka committer since Feb 2020. He has remained
> active in the community since becoming a committer.
>
> Congratulations Konstantine!
>
> Mickael, on behalf of the Apache Kafka PMC
>

[ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Mickael Maison]

Hi,

It's my pleasure to announce that Konstantine Karantasis is now a
member of the Kafka PMC.

Konstantine has been a Kafka committer since Feb 2020. He has remained
active in the community since becoming a committer.

Congratulations Konstantine!

Mickael, on behalf of the Apache Kafka PMC

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Dongjin Lee]

Hi Guozhang,

Thanks for the comment. I updated the KIP applying your comment.

@All

Any other thoughts?

Regards,
Dongjin

On Tue, Jun 15, 2021 at 1:40 AM Guozhang Wang  wrote:

> If we believe this tool does not work anymore and there's other ways to
> achieve the intended function, then we should remove it in the next
> release; otherwise, I think this KIP still is worthy. In any ways, we
> should not left a cmd tool not maintained but not removed either.
>
> Guozhang
>
> On Thu, Jun 10, 2021 at 10:05 PM Dongjin Lee  wrote:
>
> > Hi Ismael,
> >
> > > I am not convinced this tool is actually useful, I haven't seen anyone
> > using it in years.
> >
> > Sure, you may right indeed. The `ReplicaVerificationTool` may not be so
> > useful.[^0] However, I hope to propose another perspective.
> >
> > As long as this tool is provided with a launcher script in a
> distribution,
> > its command-line parameters look so weird to the users since it breaks
> > consistency. It is even worse with KIP-499
> > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=123899170
> > >[^1],
> > which tries to unify the command line parameters and deprecate old ones -
> > even the tools without launcher script (e.g., VerifiableLog4jAppender)
> now
> > uses `--bootstrap-server` parameter. This situation is rather odd, isn't
> > it?
> >
> > This improvement may not have a great value, but it may reduce
> awkwardness
> > from the user's viewpoint.
> >
> > Best,
> > Dongjin
> >
> > [^0]: With my personal experience, I used it to validate the replication
> > when working with a client so sensitive to replication missing, like a
> > Semiconductor manufacturing company.
> > [^1]: Somewhat strange, two omitted tools from KIP-499
> > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=123899170
> > >
> > all have their own launcher script.
> >
> > On Thu, Jun 10, 2021 at 2:02 PM Ismael Juma  wrote:
> >
> > > KAFKA-12600 was a general change, not related to this tool
> specifically.
> > I
> > > am not convinced this tool is actually useful, I haven't seen anyone
> > using
> > > it in years.
> > >
> > > Ismael
> > >
> > > On Wed, Jun 9, 2021 at 9:51 PM Dongjin Lee  wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > Before I submit this KIP, I reviewed some history. When KIP-499
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool
> > > > >
> > > > tried to resolve the inconsistencies between the command line tools,
> > two
> > > > tools were omitted, probably by mistake.
> > > >
> > > > - KAFKA-12878: Support --bootstrap-server
> > kafka-streams-application-reset
> > > > 
> > > > - KAFKA-12899: Support --bootstrap-server in ReplicaVerificationTool
> > > >  (this one)
> > > >
> > > > And it seems like this tool is still working. The last update was
> > > > KAFKA-12600  by
> > you,
> > > > which will also be included in this 3.0.0 release. It is why I
> > determined
> > > > that this tool is worth updating.
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > On Thu, Jun 10, 2021 at 1:26 PM Ismael Juma 
> wrote:
> > > >
> > > > > Hi Dongjin,
> > > > >
> > > > > Does this tool still work? I recall that there were some doubts
> about
> > > it
> > > > > and that's why it wasn't updated previously.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Sat, Jun 5, 2021 at 2:38 PM Dongjin Lee 
> > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to call for a vote on KIP-752: Support
> --bootstrap-server
> > in
> > > > > > ReplicaVerificationTool:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > > > > >
> > > > > > Best,
> > > > > > Dongjin
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > >
> > > > > >
> > > > > >
> > > > > > *github:  github.com/dongjinleekr
> > > > > > keybase:
> > > > > https://keybase.io/dongjinleekr
> > > > > > linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > speakerdeck:
> > > > > > speakerdeck.com/dongjin
> > > > > > *
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > >
> > > >
> > > > *github:  github.com/dongjinleekr
> > > > keybase:
> > > https://keybase.io/dongjinleekr
> > > > linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > 

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Bruno Cadonna]

Hi Konstantine,

The implementation of

KIP-698: Add Explicit User Initialization of Broker-side State to Kafka
Streams

will not be ready for 3.0, so you can remove it from the list.

Best,
Bruno

On 15.06.21 07:33, Konstantine Karantasis wrote:

Done. Moved it into the table of Adopted KIPs targeting 3.0.0 and to the
release plan of course.
Thanks for catching this Israel.

Best,
Konstantine

On Mon, Jun 14, 2021 at 7:40 PM Israel Ekpo  wrote:


Konstantine,

One of mine is missing from this list

KIP-633: Drop 24 hour default of grace period in Streams
Please could you include it?

Voting has already concluded a long time ago



On Mon, Jun 14, 2021 at 6:08 PM Konstantine Karantasis
 wrote:


Hi all.

KIP Freeze for the next major release of Apache Kafka was reached last
week.

As of now, 36 KIPs have concluded their voting process and have been
adopted.
These KIPs are targeting 3.0 (unless it's noted otherwise in the release
plan) and their inclusion as new features will be finalized right after
Feature Freeze.

At the high level, out of these 36 KIPs, 11 have been implemented already
and 25 are open or in progress.
Here is the full list of adopted KIPs:

KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)
KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)
KIP-746: Revise KRaft Metadata Records
KIP-745: Connect API to restart connector and tasks
KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with
internal implementation
KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics

version

config
KIP-741: Change default serde to be null
KIP-740: Clean up public API in TaskId and fix TaskMetadata#taskId()
KIP-738: Removal of Connect's internal converter properties
KIP-734: Improve AdminClient.listOffsets to return timestamp and offset

for

the record with the largest timestamp
KIP-733: Change Kafka Streams default replication factor config
KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
KIP-730: Producer ID generation in KRaft mode
KIP-726: Make the "cooperative-sticky, range" as the default assignor
KIP-725: Streamlining configurations for WindowedSerializer and
WindowedDeserializer.
KIP-724: Drop support for message formats v0 and v1
KIP-722: Enable connector client overrides by default
KIP-721: Enable connector log contexts in Connect Log4j configuration
KIP-720: Deprecate MirrorMaker v1
KIP-716: Allow configuring the location of the offset-syncs topic with
MirrorMaker2
KIP-715: Expose Committed offset in streams
KIP-709: Extend OffsetFetch requests to accept multiple group ids.
KIP-708: Rack awareness for Kafka Streams
KIP-707: The future of KafkaFuture
KIP-699: Update FindCoordinator to resolve multiple Coordinators at a

time

KIP-698: Add Explicit User Initialization of Broker-side State to Kafka
Streams
KIP-695: Further Improve Kafka Streams Timestamp Synchronization
KIP-691: Enhance Transactional Producer Exception Handling
KIP-679: Producer will enable the strongest delivery guarantee by default
KIP-666: Add Instant-based methods to ReadOnlySessionStore
KIP-653: Upgrade log4j to log4j2
KIP-623: Add "internal-topics" option to streams application reset tool
KIP-622: Add currentSystemTimeMs and currentStreamTimeMs to
ProcessorContext
KIP-466: Add support for List serialization and deserialization
KIP-405: Kafka Tiered Storage
KIP-390: Support Compression Level

If you notice that a KIP is missing from the list above and should be

part

of the release plan for 3.0, please reply below.
The single source of truth remains the official release plan for 3.0,

which

you may read at any time here:

https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0

Kind reminder that for all the adopted KIPs any required changes to the
documentation are also part of their respective feature.

For the KIPs that are still in progress, please work closely with your
reviewers to make sure that the features are stable and land on time for
Feature Freeze.

The next milestone for the Apache Kafka 3.0 release is Feature Freeze on
June 30th, 2021.

Best regards,
Konstantine

On Fri, Jun 4, 2021 at 3:47 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:



Hi all,

Just a quick reminder that KIP Freeze is next Wednesday, June 9th.
A vote thread needs to be open for at least 72 hours, so to everyone

that

is working hard on proposals targeting 3.0.0, please make sure that

your

[VOTE] threads are started on time.

Best,
Konstantine


On Wed, May 26, 2021 at 8:10 PM Israel Ekpo 

wrote:



+1 on the new schedule.

On Wed, May 26, 2021 at 8:14 PM Sophie Blee-Goldman
 wrote:


Ah ok, thanks Konstantine. I won't bug you about every new KIP that

comes

in between now and KIP Freeze :P

+1 on the scheduling changes as well

On Wed, May 26, 2021 at 4:00 PM David Arthur 

wrote:



The new schedule looks good to me, +1

On Wed, May 26, 2021 at 6:29 PM Ismael Juma 

wrote:



Thanks Konstantine, +1 from me.

Ismael

On Wed, May 26, 202