Re: [VOTE] KIP-773 Differentiate consistently metric latency measured in millis and nanos

[Sagar]

Looks good to me.

+1(non-binding)

Sagar.

On Sat, 4 Sep 2021 at 8:15 AM, Luke Chen  wrote:

> Thanks for the KIP.
>
> +1 (non-binding)
>
> Thanks.
> Luke
>
> On Sat, Sep 4, 2021 at 10:32 AM Guozhang Wang  wrote:
>
> > Thanks Josep,
> >
> > Took a look at the KIP, LGTM.
> >
> > On Fri, Sep 3, 2021 at 11:25 AM Josep Prat 
> > wrote:
> >
> > > Hi there,
> > >
> > > Since it's a rather small KIP, I'd like to start a vote for the KIP-773
> > > Differentiate consistently metric latency measured  in millis and
> nanos.
> > > The KIP page The KIP can be found at:
> > > https://cwiki.apache.org/confluence/x/ZwNACw
> > >
> > > Thanks in advance,
> > >
> > > --
> > >
> > > Josep Prat
> > >
> > > *Aiven Deutschland GmbH*
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > *m:* +491715557497
> > >
> > > *w:* aiven.io
> > >
> > > *e:* josep.p...@aiven.io
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] KIP-773 Differentiate consistently metric latency measured in millis and nanos

[Luke Chen]

Thanks for the KIP.

+1 (non-binding)

Thanks.
Luke

On Sat, Sep 4, 2021 at 10:32 AM Guozhang Wang  wrote:

> Thanks Josep,
>
> Took a look at the KIP, LGTM.
>
> On Fri, Sep 3, 2021 at 11:25 AM Josep Prat 
> wrote:
>
> > Hi there,
> >
> > Since it's a rather small KIP, I'd like to start a vote for the KIP-773
> > Differentiate consistently metric latency measured  in millis and nanos.
> > The KIP page The KIP can be found at:
> > https://cwiki.apache.org/confluence/x/ZwNACw
> >
> > Thanks in advance,
> >
> > --
> >
> > Josep Prat
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491715557497
> >
> > *w:* aiven.io
> >
> > *e:* josep.p...@aiven.io
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-773 Differentiate consistently metric latency measured in millis and nanos

[Guozhang Wang]

Thanks Josep,

Took a look at the KIP, LGTM.

On Fri, Sep 3, 2021 at 11:25 AM Josep Prat 
wrote:

> Hi there,
>
> Since it's a rather small KIP, I'd like to start a vote for the KIP-773
> Differentiate consistently metric latency measured  in millis and nanos.
> The KIP page The KIP can be found at:
> https://cwiki.apache.org/confluence/x/ZwNACw
>
> Thanks in advance,
>
> --
>
> Josep Prat
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491715557497
>
> *w:* aiven.io
>
> *e:* josep.p...@aiven.io
>


-- 
-- Guozhang

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #446

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 488159 lines...]
[2021-09-03T20:27:08.471Z] 
[2021-09-03T20:27:08.471Z] PlaintextConsumerTest > 
testConsumeMessagesWithCreateTime() PASSED
[2021-09-03T20:27:08.471Z] 
[2021-09-03T20:27:08.471Z] PlaintextConsumerTest > testAsyncCommit() STARTED
[2021-09-03T20:27:11.559Z] 
[2021-09-03T20:27:11.559Z] PlaintextConsumerTest > testAsyncCommit() PASSED
[2021-09-03T20:27:11.559Z] 
[2021-09-03T20:27:11.559Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() STARTED
[2021-09-03T20:27:43.221Z] 
[2021-09-03T20:27:43.221Z] PlaintextConsumerTest > 
testLowMaxFetchSizeForRequestAndPartition() PASSED
[2021-09-03T20:27:43.221Z] 
[2021-09-03T20:27:43.221Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() STARTED
[2021-09-03T20:27:59.916Z] 
[2021-09-03T20:27:59.916Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnStopPolling() PASSED
[2021-09-03T20:27:59.916Z] 
[2021-09-03T20:27:59.916Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() STARTED
[2021-09-03T20:28:04.667Z] 
[2021-09-03T20:28:04.667Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInRevocation() PASSED
[2021-09-03T20:28:04.667Z] 
[2021-09-03T20:28:04.667Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() STARTED
[2021-09-03T20:28:12.848Z] 
[2021-09-03T20:28:12.848Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithAssign() PASSED
[2021-09-03T20:28:12.848Z] 
[2021-09-03T20:28:12.848Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() STARTED
[2021-09-03T20:28:16.078Z] 
[2021-09-03T20:28:16.078Z] PlaintextConsumerTest > 
testPartitionsForInvalidTopic() PASSED
[2021-09-03T20:28:16.078Z] 
[2021-09-03T20:28:16.078Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() STARTED
[2021-09-03T20:28:21.048Z] 
[2021-09-03T20:28:21.048Z] PlaintextConsumerTest > 
testPauseStateNotPreservedByRebalance() PASSED
[2021-09-03T20:28:21.048Z] 
[2021-09-03T20:28:21.048Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() STARTED
[2021-09-03T20:28:26.278Z] 
[2021-09-03T20:28:26.278Z] PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst() PASSED
[2021-09-03T20:28:26.278Z] 
[2021-09-03T20:28:26.278Z] PlaintextConsumerTest > testSeek() STARTED
[2021-09-03T20:28:35.716Z] 
[2021-09-03T20:28:35.716Z] PlaintextConsumerTest > testSeek() PASSED
[2021-09-03T20:28:35.716Z] 
[2021-09-03T20:28:35.716Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() STARTED
[2021-09-03T20:28:44.715Z] 
[2021-09-03T20:28:44.715Z] PlaintextConsumerTest > 
testConsumingWithNullGroupId() PASSED
[2021-09-03T20:28:44.715Z] 
[2021-09-03T20:28:44.716Z] PlaintextConsumerTest > testPositionAndCommit() 
STARTED
[2021-09-03T20:28:49.837Z] 
[2021-09-03T20:28:49.837Z] PlaintextConsumerTest > testPositionAndCommit() 
PASSED
[2021-09-03T20:28:49.837Z] 
[2021-09-03T20:28:49.837Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() STARTED
[2021-09-03T20:28:55.706Z] 
[2021-09-03T20:28:55.706Z] PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes() PASSED
[2021-09-03T20:28:55.706Z] 
[2021-09-03T20:28:55.706Z] PlaintextConsumerTest > testUnsubscribeTopic() 
STARTED
[2021-09-03T20:29:00.826Z] 
[2021-09-03T20:29:00.826Z] PlaintextConsumerTest > testUnsubscribeTopic() PASSED
[2021-09-03T20:29:00.826Z] 
[2021-09-03T20:29:00.826Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() STARTED
[2021-09-03T20:29:11.462Z] 
[2021-09-03T20:29:11.462Z] PlaintextConsumerTest > 
testMultiConsumerSessionTimeoutOnClose() PASSED
[2021-09-03T20:29:11.462Z] 
[2021-09-03T20:29:11.462Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() STARTED
[2021-09-03T20:29:37.991Z] 
[2021-09-03T20:29:37.991Z] PlaintextConsumerTest > 
testMultiConsumerStickyAssignor() PASSED
[2021-09-03T20:29:37.991Z] 
[2021-09-03T20:29:37.991Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() STARTED
[2021-09-03T20:29:41.854Z] 
[2021-09-03T20:29:41.854Z] PlaintextConsumerTest > 
testFetchRecordLargerThanFetchMaxBytes() PASSED
[2021-09-03T20:29:41.854Z] 
[2021-09-03T20:29:41.854Z] PlaintextConsumerTest > testAutoCommitOnClose() 
STARTED
[2021-09-03T20:29:47.652Z] 
[2021-09-03T20:29:47.652Z] PlaintextConsumerTest > testAutoCommitOnClose() 
PASSED
[2021-09-03T20:29:47.652Z] 
[2021-09-03T20:29:47.652Z] PlaintextConsumerTest > testListTopics() STARTED
[2021-09-03T20:29:54.021Z] 
[2021-09-03T20:29:54.021Z] PlaintextConsumerTest > testListTopics() PASSED
[2021-09-03T20:29:54.021Z] 
[2021-09-03T20:29:54.021Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() STARTED
[2021-09-03T20:29:58.785Z] 
[2021-09-03T20:29:58.785Z] PlaintextConsumerTest > 
testExpandingTopicSubscriptions() PASSED
[2021-09-03T20:29:58.785Z] 
[2021-09-03T20:29:58.785Z] PlaintextConsumerTest > 
testMultiCo

[VOTE] KIP-773 Differentiate consistently metric latency measured in millis and nanos

[Josep Prat]

Hi there,

Since it's a rather small KIP, I'd like to start a vote for the KIP-773
Differentiate consistently metric latency measured  in millis and nanos.
The KIP page The KIP can be found at:
https://cwiki.apache.org/confluence/x/ZwNACw

Thanks in advance,

-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491715557497

*w:* aiven.io

*e:* josep.p...@aiven.io

[jira] [Created] (KAFKA-13272) KStream offset stuck with exactly_once after brokers outage

[Jira]

F Méthot created KAFKA-13272:


 Summary: KStream offset stuck with exactly_once after brokers 
outage
 Key: KAFKA-13272
 URL: https://issues.apache.org/jira/browse/KAFKA-13272
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.8.0
 Environment: Kafka running on Kubernetes
centos
Reporter: F Méthot


Our KStream app offset stay stuck with exactly_once after outage.

Running with KStream 2.8, kafka broker 2.8,
3 brokers.

commands topic is 10 partitions (replication 2, min-insync 2)
command-expiry-store-changelog topic is 10 partitions (replication 2, 
min-insync 2)
events topic is 10 partitions (replication 2, min-insync 2)

with this topology

Topologies:

 
{code:java}
Sub-topology: 0
 Source: KSTREAM-SOURCE-00 (topics: [commands])
 --> KSTREAM-TRANSFORM-01
 Processor: KSTREAM-TRANSFORM-01 (stores: [])
 --> KSTREAM-TRANSFORM-02
 <-- KSTREAM-SOURCE-00
 Processor: KSTREAM-TRANSFORM-02 (stores: [command-expiry-store])
 --> KSTREAM-SINK-03
 <-- KSTREAM-TRANSFORM-01
 Sink: KSTREAM-SINK-03 (topic: events)
 <-- KSTREAM-TRANSFORM-02
{code}
h3. 
Attempt 1 at reproducing this issue

 

Our stream app runs with processing.guarantee *exactly_once* 

After a Kafka test outage where all 3 brokers pod were deleted at the same time,

Brokers restarted and initialized succesfuly.

When restarting the topology above, one of the tasks would never initialize 
fully, the restore phase would keep outputting this messages every few minutes:

 
{code:java}
2021-08-16 14:20:33,421 INFO stream-thread 
[commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1] 
Restoration in progress for 1 partitions. 
{commands-processor-expiry-store-changelog-8: position=11775908, end=11775911, 
totalRestored=2002076} 
[commands-processor-51b0a534-34b6-47a4-bd9c-c57b6ecb8665-StreamThread-1] 
(org.apache.kafka.streams.processor.internals.StoreChangelogReader)
{code}
Task for partition 8 would never initialize, no more data would be read from 
the source commands topic for that partition.

 

In an attempt to recover, we restarted the stream app with stream 
processing.guarantee back to at_least_once, than it proceed with reading the 
changelog and restoring partition 8 fully.

But we noticed afterward, for the next hour until we rebuilt the system, that 
partition 8 from command-expiry-store-changelog would not be cleaned/compacted 
by the log cleaner/compacter compared to other partitions. (could be unrelated, 
because we have seen that before)

So we resorted to delete/recreate our command-expiry-store-changelog topic and 
events topic and regenerate it from the commands, reading from beginning. 

Things went back to normal
h3. Attempt 2 at reproducing this issue

We force-deleted all 3 pod running kafka.
After that, one of the partition can’t be restored. (like reported in previous 
attempt)
For that partition, we noticed these logs on the broker
{code:java}
[2021-08-27 17:45:32,799] INFO [Transaction Marker Channel Manager 1002]: 
Couldn’t find leader endpoint for partitions Set(__consumer_offsets-11, 
command-expiry-store-changelog-9) while trying to send transaction markers for 
commands-processor-0_9, these partitions are likely deleted already and hence 
can be skipped 
(kafka.coordinator.transaction.TransactionMarkerChannelManager){code}
Then

- we stop the kstream app,

- restarted kafka brokers cleanly

- Restarting the Kstream app, 

Those logs messages showed up on the kstream app log:

 
{code:java}
2021-08-27 18:34:42,413 INFO [Consumer 
clientId=commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1-consumer,
 groupId=commands-processor] The following partitions still have unstable 
offsets which are not cleared on the broker side: [commands-9], this could be 
either transactional offsets waiting for completion, or normal offsets waiting 
for replication after appending to local log 
[commands-processor-76602c87-f682-4648-859b-8fa9b6b937f3-StreamThread-1] 
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
 
{code}
This would cause our processor to not consume from that specific source 
topic-partition.
 Deleting downstream topic and replaying data would NOT fix the issue 
(EXACTLY_ONCE or AT_LEAST_ONCE)

Workaround found:

Deleted the group associated with the processor, and restarted the kstream 
application, application went on to process data normally. (We have resigned to 
use AT_LEAST_ONCE for now )

KStream config :
StreamsConfig.RETRY_BACKOFF_MS_CONFIG, 2000
StreamsConfig.REPLICATION_FACTOR_CONFIG, 2
StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000
StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 24MB
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG), “earliest”
StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE (now 
AT_LEAST_ONCE)
producer.d

Re: [Suspected Spam] Re: [VOTE] 3.0.0 RC1

[Gary Russell]

This [1] is a blocker for me.

Also, the documentation link [2] returns a 404.

[1]: https://issues.apache.org/jira/browse/KAFKA-13262
[2]: https://kafka.apache.org/30/documentation.html

From: Ron Dagostino 
Sent: Thursday, September 2, 2021 9:16 PM
To: dev@kafka.apache.org 
Cc: Users ; kafka-clients 

Subject: [Suspected Spam] Re: [VOTE] 3.0.0 RC1

Hi Konstantine.  I have opened a probable blocker ticket
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-13270&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DmGTJeScZXxP5WoszUhdqsJodj%2FCiHuFPXhSwkedR%2BI%3D&reserved=0.
  I will work on a PR
shortly.  The description on that ticket is as follows:

The implementation of 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FZOOKEEPER-3593&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5LV9W5CWAQD%2BnnBIcRXXsLUjhN6%2BqQ8ZjSNEZOlprOg%3D&reserved=0
 in
ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
problem if Kafka tries to retrieve a large amount of data across many
znodes – in such a case the ZooKeeper client will repeatedly emit a message
of the form "java.io.IOException: Packet len <> is out of range" and
the Kafka broker will never connect to ZooKeeper and fail to make progress
on the startup sequence. We can avoid the potential for this issue to occur
by explicitly setting the value to 4MB whenever we create a new ZooKeeper
client as long as no explicit value has been set via the `jute.maxbuffer`
system property.

Ron

On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo  wrote:

> Magnus,
>
> Please could you share the machine and network specs?
>
> How much CPU, RAM is available on each node?
>
> What JDK, JRE version are you using?
>
> What are your broker and client configuration values? Please could you
> share this info if possible?
>
> Thanks.
>
>
>
> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill 
> wrote:
>
> > Hi Konstantine,
> >
> > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> >
> > * Compaction seems to take slightly longer to kick in when segment sizes
> >   exceed their threshold. (Used to take less than 20 seconds, now takes
> > 20..30 seconds.)
> >
> > * CreateTopic seems to take slightly longer to propagate through the
> > cluster,
> >   e.g., before a new topic is available in metadata from other brokers.
> >
> > * CreateTopics seems to take longer when the Admin request timeout is
> set,
> >   looks like a plateau at 10 seconds:
> >   
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fimgur.com%2Fa%2Fn6y76sj&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1wARQVJ%2BycVR1GLla70EbZ9o4hPNpmey4wpGcWfzHvI%3D&reserved=0
> >
> > (This is a 3 broker cluster with identical configs between 2.8 and
> 3.0.0.)
> >
> > Nothing critical, but could be an indication of regression so I thought
> I'd
> > mention it.
> >
> > Regards,
> > Magnus
> >
> >
> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > kkaranta...@apache.org>:
> >
> > > Small correction to my previous email.
> > > The actual link for public preview of the 3.0.0 blog post draft is:
> > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.apache.org%2Fpreview%2Fkafka%2F%3FpreviewEntry%3Dwhat-s-new-in-apache6&data=04%7C01%7Cgrussell%40vmware.com%7C54b3dbe5cc6049a3ad8708d96e788b42%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637662286287405487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2F12fmUzsupiMAZy%2FZktR56j6F7njNiUVqNMo6LG6jvQ%3D&reserved=0
> > >
> > > (see also the email thread with title: [DISCUSS] Please review the
> 3.0.0
> > > blog post)
> > >
> > > Best,
> > > Konstantine
> > >
> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > kkaranta...@apache.org> wrote:
> > >
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > It corresponds to a major release that includes many new features,
> > > > including:
> > > >
> > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > other improvements in

[jira] [Created] (KAFKA-13271) Error while fetching metadata with correlation id 219783 : LEADER_NOT_AVAILABLE

[Jira]

Fátima Galera created KAFKA-13271:
-

 Summary: Error while fetching metadata with correlation id 219783 
: LEADER_NOT_AVAILABLE
 Key: KAFKA-13271
 URL: https://issues.apache.org/jira/browse/KAFKA-13271
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 2.2.2
 Environment: Production environment
Reporter: Fátima Galera
 Fix For: 2.2.3


Hi dear kafka support

 

We are getting below error after a new connector creation

 

[2021-09-02 19:15:23,878] WARN [Producer clientId=producer-178] Error while 
fetching metadata with correlation id 219783 : \{ 
tkr_prd2.tkr.glrep_file2=LEADER_NOT_AVAILABLE} 
(org.apache.kafka.clients.NetworkClient)
[2021-09-02 19:15:23,982] WARN [Producer clientId=producer-178] Error while 
fetching metadata with correlation id 219784 : \{ 
tkr_prd2.tkr.glrep_file2=LEADER_NOT_AVAILABLE} 
(org.apache.kafka.clients.NetworkClient)
[2021-09-02 19:15:24,095] ERROR WorkerSourceTask\{id=tkr_prd2-0} failed to send 
record to tkr_prd2.tkr.glrep_file2: {} (org.a 
pache.kafka.connect.runtime.WorkerSourceTask)
[2021-09-02 19:15:24,149] ERROR WorkerSourceTask\{id=tkr_prd2-0} failed to send 
record to tkr_prd2.tkr.glrep_file2: {} (org.a 
pache.kafka.connect.runtime.WorkerSourceTask)

 

We are not able to get the offset of this topic. We already changed from 

 {{listeners=PLAINTEXT://hostname:9092 to 
}}{{listeners=PLAINTEXT://localhost:9092}}{{}} 

in /etc/kafka/server.properties file and restarted kafka services. But if we 
use this new value kafka connector service is not able to start because it is 
not able to find the services with the IP of the host. Could you please let us 
know what can we do? We already created the connector several times using 
different names

 

https://stackoverflow.com/questions/35788697/leader-not-available-kafka-in-console-producer

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] KIP-773 Differentiate consistently metric latency measured in millis and nanos

[Josep Prat]

Hi there,

I would like to start the discussion thread for KIP-773 Differentiate
consistently metric latency measured in millis and nanos. The KIP can be
found at: https://cwiki.apache.org/confluence/x/ZwNACw

Thanks in advance,

-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491715557497

*w:* aiven.io

*e:* josep.p...@aiven.io

Re: [VOTE] 3.0.0 RC1

[David Jacot]

Hi Konstantine,

I'd like to raise a potential blocker:
https://issues.apache.org/jira/browse/KAFKA-13266.

When this bug is hit, the partition remains failed until the broker is
restated or
the partition is updated (e.g. its epoch is bumped). It only affects the
KRaft mode.

The PR is ready: https://issues.apache.org/jira/browse/KAFKA-13266.

Best,
David

On Fri, Sep 3, 2021 at 3:17 AM Ron Dagostino  wrote:

> Hi Konstantine.  I have opened a probable blocker ticket
> https://issues.apache.org/jira/browse/KAFKA-13270.  I will work on a PR
> shortly.  The description on that ticket is as follows:
>
> The implementation of https://issues.apache.org/jira/browse/ZOOKEEPER-3593
> in
> ZooKeeper version 3.6.0 decreased the default value for the ZooKeeper
> client's `jute.maxbuffer` configuration from 4MB to 1MB. This can cause a
> problem if Kafka tries to retrieve a large amount of data across many
> znodes – in such a case the ZooKeeper client will repeatedly emit a message
> of the form "java.io.IOException: Packet len <> is out of range" and
> the Kafka broker will never connect to ZooKeeper and fail to make progress
> on the startup sequence. We can avoid the potential for this issue to occur
> by explicitly setting the value to 4MB whenever we create a new ZooKeeper
> client as long as no explicit value has been set via the `jute.maxbuffer`
> system property.
>
> Ron
>
> On Thu, Sep 2, 2021 at 5:52 PM Israel Ekpo  wrote:
>
> > Magnus,
> >
> > Please could you share the machine and network specs?
> >
> > How much CPU, RAM is available on each node?
> >
> > What JDK, JRE version are you using?
> >
> > What are your broker and client configuration values? Please could you
> > share this info if possible?
> >
> > Thanks.
> >
> >
> >
> > On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill 
> > wrote:
> >
> > > Hi Konstantine,
> > >
> > > Some findings from running 3.0.0-RC1 with the librdkafka test suite:
> > >
> > > * Compaction seems to take slightly longer to kick in when segment
> sizes
> > >   exceed their threshold. (Used to take less than 20 seconds, now takes
> > > 20..30 seconds.)
> > >
> > > * CreateTopic seems to take slightly longer to propagate through the
> > > cluster,
> > >   e.g., before a new topic is available in metadata from other brokers.
> > >
> > > * CreateTopics seems to take longer when the Admin request timeout is
> > set,
> > >   looks like a plateau at 10 seconds:
> > >   https://imgur.com/a/n6y76sj
> > >
> > > (This is a 3 broker cluster with identical configs between 2.8 and
> > 3.0.0.)
> > >
> > > Nothing critical, but could be an indication of regression so I thought
> > I'd
> > > mention it.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine Karantasis <
> > > kkaranta...@apache.org>:
> > >
> > > > Small correction to my previous email.
> > > > The actual link for public preview of the 3.0.0 blog post draft is:
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > >
> > > > (see also the email thread with title: [DISCUSS] Please review the
> > 3.0.0
> > > > blog post)
> > > >
> > > > Best,
> > > > Konstantine
> > > >
> > > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis <
> > > > kkaranta...@apache.org> wrote:
> > > >
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second release candidate for Apache Kafka 3.0.0.
> > > > > It corresponds to a major release that includes many new features,
> > > > > including:
> > > > >
> > > > > * The deprecation of support for Java 8 and Scala 2.12.
> > > > > * Kafka Raft support for snapshots of the metadata topic and
> > > > > other improvements in the self-managed quorum.
> > > > > * Deprecation of message formats v0 and v1.
> > > > > * Stronger delivery guarantees for the Kafka producer enabled by
> > > default.
> > > > > * Optimizations in OffsetFetch and FindCoordinator requests.
> > > > > * More flexible Mirror Maker 2 configuration and deprecation of
> > > > > Mirror Maker 1.
> > > > > * Ability to restart a connector's tasks on a single call in Kafka
> > > > Connect.
> > > > > * Connector log contexts and connector client overrides are now
> > enabled
> > > > > by default.
> > > > > * Enhanced semantics for timestamp synchronization in Kafka
> Streams.
> > > > > * Revamped public API for Stream's TaskId.
> > > > > * Default serde becomes null in Kafka Streams and several
> > > > > other configuration changes.
> > > > >
> > > > > You may read and review a more detailed list of changes in the
> 3.0.0
> > > blog
> > > > > post draft here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6
> > > > >
> > > > > Release notes for the 3.0.0 release:
> > > > >
> > >
> https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote