from:"Jose Manuel Vega Monroy"

Kafka Consumer JMX incoming byte rate to 0

2022-07-25 Thread Jose Manuel Vega Monroy

Hi there,

Just noticed from time to time, but not so often, Kafka consumer JMX incoming 
byte rate to 0, meanwhile consumer is consuming as expected.

SELECT average(newrelic.timeslice.value)
FROM Metric
WHERE metricTimesliceName = 
'MessageBroker/Kafka/Internal/consumer-node-metrics/incoming-byte-rate'

Kafka consumer version 2.7.1, and broker side is 2.6.2

Any idea which could be the problem?

We are thinking about downgrading consumer version to discard version related.

Thanks

[Logo, company name  Description automatically generated]
Jose Manuel Vega Monroy
Senior Backend Developer
Direct: +350
Mobile: +34(0) 633710634
WHG (International) Ltd  | 6/1 Waterport Place | Gibraltar |



Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidiaries and affiliated companies can accept any responsibility for any 
viruses contained within this e-mail and it is your responsibility to scan any 
emails and their attachments. William Hill PLC, its subsidiaries and affiliated 
companies may monitor e-mail traffic data and also the content of e-mails for 
effective operation of the e-mail system, or for security, purposes.

Partitions have leader brokers without a matching listener

2021-06-22 Thread Jose Manuel Vega Monroy

Hi there,

Recently with our Kafka broker 2.2.1 version started to face errors like next
one suddenly:

11 partitions have leader brokers without a matching listener, including

Any idea what’s the problem, how happened and impact?

There wasn’t any release apparently.

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Senior Backend Developer
Direct: +350
Mobile: +34(0) 633710634
WHG (International) Ltd | 6/1 Waterport Place | Gibraltar |

Confidentiality: The contents of this e-mail and any attachments transmitted
with it are intended to be confidential to the intended recipient; and may be
privileged or otherwise protected from disclosure. If you are not an intended
recipient of this e-mail, do not duplicate or redistribute it by any means.
Please delete it and any attachments and notify the sender that you have
received it in error. This e-mail is sent by a William Hill PLC group company.
The William Hill group companies include, among others, William Hill PLC
(registered number 4212563), William Hill Organization Limited (registered
number 278208), William Hill US HoldCo Inc, WHG (International) Limited
(registered number 99191) and Mr Green Limited (registered number C43260). Each
of William Hill PLC and William Hill Organization Limited is registered in
England and Wales and has its registered office at 1 Bedford Avenue, London,
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
County DE 19801 Delaware, United States of America. WHG (International) Limited
is registered in Gibraltar and has its registered office at 6/1 Waterport
Place, Gibraltar. Mr Green Limited is registered in Malta and has its
registered office at Tagliaferro Business Centre, Level 7, 14 High Street,
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents
of this e-mail are subject to contract; and are not an official statement, and
do not necessarily represent the views, of William Hill PLC, its subsidiaries
or affiliated companies. Please note that neither William Hill PLC, nor its
subsidiaries and affiliated companies can accept any responsibility for any
viruses contained within this e-mail and it is your responsibility to scan any
emails and their attachments. William Hill PLC, its subsidiaries and affiliated
companies may monitor e-mail traffic data and also the content of e-mails for
effective operation of the e-mail system, or for security, purposes.

Re: [EXTERNAL] SSL error while doing curl on kafka

2021-01-19 Thread Jose Manuel Vega Monroy

@Sachit

SEC_ERROR_UNTRUSTED_ISSUER --> problem with SSL certificate, unstrusted

So you would need CA certificate which issued into truststore used by curl for 
calls to trust.

Depending on OS could be in different location.

But not sure what you trying to do, if you really interested on Kafka client 
connection than curl.

Thanks

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 19/01/2021, 09:44, "Sachit Murarka"  wrote:

Hello All,

I am doing curl o : of kafka. It is throwing below error post
applying SSL. Can you please check?

NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.


Kind Regards,
Sachit Murarka


Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidiaries and affiliated companies can accept any responsibility for any 
viruses contained within this e-mail and it is your responsibility to scan any 
emails and their attachments. William Hill PLC, its subsidiaries and affiliated 
companies may monitor e-mail traffic data and also the content of e-mails for 
effective operation of the e-mail system, or for security, purposes.

Re: [EXTERNAL] Unable to connect to SSL enabled kafka

2021-01-18 Thread Jose Manuel Vega Monroy

@Sachit

You can use this in your client to see details of SSL connection and handshake.

-Djavax.net.debug=ssl,handshake

Ensure your certificate is valid, signed and imported properly in your 
keystore, and having root CA certificate into your truststore.

Additionally, review SSL config in your client is right one, for example SSL 
protocol version or SSL auth if you using it, plus pointing to trustsore and 
keystore files path.

Cheers,

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




From: Sachit Murarka 
Date: Monday, 18 January 2021 at 13:32
To: Jose Manuel Vega Monroy 
Cc: "users@kafka.apache.org" 
Subject: Re: [EXTERNAL] Unable to connect to SSL enabled kafka

Hey Jose,

Used these sets of commands for SSL config.

keytool -keystore  client.truststore.jks -storepass pass -alias CARoot -import 
-file root.crt -noprompt
keytool -keystore client.keystore.jks -storepass pass -alias client -validity 
365 -keyalg RSA -genkey -keypass pass -dname 
"CN=client,OU=xyz,O=abc,L=BLR,ST=ka,C=IN"
keytool -keystore client.keystore.jks -storepass pass -alias client -certreq 
-file client.unsigned.crt
openssl x509 -req -CA root.crt -CAkey root.key -in client.unsigned.crt -out 
client.signed.crt -days 365 -CAcreateserial -passin pass:pass -extensions SAN 
-extfile <(printf "\n[SAN]\nsubjectAltName=DNS:client,DNS:localhost")
keytool -keystore client.keystore.jks -storepass pass -alias CARoot -import 
-file root.crt -noprompt
keytool -keystore client.keystore.jks -storepass pass -alias client -import 
-file client.signed.crt
Not sure what is causing the issue exactly.

Kind Regards,
Sachit Murarka


On Mon, Jan 18, 2021 at 5:49 PM Jose Manuel Vega Monroy 
mailto:jose.mon...@williamhill.com>> wrote:
@Sachit

Review your SSL client config.

Cheers,

 <http://www.williamhill.com/>
 
<http://www.whenthefunstops.co.uk/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwMFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=qVT7wcON5mp40KH_xQ8EYLqCjpSmpEwpY1vf4EOvqwk&s=h3nzeeeSku1rOVllWxdpg11-1tKZ96zi6QB1MsDH8cw&e=>>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 18/01/2021, 12:47, "Sachit Murarka" 
mailto:connectsac...@gmail.com>> wrote:

Hey Users,

I am getting the following error. Can anyone suggest?

Error in attempt 3 getting Kafka offsets:
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake
failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
message: server_hello
at sun.security.ssl.Alert.createSSLException(Alert.java:129)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:270)
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:261)
at
sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at

sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:987)
at

sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:974)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:921)
at

org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:402)
at

org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:484)
at

org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:340)
at

org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:265)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:170)
at

org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547)
at

Re: [EXTERNAL] Unable to connect to SSL enabled kafka

2021-01-18 Thread Jose Manuel Vega Monroy

@Sachit

Review your SSL client config.

Cheers,

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 18/01/2021, 12:47, "Sachit Murarka"  wrote:

Hey Users,

I am getting the following error. Can anyone suggest?

Error in attempt 3 getting Kafka offsets:
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake
failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
message: server_hello
at sun.security.ssl.Alert.createSSLException(Alert.java:129)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:270)
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:261)
at
sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at

sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:987)
at

sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:974)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:921)
at

org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:402)
at

org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:484)
at

org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:340)
at

org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:265)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:170)
at

org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547)
at

org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at

org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at

org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
at

org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230)
at

org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:444)
at

org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1267)
at

org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
at

org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1168)
at

org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$partitionsAssignedToConsumer$2(KafkaOffsetReader.scala:538)
at

org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$withRetriesWithoutInterrupt$1(KafkaOffsetReader.scala:600)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at

org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77)
at

org.apache.spark.sql.kafka010.KafkaOffsetReader.withRetriesWithoutInterrupt(KafkaOffsetReader.scala:599)
at

org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$partitionsAssignedToConsumer$1(KafkaOffsetReader.scala:536)
at

org.apache.spark.sql.kafka010.KafkaOffsetReader.runUninterruptibly(KafkaOffsetReader.scala:567)
at

org.apache.spark.sql.kafka010.KafkaOffsetReader.partitionsAssignedToConsumer(KafkaOffsetReader.scala:536)
at

org.apache.spark.sql.kafka010.KafkaOffsetReader.fetchEarliestOffsets(KafkaOffsetReader.scala:298)
at

org.apache.spark.sql.kafka010.KafkaMicroBatchStream.$anonfun$getOrCreateInitialPartitionOffsets$1(KafkaMicroBatchStream.scala:151)
at scala.Option.getOrElse(Option.scala:189)
at

org.apache.spark.sql.kafka010.KafkaMicroBatchStream.getOrCreateInitialPartitionOffsets(KafkaMicroBatchStream.scala:148)
at

org.apache.spark.sql.kafka010.KafkaMicroBatchStream.initialOffset(KafkaMicroBatchStream.scala:76)
at

org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfu

Kafka consumer and producer JMX metrics

2020-10-16 Thread Jose Manuel Vega Monroy

Hi there,

We are switching our monitoring tool, and dealing with JMX metrics, notice we
were using for consumer and producer:

However, can find documentation pointing to kafka.producer and kafka.consumer
mbeans, and looking our brokers are not exposing those ones somehow.

Please could you point why and how we could expose?

Using version 2.2.1

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Re: [EXTERNAL] SSL handshake failed Error

2020-08-11 Thread Jose Manuel Vega Monroy

"failed authentication due to: SSL handshake failed" --> Ensure having keys, 
certificates and CA certificates in place; are the brokers connecting together 
to discard issue in broker side?

"javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?" 
--> Having security.inter.broker.protocol=SSL for brokers communication?


 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 11/08/2020, 17:09, "Павел Шевцов"  wrote:


I configured Kafka to work over SSL without authorization.
I rebooted Kafka and I get a certificate on a test connection.
(openssl s_client -connect :9093)

But when I try to connect with the producer, I get an error - "failed 
authentication due to: SSL handshake failed 
(org.apache.kafka.clients.NetworkClient)"
I added debugs (export KAFKA_OPTS="-Djavax.net.debug=ssl") and I get a 
message (javax.net.ssl.SSLException: Unrecognized SSL message, plaintext 
connection?)

My kafka server.properties
---
broker.id=10
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://:9092,SSL://:9093



ssl.keystore.location=/etc/ssl/kafka/kafka.server.keystore.jks
ssl.keystore.password=
ssl.key.password=
ssl.truststore.location=/etc/ssl/kafka/kafka.server.truststore.jks
ssl.truststore.password=
ssl.endpoint.identification.algorithm=



Command to use producer
/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 
:9093 --topic kafka-security-topic --producer.config 
/root/client-ssl/client.properties

client.properties
--
security.protocol=SSL
ssl.truststore.location=/root/client-ssl/kafka.client.truststore.jks
ssl.truststore.password=clientpass


Kafka version - 2.13-2.6.0

Any ideas?



Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidiaries and affiliated companies can accept any responsibility for any 
viruses contained within this e-mail and it is your responsibility to scan any 
emails and their attachments. William Hill PLC, its subsidiaries and affiliated 
companies may monitor e-mail traffic data and also the content of e-mails for 
effective operation of the e-mail system, or for security, purposes.

Re: [EXTERNAL] Re: Master-Slave being DB locked both

2020-05-06 Thread Jose Manuel Vega Monroy

@Jean

We are not defining neither those sizes nor sync mode, so I guess default 
values and synchronous?

Thanks

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 06/05/2020, 16:35, "Jean-Baptiste Onofre"  wrote:

Hi,

If it’s on NFS, what’s your rsize/wsize and sync mode enabled ?

Regards
JB

> Le 6 mai 2020 à 16:29, Jose Manuel Vega Monroy 
 a écrit :
>
> Hi there,
>
> With broker 5.15.0-1.el7.centos, looking like master and slave both 
brokers keeping DB locked with NFS remote share.
>
> 2020-05-06 15:08:52,372 [main] INFO  
(AbstractApplicationContext.java:583) - Refreshing 
org.apache.activemq.xbean.XBeanBrokerFactory$1@5025a98f: startup date [Wed May 
06 15:08:52 BST 2020]; root of context hierarchy
> 2020-05-06 15:08:52,444 [main] INFO  (XmlBeanDefinitionReader.java:317) - 
Loading XML bean definitions from class path resource [activemq.xml]
> 2020-05-06 15:08:52,909 [main] INFO  (XmlBeanDefinitionReader.java:317) - 
Loading XML bean definitions from class path resource [jetty.xml]
> 2020-05-06 15:08:53,357 [main] INFO  (BrokerService.java:683) - Using 
Persistence Adapter: KahaDBPersistenceAdapter[/mnt/sbm]
> 2020-05-06 15:08:59,463 [main] INFO  (SharedFileLocker.java:70) - 
Database /mnt/sbm/lock is locked by another server. This broker is now in slave 
mode waiting a lock to be acquired
>
> Any idea why this could be happening?
>
> Having in /etc/fstab:
>
> 
://mnt/sbmnfstcp,rsize=32768,wsize=32768,intr,noatime,actimeo=3,vers=300
>
> And config:
>
>  
>enableIndexWriteAsync="true"
>ignoreMissingJournalfiles="true"
>checkForCorruptJournalFiles="true"
>checksumJournalFiles="true"
>preallocationStrategy="zeros"
>/>
>
>
> Thanks
>
> Confidentiality: The contents of this e-mail and any attachments 
transmitted with it are intended to be confidential to the intended recipient; 
and may be privileged or otherwise protected from disclosure. If you are not an 
intended recipient of this e-mail, do not duplicate or redistribute it by any 
means. Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidiaries and affiliated companies can accept any responsibility for any 
viruses contained within this e-mail and it is your responsibility to scan any 
emails and their attachments. William Hill PLC, its subsidiaries and affiliated 
companies may monitor e-mail traffic data and also the content of e-mails for 
effective operation of the e-mail system, or for security, purposes.



Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registere

Re: [EXTERNAL] Re: UNKNOWN_PRODUCER_ID

2019-12-02 Thread Jose Manuel Vega Monroy

@Matthias

I will raise with the team that possible workaround meanwhile upgrading.

Thanks


 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 01/12/2019, 21:54, "Matthias J. Sax"  wrote:

Correct.

As long as there are only LOG statements about it, you don't need to
worry. If the producer cannot recover from this error internally, it
would throw an `UnknowdProducerIdException`; for this case, the write
was _not_ successful. Your application could handle this case by
creating a new `KafkaProducer` with the same `transactional.id` and
retry the transaction.

Note that KIP-360 is partly implemented in 2.4.0 already what should
reduce the likelihood to hit this issue.


-Matthias

    On 12/1/19 9:42 AM, Jose Manuel Vega Monroy wrote:
> Hi Matthias,
>
> So looking will be fixed in Kafka 2.5.0, as per 
https://issues.apache.org/jira/browse/KAFKA-8710, so we will take into account 
to upgrade properly to that version when ready and stable.
>
> We see this kind of errors in broker logs:
>
> [2019-12-01 15:27:39,790] ERROR [ReplicaManager broker=3] Error 
processing append operation on partition OB_SP11-12 
(kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.UnknownProducerIdException: Found no 
record of producerId=23001 on the broker at offset 24761in partition 
OB_SP11-12. It is possible that the last message with the producerId=23001 has 
been removed due to hitting the retention limit.
>
> Just to confirm, I guess they are correlated to producer errors right?
>
> Thanks
>
>  <http://www.williamhill.com/>
>  <http://www.whenthefunstops.co.uk/>
> Jose Manuel Vega Monroy
> Java Developer / Software Developer Engineer in Test
> Direct: +0035 0 2008038 (Ext. 8038)
> Email: jose.mon...@williamhill.com
> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>
>
>
>
> On 12/10/2019, 22:11, "Matthias J. Sax"  wrote:
>
> >> Does it mean producer’s metadata was removed due to retention time,
>
> Yes.
>
> >> and message is failed to be sent?
>
> No. Note, it's a warning not an error.
>
> The producer resets its internal state for this case and will send the
> message successfully. It's a known issue and addressed via KIP-360 
that
> provides a more detailed explanation:
    > 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer
>
>
> -Matthias
>
> On 10/12/19 12:18 PM, Jose Manuel Vega Monroy wrote:
> > Hi there,
> >
> >
> >
> > We are seeing WARN log traces from time to time in our Kafka 
producer:
> >
> >
> >
> > 2019-10-12 15:19:14:707
> > WARN[org.apache.kafka.clients.producer.internals.Sender] [Producer
> > clientId=prdxpds21dsm001.prod.williamhill.plc] Got error produce
> > response with correlation id 4899937 on topic-partition OB_SP25-3,
> > retrying (0 attempts left). Error: UNKNOWN_PRODUCER_ID
> >
> >
> >
> > 2019-10-1220:02:21:686 WARN
> > [org.apache.kafka.clients.producer.internals.Sender] [Producer
> > clientId=prdxpds21dsm001.prod.williamhill.plc] Got errorproduce 
response
> > with correlation id 5354758 on topic-partition OB_SP87-16, retrying 
(0
> > attempts left). Error:UNKNOWN_PRODUCER_ID
> >
> >
> >
> > Does it mean producer’s metadata was removed due to retention time, 
and
> > message is failed to be sent?
> >
> >
> >
> > I’m not sure but looking like happening since we enabled in 
producer:
> >
> >
> >
> > *enable.idempotence=true*
> >
> >
> >
>     > Any idea?
    > >
> >
> >
> > Thanks
> >
> >
> >
> > 
https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180
> > <http://www.williamhill.com/>
> >
>

Re: [EXTERNAL] Re: UNKNOWN_PRODUCER_ID

2019-12-01 Thread Jose Manuel Vega Monroy

Hi Matthias,

So looking will be fixed in Kafka 2.5.0, as per 
https://issues.apache.org/jira/browse/KAFKA-8710, so we will take into account 
to upgrade properly to that version when ready and stable.

We see this kind of errors in broker logs:

[2019-12-01 15:27:39,790] ERROR [ReplicaManager broker=3] Error processing 
append operation on partition OB_SP11-12 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.UnknownProducerIdException: Found no record of 
producerId=23001 on the broker at offset 24761in partition OB_SP11-12. It is 
possible that the last message with the producerId=23001 has been removed due 
to hitting the retention limit.

Just to confirm, I guess they are correlated to producer errors right?

Thanks

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

On 12/10/2019, 22:11, "Matthias J. Sax"  wrote:

>> Does it mean producer’s metadata was removed due to retention time,

Yes.

>> and message is failed to be sent?

No. Note, it's a warning not an error.

The producer resets its internal state for this case and will send the
message successfully. It's a known issue and addressed via KIP-360 that
provides a more detailed explanation:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer

    -Matthias

On 10/12/19 12:18 PM, Jose Manuel Vega Monroy wrote:
> Hi there,
>
>
>
> We are seeing WARN log traces from time to time in our Kafka producer:
>
>
>
> 2019-10-12 15:19:14:707
> WARN[org.apache.kafka.clients.producer.internals.Sender] [Producer
> clientId=prdxpds21dsm001.prod.williamhill.plc] Got error produce
> response with correlation id 4899937 on topic-partition OB_SP25-3,
> retrying (0 attempts left). Error: UNKNOWN_PRODUCER_ID
>
>
>
> 2019-10-1220:02:21:686 WARN
> [org.apache.kafka.clients.producer.internals.Sender] [Producer
> clientId=prdxpds21dsm001.prod.williamhill.plc] Got errorproduce response
> with correlation id 5354758 on topic-partition OB_SP87-16, retrying (0
> attempts left). Error:UNKNOWN_PRODUCER_ID
>
>
>
> Does it mean producer’s metadata was removed due to retention time, and
> message is failed to be sent?
>
>
>
> I’m not sure but looking like happening since we enabled in producer:
>
>
>
> *enable.idempotence=true*
>
>
>
> Any idea?
>
>
>
> Thanks
>
>
>
> https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180
    > <http://www.williamhill.com/>
>
> https://www.williamhillplc.com/content/signature/senet.gif?width=180
> <http://www.whenthefunstops.co.uk/>
>
>
>
> *Jose Manuel Vega Monroy **
> **Java Developer / Software Developer Engineer in Test*
>
> Direct: +*0035 0 2008038 (Ext. 8038)*
> Email: jose.mon...@williamhill.com <mailto:jose.mon...@williamhill.com>
>
> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>
>
>
>
>
>
>

Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549,

Re: [EXTERNAL] Kafka upgrade from 0.10 to 2.3.x

2019-11-22 Thread Jose Manuel Vega Monroy

Hi Vitalii,

About log format, if clients are old ones will start to fail since they 
couldn't understand the messages.

So bump them if you planning upgrade the log format version.

Thanks

Get Outlook for Android


From: Vitalii Stoianov 
Sent: Thursday, November 21, 2019 10:31:22 PM
To: users@kafka.apache.org 
Subject: [EXTERNAL] Kafka upgrade from 0.10 to 2.3.x

Hi All,

All the docs that I was able to find describe the rolling upgrade.
But I didn't find any docs that describe how to perform no-rolling upgrade.

So if system can afford downtime how in this case to upgrade kafka form
version 0.10.0 to 2.3.x.
Is it still required to do few restarts or we can just upgrade all in one
go?

For example:
0. Stop producers -> consumer -> brokers.
1. Upgrade brokers, clients.
2. Upgrade brokers, inter.broker.protocol.version.
3. Upgrade brokers, log.message.format.version.
4. Start brokers.
5. Start consumers.
6. Start producers.

In case of upgrade from 0.10.0 to 2.3.x Docs states that message format has
changed since version 0.10.0
https://urldefense.proofpoint.com/v2/url?u=https-3A__kafka.apache.org_22_documentation.html-23messageset&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1Bbq_fCBGYdws-Lil0jrt3sHYfBeO69HPka-rrOZbH4&s=B_4bFQr79wzHjrL7LRtsG7wOCcTJqmcQqOUfoAC80gc&e=
  (old format that
is used by 0.10.0) and in 0.11.0 was introduced new format that is used
till now 2.3.x (as I understand).

What will happen with kafka brokers data logs on disk after message format
change change ?
And if data on disk is not changed during change of message format: What
will happen if new consumer receive message in old message format because
it was not read by consumer before upgrade?

Regards,
Vitalii.
Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidiaries and affiliated companies can accept any responsibility for any 
viruses contained within this e-mail and it is your responsibility to scan any 
emails and their attachments. William Hill PLC, its subsidiaries and affiliated 
companies may monitor e-mail traffic data and also the content of e-mails for 
effective operation of the e-mail system, or for security, purposes.

Message order and retries question

2019-11-08 Thread Jose Manuel Vega Monroy

Hi there,

I have a question about message order and retries.

After checking official documentation, and asking your feedback, we set this
kafka client configuration in each producer:

retries = 1

# note to ensure order enable.idempotence=true, which forcing to acks=all
and max.in.flight.requests.per.connection<=5

enable.idempotence = true

max.in.flight.requests.per.connection = 4

acks = "all"

However, somehow while rolling upgrade, we saw producer retrying a lot of times
(for example, 16 times), and finally sending fine when broker was up and
running back, with exceptions like this:

Cause: org.apache.kafka.common.errors.OutOfOrderSequenceException: The broker
received an out of order sequence number..
Cause: org.apache.kafka.common.errors.NotLeaderForPartitionException: This
server is not the leader for that topic-partition..

Is that behaviour expected? It’s that retries configuration right trying to
ensure the message order, or maybe we should remove retries configuration from
our producers?

As well we found this related to retries:

The default value for the producer's retries config was changed to
Integer.MAX_VALUE, as we introduced
delivery.timeout.ms<http://delivery.timeout.ms/> in KIP-91, which sets an upper
bound on the total time between sending a record and receiving acknowledgement
from the broker. By default, the delivery timeout is set to 2 minutes.

Allowing retries without setting `max.in.flight.requests.per.connection` to `1`
will potentially change the ordering of records because if two batches are sent
to a single partition, and the first fails and is retried but the second
succeeds, then the records in the second batch may appear first. Note
additionally that produce requests will be failed before the number of retries
has been exhausted if the timeout configured by
delivery.timeout.ms<http://delivery.timeout.ms/> expires first before
successful acknowledgement. Users should generally prefer to leave this config
unset and instead use delivery.timeout.ms<http://delivery.timeout.ms/> to
control retry behavior.

Note this was faced while rolling upgrade from 2.1.1 to 2.2.1.

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Re: [EXTERNAL] SSL setup failing

2019-10-28 Thread Jose Manuel Vega Monroy

@Peter

I have the feeling is related to client.auth required, in the end each broker 
is a client for the rest in the cluster.

Try with client.auth=none, and check if the connect.

If so, the review SSL conf related to that.

Cheers

Get Outlook for Android<https://aka.ms/ghei36>

From: Péter Nagykátai 
Sent: Monday, October 28, 2019 2:47:38 PM
To: users@kafka.apache.org 
Subject: Re: [EXTERNAL] SSL setup failing

@Jose
>9092 is as well SSL protocol?
Yes, it is. As you see in the config snippet from my initial email.

> Zookeeper is connecting over SSL?
Yes, at least as far as I can tell. It's set up there too but neither of
those are making verifying that easy...

>So then I would review all certificates to check if valid.
I did that after your first response.

>As well there is a Kafka broker property 'advertised.host.name' you could
set with same hostname in the certificate.
I added this property but didn't change anything, I get the exact same
error messages.

Thanks

On Mon, Oct 28, 2019 at 2:08 PM Jose Manuel Vega Monroy <
jose.mon...@williamhill.com> wrote:

> @Peter
>
> 9092 is as well SSL protocol? Zookeeper is connecting over SSL?
>
> So then I would review all certificates to check if valid.
>
> As well there is a Kafka broker property 'advertised.host.name' you could
> set with same hostname in the certificate.
>
> Thanks
>
>  <http://www.williamhill.com/>
>  
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=EpV0EqDiDgDPupfjZMqzuAv4qAvJraWxyVssyKeT39o&s=nltRImLIvceytcuboTTrAYca_JBmNpjbPGfTEcexHVw&e=
>  >
> Jose Manuel Vega Monroy
> Java Developer / Software Developer Engineer in Test
> Direct: +0035 0 2008038 (Ext. 8038)
> Email: jose.mon...@williamhill.com
> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>
>
>
>
> On 28/10/2019, 14:03, "Péter Nagykátai"  wrote:
>
> Sorry, if I was unclear before. I'm absolutely new to Kafka and how it
> works.
>
> @Jose
> >That happening when clients trying to SSL connect?
> There are no clients at the moment just one Kafka broker which spews
> the
> errors in the server.log. To be specific, there is a ZooKeeper client
> which
> has no issues:
> >> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
>
> @Manna
> >Are you talking about local network loopback?
> No, at least I don't think so. I'm simply trying to have the broker in
> a
> stable running state but after it starts, it tries to connect to the
> listed
> internal broker, which is itself since at the moment there aren't other
> brokers in the cluster.
>
> >Also, have you tried ssl debug using openssl? What did you observe?
> Per Jose's advice I checked the certificates I generated last week and
> everything checked out on the 'rootCA' node.
>
> >How have you setup your signed certificates?
> I have a secured node where I generate certificates for every server
> in the
> cluster (with an intermediate CA). Here are the commands I used:
>
> `openssl genrsa -out kafka-1.key.pem 2048`
> `openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new
> -sha256 -out kafka-1.csr.pem`
> `openssl ca -config openssl_intermediate.cnf -extensions server_cert
> -days
> 375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem`
> `openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem
> -out
> kafka-1.p12 -name kafka-1`
> `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> -alias kafka-1 -destkeystore kafka-1.jks`
>
> Also, for the root+intermediate chain:
> `keytool -importcert -alias ca-root -keystore truststore.jks -file
> ca-chain.cert.pem`
>
> >Does your CN/SAN matches with your advertised.listeners setup?
> Yes.
>
> >Have you setup hostname verification correctly?
> My Kafka configuration file only have the settings I pasted before, the
> rest aren't network specific.
>
>
> My (beginner) opinion is that Kafka tries to authenticate itself as a
> client and gets confused when getting 'server_hello' message.
> ("Unexpected
> handshake message: server_hello")
>
> Thanks!
>
> On Mon, Oct 28, 2019 at 12:25 PM M. Manna  wrote:
>
> > Hi,
> >
> > not sure what it means "Tries to communicate with itself". Are you
> talking
> > abo

Re: [EXTERNAL] SSL setup failing

2019-10-28 Thread Jose Manuel Vega Monroy

@Peter

9092 is as well SSL protocol? Zookeeper is connecting over SSL?

So then I would review all certificates to check if valid.

As well there is a Kafka broker property 'advertised.host.name' you could set 
with same hostname in the certificate.

Thanks

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 28/10/2019, 14:03, "Péter Nagykátai"  wrote:

Sorry, if I was unclear before. I'm absolutely new to Kafka and how it
works.

@Jose
>That happening when clients trying to SSL connect?
There are no clients at the moment just one Kafka broker which spews the
errors in the server.log. To be specific, there is a ZooKeeper client which
has no issues:
>> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)

@Manna
>Are you talking about local network loopback?
No, at least I don't think so. I'm simply trying to have the broker in a
stable running state but after it starts, it tries to connect to the listed
internal broker, which is itself since at the moment there aren't other
brokers in the cluster.

>Also, have you tried ssl debug using openssl? What did you observe?
Per Jose's advice I checked the certificates I generated last week and
everything checked out on the 'rootCA' node.

>How have you setup your signed certificates?
I have a secured node where I generate certificates for every server in the
cluster (with an intermediate CA). Here are the commands I used:

`openssl genrsa -out kafka-1.key.pem 2048`
`openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new
-sha256 -out kafka-1.csr.pem`
`openssl ca -config openssl_intermediate.cnf -extensions server_cert -days
375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem`
`openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem -out
kafka-1.p12 -name kafka-1`
`keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
-alias kafka-1 -destkeystore kafka-1.jks`

Also, for the root+intermediate chain:
`keytool -importcert -alias ca-root -keystore truststore.jks -file
ca-chain.cert.pem`

>Does your CN/SAN matches with your advertised.listeners setup?
Yes.

>Have you setup hostname verification correctly?
My Kafka configuration file only have the settings I pasted before, the
rest aren't network specific.


My (beginner) opinion is that Kafka tries to authenticate itself as a
client and gets confused when getting 'server_hello' message. ("Unexpected
handshake message: server_hello")

Thanks!

On Mon, Oct 28, 2019 at 12:25 PM M. Manna  wrote:

> Hi,
>
> not sure what it means "Tries to communicate with itself". Are you talking
> about local network loopback?
>
> Also, have you tried ssl debug using openssl? What did you observe?
>
> The exception is handshake exception. This is quite common when your cert
> validation fails. How have you setup your signed certificates? Does your
> CN/SAN matches with your advertised.listeners setup? Have you setup
> hostname verification correctly?
>
> Thanks,
>
> On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai 
> wrote:
>
> > @Jose
> >
> > >It looks like communication problem between brokers.
> > As I mentioned, "I can't get the first broker started". The message 
above
> > is from when the broker tries to communicate with "itself": [Controller
> > id=1001, targetBrokerId=1001]).
> >
> > Nevertheless, I went through the checklist and everything is in order.
> For
> > the first couple of tries, I got different SSL errors but I could work
> > those out (that time I messed up the certificates), but now the problem
> is:
> > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake
> > **message:
> > server_hello*
> >
> > Peter
> >
> > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
> > jose.mon...@williamhill.com> wrote:
> >
> > > @Peter
> > >
> > > It looks like communication problem between brokers. But ensure:
> > >
> > > 1) Crtificates are valid and properly signed by root CA or 
intermediate
> > > one in the chain
> > > 2) Clients and brokers having private key and certificate in

Re: [EXTERNAL] SSL setup failing

2019-10-28 Thread Jose Manuel Vega Monroy

@Peter

That happening when clients trying to SSL connect?

Review SSL configuration related ssl.client.auth=required is right in client 
side.

Here you have explained possible errors, and you could find what happening: 
https://docs.apigee.com/api-platform/troubleshoot/runtime/ssl-handshake-failures

For example, could be certificate CN name not matching hostname used to connect 
client-server, throwing something like this:

java.security.cert.CertificateException: No name matching localhost found

Thanks

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 28/10/2019, 12:11, "Péter Nagykátai"  wrote:

@Jose

>It looks like communication problem between brokers.
As I mentioned, "I can't get the first broker started". The message above
is from when the broker tries to communicate with "itself": [Controller
id=1001, targetBrokerId=1001]).

Nevertheless, I went through the checklist and everything is in order. For
the first couple of tries, I got different SSL errors but I could work
those out (that time I messed up the certificates), but now the problem is:
>> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake 
**message:
server_hello*

    Peter

    On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
jose.mon...@williamhill.com> wrote:

> @Peter
>
> It looks like communication problem between brokers. But ensure:
>
> 1) Crtificates are valid and properly signed by root CA or intermediate
> one in the chain
> 2) Clients and brokers having private key and certificate in their
> keystore and properly configured to point to its path
> 3) Clients and brokers having CA certificates in the truststore and
> properly configured to point to its path
> 4) Clients and brokersbroker having root CA certificate in their keystore
> and properly configured to.point to its path
> 5) Permissions are right ones fro trustore and keystore
>
> Thanks
>
> Get Outlook for Android 
<https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=uwX9KbgU8pPFAgG1vr70bVs7l0XYYV9Rn8yZR07Fgsg&s=ib4HzMkTN7pBJB_cZVJPatBggBNpQjzQOfaDArTFrkQ&e=
 >
>
> --
> *From:* Péter Nagykátai 
> *Sent:* Monday, 28 October 2019, 00:13
> *To:* users@kafka.apache.org
> *Subject:* [EXTERNAL] SSL setup failing
>
> Hi!
>
> I'm experimenting with setting up a log ingesting cluster and Kafka would
> be part of it. Unfortunately, I can't get the first broker started. I need
> to secure the communication between a dozen nodes and Kaquiafka would only
> be
> one part of it. I have a secured node where I generate certificates for
> every server in the cluster (with an intermediate CA). AFAIK, I need to 
use
> '.jks' files for Kafka, so I've generated a '.p12' file from the openssl
> certificate and key then used `keytool` to generate a keystore:
> `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> -alias kafka-1 -destkeystore kafka-1.jks`
> I generated a truststore for the root and intermediate chain as well:
> `keytool -importcert -alias ca-root -keystore truststore.jks -file
> ca-chain.cert.pem
>
> Relevant part of the 'server.properties' configuration:
> 
> listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> inter.broker.listener.name=INTERNAL
> listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> security.protocol=SSL
> ssl.client.auth=required
> ssl.truststore.location=/***/truststore.jks
> ssl.truststore.password=*
> ssl.keystore.location=/***/kafka-1.jks
> ssl.keystore.password=*
> 
>
> After starting Kafka (as a service) I get the the following in the
> 'server.log':
> >>...
> >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> >> INFO [SocketServer brokerId=1001] Failed authentication with
> /XXX.XXX.XXX.XXX (SSL handshake failed)
> (org.apache.kafka.common.network.Selector)
> >> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
> with kafka-1/XXX.XXX.XXX.X

Re: [EXTERNAL] SSL setup failing

2019-10-28 Thread Jose Manuel Vega Monroy

@Peter

It looks like communication problem between brokers. But ensure:

1) Crtificates are valid and properly signed by root CA or intermediate one in 
the chain
2) Clients and brokers having private key and certificate in their keystore and 
properly configured to point to its path
3) Clients and brokers having CA certificates in the truststore and properly 
configured to point to its path
4) Clients and brokersbroker having root CA certificate in their keystore and 
properly configured to.point to its path
5) Permissions are right ones fro trustore and keystore

Thanks

Get Outlook for Android


From: Péter Nagykátai 
Sent: Monday, 28 October 2019, 00:13
To: users@kafka.apache.org
Subject: [EXTERNAL] SSL setup failing

Hi!

I'm experimenting with setting up a log ingesting cluster and Kafka would
be part of it. Unfortunately, I can't get the first broker started. I need
to secure the communication between a dozen nodes and Kaquiafka would only be
one part of it. I have a secured node where I generate certificates for
every server in the cluster (with an intermediate CA). AFAIK, I need to use
'.jks' files for Kafka, so I've generated a '.p12' file from the openssl
certificate and key then used `keytool` to generate a keystore:
`keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
-alias kafka-1 -destkeystore kafka-1.jks`
I generated a truststore for the root and intermediate chain as well:
`keytool -importcert -alias ca-root -keystore truststore.jks -file
ca-chain.cert.pem

Relevant part of the 'server.properties' configuration:

listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
security.protocol=SSL
ssl.client.auth=required
ssl.truststore.location=/***/truststore.jks
ssl.truststore.password=*
ssl.keystore.location=/***/kafka-1.jks
ssl.keystore.password=*


After starting Kafka (as a service) I get the the following in the
'server.log':
>>...
>> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
>> INFO [SocketServer brokerId=1001] Failed authentication with
/XXX.XXX.XXX.XXX (SSL handshake failed)
(org.apache.kafka.common.network.Selector)
>> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
(org.apache.kafka.common.network.Selector)
>> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node 1001
(kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL handshake
failed (org.apache.kafka.clients.NetworkClient)
>>...
>> WARN SSL handshake failed (kafka.utils.CoreUtils$)
>> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake
failed
>> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
message: server_hello
>>...

I couldn't find any lead with that error message and got stuck. Any ideas
what that error message means and how to solve it?

Specs:
- Ubuntu 18.04.3 LTS
- OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
- Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
- OpenSSL 1.1.1

Thank you!
Peter

Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidia

InvalidPidMappingException and CloseSafeProducer

2019-10-22 Thread Jose Manuel Vega Monroy

Hi there,

Around 3 AM we faced these errors in a Kafka client when trying to produce:

org.apache.kafka.common.KafkaException: 
org.apache.kafka.common.errors.InvalidPidMappingException: The producer 
attempted to use a producer id which is not currently assigned to its 
transactional id
at 
org.apache.kafka.clients.producer.internals.TransactionManager$AddPartitionsToTxnHandler.handleResponse(TransactionManager.java:1039)
at 
org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
at 
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.InvalidPidMappingException: The 
producer attempted to use a producer id which is not currently assigned to its 
transactional id
2019-10-22 03:01:49.416 WARN 16012 --- [Thread-5] 
o.s.k.core.DefaultKafkaProducerFactory : Error during transactional operation; 
producer removed from cache; possible cause: broker restarted during 
transaction: CloseSafeProducer 
[delegate=org.apache.kafka.clients.producer.KafkaProducer@173c048c, 
txId=oxirep-batch-7ecefb68-0462-4dd6-8f36-dd9489016ac80]

Any idea why?

I’m still investigating, but in theory no brokers were restarted around that 
time.

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA



Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidiaries and affiliated companies can accept any responsibility for any 
viruses contained within this e-mail and it is your responsibility to scan any 
emails and their attachments. William Hill PLC, its subsidiaries and affiliated 
companies may monitor e-mail traffic data and also the content of e-mails for 
effective operation of the e-mail system, or for security, purposes.

NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required..

2019-10-21 Thread Jose Manuel Vega Monroy

Hi there,

While rolling deploying SSL changes to get Kafka working over SSL + inter
broker protocol updated from SASL_PLAINTEXT to SSL, getting like problems to
publish messages with:

org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are
rejected since there are fewer in-sync replicas than required..

With configuration:

default.replication.factor=3

min.insync.replicas=2

num.replica.fetchers=1

offsets.topic.replication.factor=3

Expected? I couldn’t say when exactly happened, but for sure having a few nodes
still pending to release, and a few ones already released (note we have a
cluster with 5 nodes).

Those messages are retried, or finally lost to publish?

Maybe we could tune up a little bit our broker configuration?

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Closing socket - IllegalArgumentException - Invalid version for API key OFFSET_FOR_LEADER_EPOCH: 2

2019-10-16 Thread Jose Manuel Vega Monroy

Hi there,

During rolling upgrade to allow Kafka to listen SSL communications via new port 
we noticed this:

[2019-10-16 14:28:48,900] ERROR Closing socket for 
10.191.68.155:9092-10.191.68.161:33320 because of error 
(kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error getting request 
for apiKey: 23 and apiVersion: 2
Caused by: java.lang.IllegalArgumentException: Invalid version for API key 
OFFSET_FOR_LEADER_EPOCH: 2
at org.apache.kafka.common.protocol.ApiKeys.schemaFor(ApiKeys.java:173)
at org.apache.kafka.common.protocol.ApiKeys.requestSchema(ApiKeys.java:141)
at org.apache.kafka.common.protocol.ApiKeys.parseRequest(ApiKeys.java:149)
at 
org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:112)
at 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:99)
at kafka.network.RequestChannel$Request.(RequestChannel.scala:93)
at 
kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:520)
at 
kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:511)
at scala.collection.Iterator.foreach(Iterator.scala:929)
at scala.collection.Iterator.foreach$(Iterator.scala:929)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.network.Processor.processCompletedReceives(SocketServer.scala:511)
at kafka.network.Processor.run(SocketServer.scala:436)
at java.lang.Thread.run(Thread.java:748)

Any idea why?

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA



Confidentiality: The contents of this e-mail and any attachments transmitted 
with it are intended to be confidential to the intended recipient; and may be 
privileged or otherwise protected from disclosure. If you are not an intended 
recipient of this e-mail, do not duplicate or redistribute it by any means. 
Please delete it and any attachments and notify the sender that you have 
received it in error. This e-mail is sent by a William Hill PLC group company. 
The William Hill group companies include, among others, William Hill PLC 
(registered number 4212563), William Hill Organization Limited (registered 
number 278208), William Hill US HoldCo Inc, WHG (International) Limited 
(registered number 99191) and Mr Green Limited (registered number C43260). Each 
of William Hill PLC and William Hill Organization Limited is registered in 
England and Wales and has its registered office at 1 Bedford Avenue, London, 
WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has 
its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle 
County DE 19801 Delaware, United States of America. WHG (International) Limited 
is registered in Gibraltar and has its registered office at 6/1 Waterport 
Place, Gibraltar. Mr Green Limited is registered in Malta and has its 
registered office at Tagliaferro Business Centre, Level 7, 14 High Street, 
Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents 
of this e-mail are subject to contract; and are not an official statement, and 
do not necessarily represent the views, of William Hill PLC, its subsidiaries 
or affiliated companies. Please note that neither William Hill PLC, nor its 
subsidiaries and affiliated companies can accept any responsibility for any 
viruses contained within this e-mail and it is your responsibility to scan any 
emails and their attachments. William Hill PLC, its subsidiaries and affiliated 
companies may monitor e-mail traffic data and also the content of e-mails for 
effective operation of the e-mail system, or for security, purposes.

UNKNOWN_PRODUCER_ID

2019-10-12 Thread Jose Manuel Vega Monroy

Hi there,

We are seeing WARN log traces from time to time in our Kafka producer:

2019-10-12 15:19:14:707 WARN 
[org.apache.kafka.clients.producer.internals.Sender] [Producer 
clientId=prdxpds21dsm001.prod.williamhill.plc] Got error produce response with 
correlation id 4899937 on topic-partition OB_SP25-3, retrying (0 attempts 
left). Error: UNKNOWN_PRODUCER_ID

2019-10-12 20:02:21:686 WARN 
[org.apache.kafka.clients.producer.internals.Sender] [Producer 
clientId=prdxpds21dsm001.prod.williamhill.plc] Got error produce response with 
correlation id 5354758 on topic-partition OB_SP87-16, retrying (0 attempts 
left). Error: UNKNOWN_PRODUCER_ID

Does it mean producer’s metadata was removed due to retention time, and message 
is failed to be sent?

I’m not sure but looking like happening since we enabled in producer:

enable.idempotence=true

Any idea?

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

OutOfOrderSequenceException

2019-10-07 Thread Jose Manuel Vega Monroy

Hi there,

Finally we upgraded our producer configuration to ensure message order:

retries = 1
# note to ensure order enable.idempotence=true, which forcing to acks=all and 
max.in.flight.requests.per.connection<=5
enable.idempotence = true
max.in.flight.requests.per.connection = 4
acks = "all"

However, recently we faced this exception:

org.apache.kafka.common.errors.OutOfOrderSequenceException: The broker received 
an out of order sequence number..

Any idea why happened? Is that expected?

It could be related to retries configuration? It’s that configuration properly 
set?

From official 
documentation<https://kafka.apache.org/21/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html>
 we found recommending unset retries, being default value to Integer.MAX_VALUE:


“To take advantage of the idempotent producer, it is imperative to avoid 
application level re-sends since these cannot be de-duplicated. As such, if an 
application enables idempotence, it is recommended to leave the retries config 
unset, as it will be defaulted to Integer.MAX_VALUE. Additionally, if a 
send(ProducerRecord) returns an error even with infinite retries (for instance 
if the message expires in the buffer before being sent), then it is recommended 
to shut down the producer and check the contents of the last produced message 
to ensure that it is not duplicated. Finally, the producer can only guarantee 
idempotence for messages sent within a single session.”

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Re: [EXTERNAL] Re: Kafka - Possible messages order

2019-09-02 Thread Jose Manuel Vega Monroy

Hi there,

I need to verify, but probably due to our Kafka client being so old.

Thanks
 
 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy 
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
 
 
 

On 02/09/2019, 12:15, "Jose Manuel Vega Monroy"  
wrote:

Hi there,

Finally we enabled idempotence, and setting as per official documentation 
max.in.flight.requests.per.connection <= 5 and acks = all.

'enable.idempotence'=> 'true',
'max.in.flight.requests.per.connection' => '4',
'acks'  => 'all',

" When set to 'true', the producer will ensure that exactly one copy of 
each message is written in the stream. If 'false', producer retries due to 
broker failures, etc., may write duplicates of the retried message in the 
stream. Note that enabling idempotence requires 
max.in.flight.requests.per.connection to be less than or equal to 5, retries to 
be greater than 0 and acks must be 'all'. If these values are not explicitly 
set by the user, suitable values will be chosen. If incompatible values are 
set, a ConfigException will be thrown."

And now facing problems to create producer, forcing 
max.in.flight.requests.per.connection = 1:

Caused by: org.apache.kafka.common.config.ConfigException: Must set 
max.in.flight.requests.per.connection to 1 in orderto use the idempotent 
producer. Otherwise we cannot guarantee idempotence.
at 
org.apache.kafka.clients.producer.KafkaProducer.configureInflightRequests(KafkaProducer.java:482)
at 
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:360)
... 17 common frames omitted

Any idea about?

Our Kafka broker version is 2.2.1.

Thanks
 
 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy 
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
 
 
 

On 29/08/2019, 17:47, "Anatoliy Soldatov"  
wrote:

What exactly is the problem?

Kafka states it in documentation and it is not new.

Quote from docs: max.in.flight.requests.per.connection – The maximum 
number of unacknowledged requests the client will send on a single connection 
before blocking. Note that if this setting is set to be greater than 1 and 
there are failed sends, there is a risk of message re-ordering due to retries 
(i.e., if retries are enabled).
    
The possible problem is only with some people not able to read 
documentation.

29 авг. 2019 г., в 17:11, Jose Manuel Vega Monroy 
mailto:jose.mon...@williamhill.com>> написал(а):

Hi there,

Recently we found this blog entry about a possible problem with 
messages order: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.softwaremill.com_does-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=eoZO2gqSx3WfGCIIcane_DH28EY2j1xHy5pwK9o70L4&e=
 
<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fblog.softwaremill.com-252Fdoes-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034629468-26sdata-3D-252BulzkEeXQ6xhczBYfYQ8wnvWrpW-252Fs3wVtQaGGwIgPbM-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=dFwxCXAt09CuEx1CL20C8BR4SKDsWzZgfdIQ3U6ZAi8&e=
 >

Please could you confirm about? And if so, how to fix? Blog entry 
suggesting how to.

Thanks


[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_WHlogo.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ph2WXQKWT4O1uhip6cQzhhNHSKnxw6S5xChr13VQmMk&e=
 
]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.williamhill.com-252F-26data-3D02-257C01-257Caksoldatov-

Re: [EXTERNAL] Re: Kafka - Possible messages order

2019-09-02 Thread Jose Manuel Vega Monroy

Hi there,

Finally we enabled idempotence, and setting as per official documentation 
max.in.flight.requests.per.connection <= 5 and acks = all.

'enable.idempotence'=> 'true',
'max.in.flight.requests.per.connection' => '4',
'acks'  => 'all',

" When set to 'true', the producer will ensure that exactly one copy of each 
message is written in the stream. If 'false', producer retries due to broker 
failures, etc., may write duplicates of the retried message in the stream. Note 
that enabling idempotence requires max.in.flight.requests.per.connection to be 
less than or equal to 5, retries to be greater than 0 and acks must be 'all'. 
If these values are not explicitly set by the user, suitable values will be 
chosen. If incompatible values are set, a ConfigException will be thrown."

And now facing problems to create producer, forcing 
max.in.flight.requests.per.connection = 1:

Caused by: org.apache.kafka.common.config.ConfigException: Must set 
max.in.flight.requests.per.connection to 1 in orderto use the idempotent 
producer. Otherwise we cannot guarantee idempotence.
at 
org.apache.kafka.clients.producer.KafkaProducer.configureInflightRequests(KafkaProducer.java:482)
at 
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:360)
... 17 common frames omitted

Any idea about?

Our Kafka broker version is 2.2.1.

Thanks

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy 
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

On 29/08/2019, 17:47, "Anatoliy Soldatov"  wrote:

What exactly is the problem?

Kafka states it in documentation and it is not new.

Quote from docs: max.in.flight.requests.per.connection – The maximum number 
of unacknowledged requests the client will send on a single connection before 
blocking. Note that if this setting is set to be greater than 1 and there are 
failed sends, there is a risk of message re-ordering due to retries (i.e., if 
retries are enabled).

    The possible problem is only with some people not able to read 
documentation.

29 авг. 2019 г., в 17:11, Jose Manuel Vega Monroy 
mailto:jose.mon...@williamhill.com>> написал(а):

Hi there,

Recently we found this blog entry about a possible problem with messages 
order: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.softwaremill.com_does-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=eoZO2gqSx3WfGCIIcane_DH28EY2j1xHy5pwK9o70L4&e=

<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fblog.softwaremill.com-252Fdoes-2Dkafka-2Dreally-2Dguarantee-2Dthe-2Dorder-2Dof-2Dmessages-2D3ca849fd19d2-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034629468-26sdata-3D-252BulzkEeXQ6xhczBYfYQ8wnvWrpW-252Fs3wVtQaGGwIgPbM-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=dFwxCXAt09CuEx1CL20C8BR4SKDsWzZgfdIQ3U6ZAi8&e=
 >

Please could you confirm about? And if so, how to fix? Blog entry 
suggesting how to.

Thanks

[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_WHlogo.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=ph2WXQKWT4O1uhip6cQzhhNHSKnxw6S5xChr13VQmMk&e=

]<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur02.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fwww.williamhill.com-252F-26data-3D02-257C01-257Caksoldatov-2540avito.ru-257C6ef8bd10a67b4bef52fb08d72c8ad121-257Caf0e07b3b90b472392e63fab11dd5396-257C1-257C0-257C637026847034639463-26sdata-3DKowmt6DYcYKQgMC-252FI5Yv-252BeZNsaPhh8oK8-252F9jpvfyuSk-253D-26reserved-3D0&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA0CkD_inthU1Io&s=T7tzoiDU2XkOMiQ_Hq0KEBKH3DoW7lM2Hv2jGqwutjc&e=
 >

[https://urldefense.proofpoint.com/v2/url?u=https-3A__www.williamhillplc.com_content_signature_senet.gif-3Fwidth-3D180&d=DwIGaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=1MQ2lQLysHJNStmrKbWGa06C-07LwA

Kafka - Possible messages order

2019-08-29 Thread Jose Manuel Vega Monroy

Hi there,

Recently we found this blog entry about a possible problem with messages order: 
https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2

Please could you confirm about? And if so, how to fix? Blog entry suggesting 
how to.

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Re: [EXTERNAL] Handling of inter.broker.protocol.version and lo.message.format.version after upgrading

2019-07-24 Thread Jose Manuel Vega Monroy

Hi Sebastian,

Recently we upgraded exactly from same version to that version, and you need 
just two rolling restarts if you want two keep same log format version, or 
three to upgrade that version too.

Personally, I would recommend you to keep those properties, for example you 
could keep the current log format version for compatibility with old clients 
versions.

Cheers,

Get Outlook for Android



From: Sebastian Schmitz
Sent: Thursday, 25 July, 03:22
Subject: [EXTERNAL] Handling of inter.broker.protocol.version and 
lo.message.format.version after upgrading
To: users@kafka.apache.org


Hey guys,

I wonder what I should do with both settings for future upgrades after
having finished the upgrade from 0.10.1 to 2.1.1.

Should I just remove it from the config so it will default to the
current version of Kafka or do I have to do the rolling upgrade with
changing versions and restarting Kafka three times each time?

Thanks

Sebastian


--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please

notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.

Kafka Rolling Upgrade to 2.2.1

2019-07-17 Thread Jose Manuel Vega Monroy

Hi there,

I'm trying to rolling upgrade from 2.1.1 with inter broker protocol version 2.0 
to broker version 2.2.1.

So which inter protocol version I should set, 2.2 or keeping the current 2.0?

Thanks

Get Outlook for Android

Re: Kafka 2.1.1 migration - JMX metrics problem

2019-06-25 Thread Jose Manuel Vega Monroy

Hi there,

We found it was related to quotas definition, having defined clients and users 
ones with default value at the same time.

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




From: Jose Manuel Vega Monroy 
Date: Monday, 24 June 2019 at 16:55
To: "users@kafka.apache.org" 
Cc: Luca Segafredo , Andreea Nicula 
, Laur Aliste , 
Pedro Da Costa , Cezary Gajdzinski 
, Paul Johnson 
Subject: Kafka 2.1.1 migration - JMX metrics problem

Hi there,

Recently we rolling upgraded our internal brokers from version 0.11.0.1 to 
version 2.1.1 in different envs, with the next definition of JMX metrics for 
Introscope Wily:

JMX|kafka.server|Fetch:queue-size,\
JMX|kafka.server|Fetch|*:byte-rate,\
JMX|kafka.server|Fetch|*:throttle-time,\
JMX|kafka.server|Produce:queue-size,\
JMX|kafka.server|Produce|*:byte-rate,\
JMX|kafka.server|Produce|*:throttle-time,\

However, somehow even defining client.id and group.id in our consumer/producer 
app conf, looking like Wily is taken the apps metrics by the username we are 
defining in sasl.jaas.config for security protocol SASL_PLAINTEXT, instead of 
client.id like used to be before migration.

Any idea why this is happening?

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Kafka 2.1.1 migration - JMX metrics problem

2019-06-24 Thread Jose Manuel Vega Monroy

Hi there,

Recently we rolling upgraded our internal brokers from version 0.11.0.1 to 
version 2.1.1 in different envs, with the next definition of JMX metrics for 
Introscope Wily:

JMX|kafka.server|Fetch:queue-size,\
JMX|kafka.server|Fetch|*:byte-rate,\
JMX|kafka.server|Fetch|*:throttle-time,\
JMX|kafka.server|Produce:queue-size,\
JMX|kafka.server|Produce|*:byte-rate,\
JMX|kafka.server|Produce|*:throttle-time,\

However, somehow even defining client.id and group.id in our consumer/producer 
app conf, looking like Wily is taken the apps metrics by the username we are 
defining in sasl.jaas.config for security protocol SASL_PLAINTEXT, instead of 
client.id like used to be before migration.

Any idea why this is happening?

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Re: [EXTERNAL] Re: Kafka - JMX Metrics for Fetch and Produce

2019-04-21 Thread Jose Manuel Vega Monroy

Hi there,

Included "Bandwidth quota metrics per (user, client-id), user or client-id"?

That was the metric we were monitoring, with regex to show any user and 
clientId.

Thanks
 
 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy 
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
 
 
 

On 21/04/2019, 21:10, "Manikumar"  wrote:

Hi,

RequestsPerSec metric was changed to include the API version in Kafka 2.0
release.
You may need to update jmx query.


https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_documentation_-23upgrade-5F200-5Fnotable&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=HpbvSUu37R0CDXVebPOpAiXnWZvMUBHNRztCp9xwf20&s=zZQcoInkTwcV58vNqte10APezEUrDt-GvCevjKeN_OE&e=

https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D272-253A-2BAdd-2BAPI-2Bversion-2Btag-2Bto-2Bbroker-2527s-2BRequestsPerSec-2Bmetric&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=HpbvSUu37R0CDXVebPOpAiXnWZvMUBHNRztCp9xwf20&s=s9z5arHWXznh1HdIQ4CTacRYCl885Ss84-LVkFSahro&e=

On Mon, Apr 22, 2019 at 12:23 AM Jose Manuel Vega Monroy <
jose.mon...@williamhill.com> wrote:

> Hi there,
>
>
>
> Recently I performed several * rolling upgrades following official steps*
> for our Kafka brokers *from 0.11.0.1 to newer versions in different
> environments*, and apparently working fine in all cases from functional
> point of view: *producers and consumers working as expected*.
>
>
>
> Specifically, I upgraded:
>
>
>
>1. From *0.11.0.1 to 1.0.0*, and then from *1.0.0 to 2.0.0*, and then*
>to* *2.1.1*
>2. *From 0.11.0.1 directly to 2.1.1*
>
>
>
> However, in all cases *JMX metrics for Fetch and Produce* which used to
> show *all producers and consumers working with brokers* are gone, just
> showing queue-size, in our *JMX monitoring clients* -specifically, 
*Introscope
> Wily*- keeping same configuration*.*
>
>
>
> In fact, I removed Wily filter configuration for JMX in *order to show
> all possible metrics, and keeping both Fetch and Produce still gone*.
>
>
>
> Note I checked if having proper version after rolling upgrade, for
> example, for *2.1.1*, and being as expected:
>
>
>
> *ll /opt/kafka/libs/*
>
> *total 54032*
>
> *-rw-r--r-- 1 kafka kafka69409 Jan  4 08:42 activation-1.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka14768 Jan  4 08:42
> aopalliance-repackaged-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka90347 Jan  4 08:42 argparse4j-0.7.0.jar*
>
> *-rw-r--r-- 1 kafka kafka20437 Jan  4 08:40
> audience-annotations-0.5.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   501879 Jan  4 08:43 commons-lang3-3.8.1.jar*
>
> *-rw-r--r-- 1 kafka kafka96801 Feb  8 18:32 connect-api-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka18265 Feb  8 18:32
> connect-basic-auth-extension-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka20509 Feb  8 18:32 connect-file-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka45489 Feb  8 18:32 connect-json-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   466588 Feb  8 18:32 connect-runtime-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka90358 Feb  8 18:32
> connect-transforms-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka  2442625 Jan  4 08:43 guava-20.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   186763 Jan  4 08:42 hk2-api-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   189454 Jan  4 08:42 hk2-locator-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   135317 Jan  4 08:42 hk2-utils-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka66894 Jan 11 21:28
> jackson-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   325619 Jan 11 21:27 jackson-core-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka  1347236 Jan 11 21:27 
jackson-databind-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32373 Jan 11 21:28
> jackson-jaxrs-base-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka15861 Jan 11 21:28
> jackson-jaxrs-json-provider-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32627 Jan 11 21:28
> jackson-module-jaxb-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   737884 Jan  4 08:43 javassist

Kafka - JMX Metrics for Fetch and Produce

2019-04-21 Thread Jose Manuel Vega Monroy

 18:32 kafka_2.12-2.1.1-sources.jar.asc
-rw-r--r-- 1 kafka kafka  2956429 Feb  8 18:32 kafka_2.12-2.1.1-test.jar
-rw-r--r-- 1 kafka kafka  821 Feb  8 18:32 kafka_2.12-2.1.1-test.jar.asc
-rw-r--r-- 1 kafka kafka   740522 Feb  8 18:32 kafka_2.12-2.1.1-test-sources.jar
-rw-r--r-- 1 kafka kafka  821 Feb  8 18:32 
kafka_2.12-2.1.1-test-sources.jar.asc
-rw-r--r-- 1 kafka kafka  1893108 Feb  8 18:31 kafka-clients-2.1.1.jar
-rw-r--r-- 1 kafka kafka14499 Feb  8 18:32 kafka-log4j-appender-2.1.1.jar
-rw-r--r-- 1 kafka kafka   871817 Feb  8 18:32 kafka-streams-2.1.1.jar
-rw-r--r-- 1 kafka kafka39331 Feb  8 18:32 kafka-streams-examples-2.1.1.jar
-rw-r--r-- 1 kafka kafka   170375 Feb  8 18:33 
kafka-streams-scala_2.12-2.1.1.jar
-rw-r--r-- 1 kafka kafka38309 Feb  8 18:32 
kafka-streams-test-utils-2.1.1.jar
-rw-r--r-- 1 kafka kafka   328053 Feb  8 18:32 kafka-tools-2.1.1.jar
-rw-r--r-- 1 kafka kafka   489884 Jan  4 08:39 log4j-1.2.17.jar
-rw-r--r-- 1 kafka kafka   511547 Jan  4 08:39 lz4-java-1.5.0.jar
-rw-r--r-- 1 kafka kafka55051 Jan  4 08:43 maven-artifact-3.6.0.jar
-rw-r--r-- 1 kafka kafka82123 Jan  4 08:40 metrics-core-2.2.0.jar
-rw-r--r-- 1 kafka kafka20235 Jan  4 08:42 osgi-resource-locator-1.0.1.jar
-rw-r--r-- 1 kafka kafka   261617 Jan  4 08:43 plexus-utils-3.1.0.jar
-rw-r--r-- 1 kafka kafka   130999 Jan  4 08:43 reflections-0.9.11.jar
-rw-r--r-- 1 kafka kafka 13908249 Jan  7 19:26 rocksdbjni-5.14.2.jar
-rw-r--r-- 1 kafka kafka  5277511 Jan  4 08:40 scala-library-2.12.7.jar
-rw-r--r-- 1 kafka kafka54709 Jan  4 08:40 scala-logging_2.12-3.9.0.jar
-rw-r--r-- 1 kafka kafka  3616372 Jan  4 08:40 scala-reflect-2.12.7.jar
-rw-r--r-- 1 kafka kafka41203 Jan  4 08:36 slf4j-api-1.7.25.jar
-rw-r--r-- 1 kafka kafka12244 Jan  4 08:39 slf4j-log4j12-1.7.25.jar
-rw-r--r-- 1 kafka kafka  2018686 Jan  4 08:39 snappy-java-1.1.7.2.jar
-rw-r--r-- 1 kafka kafka63777 Jan  4 08:42 validation-api-1.1.0.Final.jar
-rw-r--r-- 1 kafka kafka74626 Jan  4 08:40 zkclient-0.11.jar
-rw-r--r-- 1 kafka kafka   906708 Jan  4 08:40 zookeeper-3.4.13.jar
-rw-r--r-- 1 kafka kafka  3810541 Jan  4 08:39 zstd-jni-1.3.7-1.jar

Any idea why is this happening? Maybe upgrade was wrong? Maybe related to any 
broker missconfiguration?

Thanks

[https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/>
[https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/>

Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com>
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA

Kafka Consumer JMX incoming byte rate to 0

Partitions have leader brokers without a matching listener

Re: [EXTERNAL] SSL error while doing curl on kafka

Re: [EXTERNAL] Unable to connect to SSL enabled kafka

Re: [EXTERNAL] Unable to connect to SSL enabled kafka

Kafka consumer and producer JMX metrics

Re: [EXTERNAL] SSL handshake failed Error

Re: [EXTERNAL] Re: Master-Slave being DB locked both

Re: [EXTERNAL] Re: UNKNOWN_PRODUCER_ID

Re: [EXTERNAL] Re: UNKNOWN_PRODUCER_ID

Re: [EXTERNAL] Kafka upgrade from 0.10 to 2.3.x

Message order and retries question

Re: [EXTERNAL] SSL setup failing

Re: [EXTERNAL] SSL setup failing

Re: [EXTERNAL] SSL setup failing

Re: [EXTERNAL] SSL setup failing

InvalidPidMappingException and CloseSafeProducer

NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required..

Closing socket - IllegalArgumentException - Invalid version for API key OFFSET_FOR_LEADER_EPOCH: 2

UNKNOWN_PRODUCER_ID

OutOfOrderSequenceException

Re: [EXTERNAL] Re: Kafka - Possible messages order

Re: [EXTERNAL] Re: Kafka - Possible messages order

Kafka - Possible messages order

Re: [EXTERNAL] Handling of inter.broker.protocol.version and lo.message.format.version after upgrading

Kafka Rolling Upgrade to 2.2.1

Re: Kafka 2.1.1 migration - JMX metrics problem

Kafka 2.1.1 migration - JMX metrics problem

Re: [EXTERNAL] Re: Kafka - JMX Metrics for Fetch and Produce

Kafka - JMX Metrics for Fetch and Produce

30 matches

Site Navigation

Mail list logo

Footer information