Re: [EXTERNAL] SSL error while doing curl on kafka
Hi Jose, I was trying to validate the output of CURLfor ssl enabled kafka as there was issue with Kafka SSL while consumption through spark streaming. The client keystore was corrupt. I regenerated the certs and keystores. It worked fine. Kind Regards, Sachit Murarka On Tue, Jan 19, 2021 at 5:39 PM Jose Manuel Vega Monroy < jose.mon...@williamhill.com> wrote: > @Sachit > > SEC_ERROR_UNTRUSTED_ISSUER --> problem with SSL certificate, unstrusted > > So you would need CA certificate which issued into truststore used by curl > for calls to trust. > > Depending on OS could be in different location. > > But not sure what you trying to do, if you really interested on Kafka > client connection than curl. > > Thanks > > <http://www.williamhill.com/> > <http://www.whenthefunstops.co.uk/> > Jose Manuel Vega Monroy > Java Developer / Software Developer Engineer in Test > Direct: +0035 0 2008038 (Ext. 8038) > Email: jose.mon...@williamhill.com > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA > > > > > On 19/01/2021, 09:44, "Sachit Murarka" wrote: > > Hello All, > > I am doing curl o : of kafka. It is throwing below error > post > applying SSL. Can you please check? > > NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER) > * Peer's certificate issuer has been marked as not trusted by the user. > > > Kind Regards, > Sachit Murarka > > > Confidentiality: The contents of this e-mail and any attachments > transmitted with it are intended to be confidential to the intended > recipient; and may be privileged or otherwise protected from disclosure. If > you are not an intended recipient of this e-mail, do not duplicate or > redistribute it by any means. Please delete it and any attachments and > notify the sender that you have received it in error. This e-mail is sent > by a William Hill PLC group company. The William Hill group companies > include, among others, William Hill PLC (registered number 4212563), > William Hill Organization Limited (registered number 278208), William Hill > US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr > Green Limited (registered number C43260). Each of William Hill PLC and > William Hill Organization Limited is registered in England and Wales and > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle > County DE 19801 Delaware, United States of America. WHG (International) > Limited is registered in Gibraltar and has its registered office at 6/1 > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has > its registered office at Tagliaferro Business Centre, Level 7, 14 High > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, > the contents of this e-mail are subject to contract; and are not an > official statement, and do not necessarily represent the views, of William > Hill PLC, its subsidiaries or affiliated companies. Please note that > neither William Hill PLC, nor its subsidiaries and affiliated companies can > accept any responsibility for any viruses contained within this e-mail and > it is your responsibility to scan any emails and their attachments. William > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail > traffic data and also the content of e-mails for effective operation of the > e-mail system, or for security, purposes. >
SSL error while doing curl on kafka
Hello All, I am doing curl o : of kafka. It is throwing below error post applying SSL. Can you please check? NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER) * Peer's certificate issuer has been marked as not trusted by the user. Kind Regards, Sachit Murarka
Re: [EXTERNAL] Unable to connect to SSL enabled kafka
Hello,
Getting follow error:
UTC|TransportContext.java:345|Fatal (UNKNOWN_CA): Received fatal
alert: unknown_ca (
2021-01-18_14:41:22.26584 "throwable" : {
2021-01-18_14:41:22.26584 javax.net.ssl.SSLHandshakeException:
Received fatal alert: unknown_ca
Kind Regards,
Sachit Murarka
On Mon, Jan 18, 2021 at 6:32 PM Sachit Murarka
wrote:
> Thanks @Jose Manuel Vega Monroy for reply.
>
> I think some issue is there with CN name. I am actually doing it in
> docker.
>
> Kind Regards,
> Sachit Murarka
>
>
> On Mon, Jan 18, 2021 at 6:11 PM Jose Manuel Vega Monroy <
> jose.mon...@williamhill.com> wrote:
>
>> @Sachit
>>
>>
>>
>> You can use this in your client to see details of SSL connection and
>> handshake.
>>
>>
>>
>> -Djavax.net.debug=ssl,handshake
>>
>>
>>
>> Ensure your certificate is valid, signed and imported properly in your
>> keystore, and having root CA certificate into your truststore.
>>
>>
>>
>> Additionally, review SSL config in your client is right one, for example
>> SSL protocol version or SSL auth if you using it, plus pointing to
>> trustsore and keystore files path.
>>
>>
>>
>> Cheers,
>>
>>
>>
>> [image:
>> https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]
>> <http://www.williamhill.com/>
>>
>> [image:
>> https://www.williamhillplc.com/content/signature/senet.gif?width=180]
>> <http://www.whenthefunstops.co.uk/>
>>
>> *Jose Manuel Vega Monroy *
>> *Java Developer / Software Developer Engineer in Test*
>>
>> Direct: +*0035 0 2008038 (Ext. 8038)*
>> Email: jose.mon...@williamhill.com
>>
>> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *Sachit Murarka
>> *Date: *Monday, 18 January 2021 at 13:32
>> *To: *Jose Manuel Vega Monroy
>> *Cc: *"users@kafka.apache.org"
>> *Subject: *Re: [EXTERNAL] Unable to connect to SSL enabled kafka
>>
>>
>>
>> Hey Jose,
>>
>>
>> Used these sets of commands for SSL config.
>>
>>
>>
>> keytool -keystore client.truststore.jks -storepass pass -alias CARoot
>> -import -file root.crt -noprompt
>> keytool -keystore client.keystore.jks -storepass pass -alias client
>> -validity 365 -keyalg RSA -genkey -keypass pass -dname
>> "CN=client,OU=xyz,O=abc,L=BLR,ST=ka,C=IN"
>> keytool -keystore client.keystore.jks -storepass pass -alias client
>> -certreq -file client.unsigned.crt
>> openssl x509 -req -CA root.crt -CAkey root.key -in client.unsigned.crt
>> -out client.signed.crt -days 365 -CAcreateserial -passin pass:pass
>> -extensions SAN -extfile <(printf
>> "\n[SAN]\nsubjectAltName=DNS:client,DNS:localhost")
>> keytool -keystore client.keystore.jks -storepass pass -alias CARoot
>> -import -file root.crt -noprompt
>> keytool -keystore client.keystore.jks -storepass pass -alias client
>> -import -file client.signed.crt
>>
>> Not sure what is causing the issue exactly.
>>
>>
>>
>> Kind Regards,
>> Sachit Murarka
>>
>>
>>
>>
>>
>> On Mon, Jan 18, 2021 at 5:49 PM Jose Manuel Vega Monroy <
>> jose.mon...@williamhill.com> wrote:
>>
>> @Sachit
>>
>> Review your SSL client config.
>>
>> Cheers,
>>
>> <http://www.williamhill.com/>
>> <http://www.whenthefunstops.co.uk/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwMFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=qVT7wcON5mp40KH_xQ8EYLqCjpSmpEwpY1vf4EOvqwk&s=h3nzeeeSku1rOVllWxdpg11-1tKZ96zi6QB1MsDH8cw&e=>
>> >
>> Jose Manuel Vega Monroy
>> Java Developer / Software Developer Engineer in Test
>> Direct: +0035 0 2008038 (Ext. 8038)
>> Email: jose.mon...@williamhill.com
>> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>>
>>
>>
>>
>> On 18/01/2021, 12:47, "Sachit Murarka" wrote:
>>
>> Hey Users,
>>
>> I am getting the following error. Can anyone suggest?
>>
>> Error in attempt 3 getting Kafka offsets:
>> org.apache.kafka.common.errors.SslAuthenticationException: SSL
>> handshake
>> failed
>> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
>> message: server_hello
>>
Re: [EXTERNAL] Unable to connect to SSL enabled kafka
Thanks @Jose Manuel Vega Monroy for reply. I think some issue is there with CN name. I am actually doing it in docker. Kind Regards, Sachit Murarka On Mon, Jan 18, 2021 at 6:11 PM Jose Manuel Vega Monroy < jose.mon...@williamhill.com> wrote: > @Sachit > > > > You can use this in your client to see details of SSL connection and > handshake. > > > > -Djavax.net.debug=ssl,handshake > > > > Ensure your certificate is valid, signed and imported properly in your > keystore, and having root CA certificate into your truststore. > > > > Additionally, review SSL config in your client is right one, for example > SSL protocol version or SSL auth if you using it, plus pointing to > trustsore and keystore files path. > > > > Cheers, > > > > [image: > https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180] > <http://www.williamhill.com/> > > [image: > https://www.williamhillplc.com/content/signature/senet.gif?width=180] > <http://www.whenthefunstops.co.uk/> > > *Jose Manuel Vega Monroy * > *Java Developer / Software Developer Engineer in Test* > > Direct: +*0035 0 2008038 (Ext. 8038)* > Email: jose.mon...@williamhill.com > > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA > > > > > > > > > > *From: *Sachit Murarka > *Date: *Monday, 18 January 2021 at 13:32 > *To: *Jose Manuel Vega Monroy > *Cc: *"users@kafka.apache.org" > *Subject: *Re: [EXTERNAL] Unable to connect to SSL enabled kafka > > > > Hey Jose, > > > Used these sets of commands for SSL config. > > > > keytool -keystore client.truststore.jks -storepass pass -alias CARoot > -import -file root.crt -noprompt > keytool -keystore client.keystore.jks -storepass pass -alias client > -validity 365 -keyalg RSA -genkey -keypass pass -dname > "CN=client,OU=xyz,O=abc,L=BLR,ST=ka,C=IN" > keytool -keystore client.keystore.jks -storepass pass -alias client > -certreq -file client.unsigned.crt > openssl x509 -req -CA root.crt -CAkey root.key -in client.unsigned.crt > -out client.signed.crt -days 365 -CAcreateserial -passin pass:pass > -extensions SAN -extfile <(printf > "\n[SAN]\nsubjectAltName=DNS:client,DNS:localhost") > keytool -keystore client.keystore.jks -storepass pass -alias CARoot > -import -file root.crt -noprompt > keytool -keystore client.keystore.jks -storepass pass -alias client > -import -file client.signed.crt > > Not sure what is causing the issue exactly. > > > > Kind Regards, > Sachit Murarka > > > > > > On Mon, Jan 18, 2021 at 5:49 PM Jose Manuel Vega Monroy < > jose.mon...@williamhill.com> wrote: > > @Sachit > > Review your SSL client config. > > Cheers, > > <http://www.williamhill.com/> > <http://www.whenthefunstops.co.uk/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwMFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=qVT7wcON5mp40KH_xQ8EYLqCjpSmpEwpY1vf4EOvqwk&s=h3nzeeeSku1rOVllWxdpg11-1tKZ96zi6QB1MsDH8cw&e=> > > > Jose Manuel Vega Monroy > Java Developer / Software Developer Engineer in Test > Direct: +0035 0 2008038 (Ext. 8038) > Email: jose.mon...@williamhill.com > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA > > > > > On 18/01/2021, 12:47, "Sachit Murarka" wrote: > > Hey Users, > > I am getting the following error. Can anyone suggest? > > Error in attempt 3 getting Kafka offsets: > org.apache.kafka.common.errors.SslAuthenticationException: SSL > handshake > failed > Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake > message: server_hello > at sun.security.ssl.Alert.createSSLException(Alert.java:129) > at sun.security.ssl.Alert.createSSLException(Alert.java:117) > at > sun.security.ssl.TransportContext.fatal(TransportContext.java:314) > at > sun.security.ssl.TransportContext.fatal(TransportContext.java:270) > at > sun.security.ssl.TransportContext.fatal(TransportContext.java:261) > at > sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) > at > > sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:987) > at > > sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:974) > at java.security.AccessController.doPrivileged(Native Method) > at > > sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:921) >
Re: [EXTERNAL] Unable to connect to SSL enabled kafka
Hey Jose, Used these sets of commands for SSL config. keytool -keystore client.truststore.jks -storepass pass -alias CARoot -import -file root.crt -noprompt keytool -keystore client.keystore.jks -storepass pass -alias client -validity 365 -keyalg RSA -genkey -keypass pass -dname "CN=client,OU=xyz,O=abc,L=BLR,ST=ka,C=IN" keytool -keystore client.keystore.jks -storepass pass -alias client -certreq -file client.unsigned.crt openssl x509 -req -CA root.crt -CAkey root.key -in client.unsigned.crt -out client.signed.crt -days 365 -CAcreateserial -passin pass:pass -extensions SAN -extfile <(printf "\n[SAN]\nsubjectAltName=DNS:client,DNS:localhost") keytool -keystore client.keystore.jks -storepass pass -alias CARoot -import -file root.crt -noprompt keytool -keystore client.keystore.jks -storepass pass -alias client -import -file client.signed.crt Not sure what is causing the issue exactly. Kind Regards, Sachit Murarka On Mon, Jan 18, 2021 at 5:49 PM Jose Manuel Vega Monroy < jose.mon...@williamhill.com> wrote: > @Sachit > > Review your SSL client config. > > Cheers, > > <http://www.williamhill.com/> > <http://www.whenthefunstops.co.uk/> > Jose Manuel Vega Monroy > Java Developer / Software Developer Engineer in Test > Direct: +0035 0 2008038 (Ext. 8038) > Email: jose.mon...@williamhill.com > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA > > > > > On 18/01/2021, 12:47, "Sachit Murarka" wrote: > > Hey Users, > > I am getting the following error. Can anyone suggest? > > Error in attempt 3 getting Kafka offsets: > org.apache.kafka.common.errors.SslAuthenticationException: SSL > handshake > failed > Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake > message: server_hello > at sun.security.ssl.Alert.createSSLException(Alert.java:129) > at sun.security.ssl.Alert.createSSLException(Alert.java:117) > at > sun.security.ssl.TransportContext.fatal(TransportContext.java:314) > at > sun.security.ssl.TransportContext.fatal(TransportContext.java:270) > at > sun.security.ssl.TransportContext.fatal(TransportContext.java:261) > at > sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) > at > > sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:987) > at > > sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:974) > at java.security.AccessController.doPrivileged(Native Method) > at > > sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:921) > at > > org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:402) > at > > org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:484) > at > > org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:340) > at > > org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:265) > at > > org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:170) > at > > org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547) > at > org.apache.kafka.common.network.Selector.poll(Selector.java:483) > at > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212) > at > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:444) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1267) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1168) > at > > org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$partitionsAssignedToConsumer$2(KafkaOffsetReader.scala:538) > at > > org.apache.spark.sql.k
Unable to connect to SSL enabled kafka
$map$1(TraversableLike.scala:238) at scala.collection.immutable.Map$Map1.foreach(Map.scala:128) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$constructNextBatch$1(MicroBatchExecution.scala:368) at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.withProgressLocked(MicroBatchExecution.scala:597) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.constructNextBatch(MicroBatchExecution.scala:364) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:208) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:352) at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:350) at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:69) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:191) at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:185) at org.apache.spark.sql.execution.streaming.StreamExecution.org $apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.sca la:334) Kind Regards, Sachit Murarka
Re: Deciding number of partitions
Hi, Thanks for reply. You have shared the command. I am looking for the formula or way to calculate the number of partitions for my topic. Regards, Sachit On Sun, 11 Nov 2018, 20:08 花儿向阳开 <2468946...@qq.com wrote: > Hi, > Maybe you can use the command like: > kafka-topic.sh --create --topic --partitions > --replication-factor --zookeeper > > I hope it helps! > > > Regards! > Cathy > ------ 原始邮件 -- > 发件人: "Sachit Murarka"; > 发送时间: 2018年11月11日(星期天) 晚上10:48 > 收件人: "users"; > > 主题: Deciding number of partitions > > > > Hello All, > > I am new to kafka. Could anyone please tell how to decided number of > partitions for our topic? > > > Kind Regards, > Sachit Murarka
Deciding number of partitions
Hello All, I am new to kafka. Could anyone please tell how to decided number of partitions for our topic? Kind Regards, Sachit Murarka
Get count of messages
Hi All, Could you please help me in getting count of all messages stored in kafka from a particular offset? I have tried GetOffsetShell command, it is not giving me. Kind Regards, Sachit Murarka
