Not sure how to explain it more clearly. You cannot communicate with plain text on a TLS port and vica versa: TLS comm. cannot be accepted on plaintext ports. It’s a general thing, not just ZooKeeper.
Andor > On 2020. Mar 31., at 23:57, karthick rn <karthick.narend...@gmail.com> wrote: > > Hi Andor, > > I've tried the openssl command you shared but unable to get it working, may > be something to do with converting to PEM format the keys and certs. I'll > look into this. > >> There’s no point trying non-secure communication on the secure port as > it’s currently not unified. > I'm not getting, please can you explain it? > > Thanks, > Karthick > > > > > > > > > > > > On Tue, 31 Mar 2020 at 15:50, Patrick Hunt <ph...@apache.org> wrote: > >> We (Karthick can you? :-) ) should add this to the docs. PR would be great! >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute >> >> Thanks, >> >> Patrick >> >> On Tue, Mar 31, 2020 at 7:17 AM Andor Molnar <an...@apache.org> wrote: >> >>> Hi Karthick, >>> >>> The following command works for me on the secure port (1181): >>> >>> (echo "srvr"; sleep 1) | openssl s_client -connect zkhost:1181 -cert >>> cert.pem -key ./key.pem >>> >>> I had to add sleep, because openssl client closes the connection as soon >>> as stdin ends. >>> There’s no point trying non-secure communication on the secure port as >>> it’s currently not unified. >>> >>> Andor >>> >>> >>> >>> >>> >>>> On 2020. Mar 31., at 15:22, karthick rn <karthick.narend...@gmail.com> >>> wrote: >>>> >>>> Thanks Enrico for sharing the jira. This is great! >>>> >>>> With the below config, I'm now able to run the 4LW commands >> successfully, >>>> also the downstream systems that was relying on the 4LW commands >> started >>>> displaying the metrics. Thanks for your help. >>>> >>>> #secureClientPort=2281 >>>> >>>> clientPort=2281 >>>> >>>> client.portUnification=True >>>> >>>> >>>> - Karthick >>>> >>>> >>>> >>>> On Mon, 30 Mar 2020 at 21:59, Enrico Olivelli <eolive...@gmail.com> >>> wrote: >>>> >>>>> You may be interested in Port unification, contributed by Facebook: >>>>> >>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-3388 >>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-3371 >>>>> >>>>> Enrico >>>>> >>>>> Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn >>>>> <karthick.narend...@gmail.com> ha scritto: >>>>>> >>>>>> Hi Mate, >>>>>> >>>>>> Thanks for suggesting these options in detail >>>>>> >>>>>> 1) We are already using AdminServer as an alternate to the 4LW, >>> hopefully >>>>>> we'll look at modifying the downstream systems to use REST instead of >>> the >>>>>> 4LW commands. >>>>>> >>>>>> 2) Added "clientPort=2181" back to the configs and tested "srvr" & >>> other >>>>>> whitelisted 4LW commands and they all work now :) >>>>>> >>>>>> 3) When I configure the same port "2281" for both secure and unsecure >>>>>> communication with "client.portUnification=true", the JVM exits with >>> Bind >>>>>> exception stating the "Address already in use" & unable to start ZK. >>>>>> >>>>>> For short term, I think we'd run a mixed-mode communication like you >>>>>> mentioned in option 2 & whitelist only specific 4LW commands required >>> and >>>>>> not all. >>>>>> >>>>>> Appreciate if someone can confirm if the 4LW is expected to work >>> against >>>>>> secure client port or not so we can update the doc accordingly. >> Thanks >>>>>> again! >>>>>> >>>>>> Regards, >>>>>> Karthick >>>>>> >>>>>> On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté < >>>>> szalay.beko.m...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Karthick, >>>>>>> >>>>>>> I am not sure if "echo srvr | nc localhost 2281" is expected to work >>>>>>> against the secure client port. I don't think so, but maybe others >>> know >>>>>>> better. I think you have the following options: >>>>>>> >>>>>>> 1) use the admin server which is a HTTP interface where the 4LW >>>>> commands >>>>>>> are available on a REST protocol (see >>>>>>> >>>>> >>> >> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver >>>>>>> ) >>>>>>> >>>>>>> 2) if AdminServer is not an option for you, then you can configure >>>>>>> ZooKeeper to use both secure and unsecure ports. And use the >> unsecure >>>>> port >>>>>>> for 4LW commands, while use the secure port for the rest of the >>>>> traffic. >>>>>>> E.g.: >>>>>>> clientPort=2281 >>>>>>> secureClientPort=2282 >>>>>>> >>>>>>> 3) you can even configure ZooKeeper to use the same port for both >> TLS >>>>> and >>>>>>> unsecure communication. I haven't used 4LW commands with port >>>>> unification, >>>>>>> but I assume it works: >>>>>>> client.portUnification=true >>>>>>> >>>>>>> I hope some of these options will work for you. >>>>>>> >>>>>>> Kind regards, >>>>>>> Mate >>>>>>> >>>>>>> On Mon, Mar 30, 2020 at 12:24 AM karthick rn < >>>>> karthick.narend...@gmail.com >>>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> After configuring TLS, running "echo srvr | nc localhost 2281" or >> any >>>>>>> other >>>>>>>> 4LW doesn’t show any output. The below messages are printed on the >>>>> ZK log >>>>>>>> whilst running the ‘srvr’ command. Also tried adding >>>>>>>> "4lw.commands.whitelist=*" to zoo.cfg but still no difference. >>>>> However, >>>>>>>> disabling TLS I'm able to see all 4LW working as expected. >>>>>>>> >>>>>>>> Let me know if this is a known issue when TLS is enabled? I'm using >>>>> ZK >>>>>>> v3.6 >>>>>>>> and have seen the same behaviour with v3.5.6 & 3.5.7. >>>>>>>> >>>>>>>> I have shared my Quorum TLS configs at the bottom, in-case if you >>>>> want to >>>>>>>> check if I'm missing something. Many thanks >>>>>>>> >>>>>>>> >>>>>>>> zookeeper.log: >>>>>>>> >>>>>>>> >>>>>>>> 2020-03-29 21:09:27,079 [myid:1] - ERROR >>>>>>>> >> [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434 >>> ] >>>>> - >>>>>>>> Unsuccessful handshake with session 0x0 >>>>>>>> >>>>>>>> 2020-03-29 21:09:27,083 [myid:1] - WARN >>>>>>>> >> [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273 >>> ] >>>>> - >>>>>>>> Exception caught >>>>>>>> >>>>>>>> io.netty.handler.codec.DecoderException: >>>>>>>> io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: >>>>>>>> 737276720a >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) >>>>>>>> >>>>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>> >>> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >>>>>>>> >>>>>>>> at java.base/java.lang.Thread.run(Thread.java:834) >>>>>>>> >>>>>>>> Caused by: io.netty.handler.ssl.NotSslRecordException: not an >> SSL/TLS >>>>>>>> record: 737276720a >>>>>>>> >>>>>>>> at >>>>>>>> >>>>> >>> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1198) >>>>>>>> >>>>>>>> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) >>>>>>>> >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) >>>>>>>> >>>>>>>> ... 17 more >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> conf/zoo.cfg: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # Server configuration >>>>>>>> >>>>>>>> secureClientPort=2281 >>>>>>>> >>>>>>>> >> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # Quorum configuration >>>>>>>> >>>>>>>> sslQuorum=true >>>>>>>> >>>>>>>> ssl.quorum.keyStore.location=</path/to/keystore.jks> >>>>>>>> >>>>>>>> ssl.quorum.keyStore.password=<password> >>>>>>>> >>>>>>>> ssl.quorum.trustStore.location=</path/to/truststore.jks> >>>>>>>> >>>>>>>> ssl.quorum.trustStore.password=<password> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> bin/zkEnv.sh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >>>>>>>> \ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.keyStore.location=</path/to/keystore.jks> \ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.keyStore.password=<password>\ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.trustStore.location=</path/to/truststore.jks> \ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.trustStore.password=<password>" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >>>>>>>> \ >>>>>>>> >>>>>>>> -Dzookeeper.client.secure=true \ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.keyStore.location=</path/to/keystore.jks> \ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.keyStore.password=<password>\ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.trustStore.location=</path/to/truststore.jks> \ >>>>>>>> >>>>>>>> -Dzookeeper.ssl.trustStore.password=<password>" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> - Karthick >>>>>>>> >>>>>>> >>>>> >>> >>> >>