We run secure client port at 2182 and standard port 2181. This 2181 is in
the dynamic config strings. Then we have a firewalld rule to block incoming
traffic on 2181.
For us this works, no problem with Curator Ensemble tracker either if I
recall. Our Curator based clients connect fine on the secure port and we
run frequent dynamic reconfiguration.
Regards
Chris
On 18 November 2022 20:15:39 "Ruel, Ryan" <[email protected]> wrote:
Seems others have faced this same problem:
https://issues.apache.org/jira/browse/ZOOKEEPER-3577
/Ryan
On 11/18/22, 1:26 PM, "Ruel, Ryan" <[email protected]> wrote:
Enrico,
Sure, I can create a PR for any documentation change suggestions.
I am still having some trouble with this, though.
Using the dynamic reconfiguration command within the CLI tool, I was able
to specify a client port of 2281 in the server specification (the same port
as what I have in secureClientPort in zookeeper.conf). This was accepted by
the quorum and was working just fine.
However, after installing a new ZooKeeper node and initially configuring
it, it's unable to bind to 2281.
I'm running the most recent stable release (3.7.3).
My configuration is as follows:
zookeeper.conf (I have removed the irrelevant bits for brevity):
secureClientPort=2281
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.quorum.keyStore.location=/etc/zookeeper/keys/quorum/keyStore.jks
sslQuorum=true
X509AuthenticationProvider.superUser=SUPERUSER
ssl.quorum.trustStore.location=/etc/zookeeper/keys/quorum/trustStore.jks
authProvider.<our application>=<our custom authentication provider>
reconfigEnabled=true
ssl.trustStore.password=<our password>
dynamicConfigFile=/etc/zookeeper/zookeeper.conf.dynamic.3b00000019
zookeeper.conf.dynamic.3b00000019:
server.1=100.80.2.1:2888:3888:participant;0.0.0.0:2281
server.2=100.80.2.2:2888:3888:participant;0.0.0.0:2281
server.3=100.80.2.3:2888:3888:participant;0.0.0.0:2281
logs:
2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - bound
to port 2281
2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory -
binding to port 0.0.0.0/0.0.0.0:2281
2022-11-18 18:07:26,995 [main] ERROR quorum.QuorumPeerMain - Unexpected
exception, exiting abnormally
java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:459)
at java.base/sun.nio.ch.Net.bind(Net.java:448)
at
java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
at
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
at
io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
at
io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
at
io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260)
at
io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
So, it seems like it's trying to bind twice to port 2281 using Netty.
If I remove "secureClientPort" from zookeeper.conf and restart, then
ZooKeeper starts up, but as you can see in the following logs connections
exception out (presumably because TLS is disabled):
2022-11-18 18:23:57,851 [nioEventLoopGroup-4-1] WARN
server.NettyServerCnxn - Closing connection to /100.80.2.5:36609
java.io.IOException: Len error 369296129
at
org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:521)
at
org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:374)
at
org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:357)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
/Ryan
On 11/17/22, 8:16 AM, "Enrico Olivelli" <[email protected]> wrote:
Ruel,
Il giorno mer 16 nov 2022 alle ore 16:15 Ruel, Ryan
<[email protected]> ha scritto:
>
> It seems that specifying the SECURE client port in the reconfig command
does work, while also keeping the same port defined as "secureClientPort"
in zookeeper.conf.
>
> (I thought I had tried this, but may have missed this combination)
>
> In any case, some clarification within the documentation may be
helpful!
Would you like to send a PR to add these clarifications?
Thanks
Enrico
>
> /Ryan
>
> On 11/15/22, 10:10 AM, "Ruel, Ryan" <[email protected]> wrote:
>
> In my ZooKeeper setup, I am strictly using TLS for both client and
quorum communication.
>
> In zookeeper.conf, I have “secureClientPort=2281” defined, and do not
have any “clientPort” option set.
>
> In the 3.8.0 documentation on dynamic reconfiguration
(https://urldefense.com/v3/__https://zookeeper.apache.org/doc/r3.8.0/zookeeperReconfig.html__;!!GjvTz_vk!T5lRlM3A1syL82ZMBR5kWiVdxaCcKIlRhDY-6muCLvhez9gyJTYbbhMkWMiee0evVrX0MQc-eYwvZhl1$
), the documentation says that the old “clientPort” configuration option
should not be specified, and instead the new server keyword specification
should look like this:
>
> server.<positive id> = <address1>:<port1>:<port2>[:role];[<client
port address>:]<client port>**
>
> However, this specification doesn’t consider the secure client port
from what I can tell.
>
> In some cases where the server keyword is used, I can just eliminate
putting in the client port address and client port, such as within the
quorum peer configuration (in zookeeper.conf or within the dynamic
configuration file).
>
> In other cases, however, such as using the “reconfig” command in the
ZK cli utility, the client port MUST be specified, or a “bad argument” type
error is produced.
>
> I of course don’t want to put a dummy port number in the server
specification which would then enable insecure communication.
>
> What’s the recommendation for using secure communication only while
also using dynamic reconfiguration?
>
> P.S. Another interesting bit in the documentation is the example:
> server.1=125.23.63.23:2780:2783:participant;2791
> server.2=125.23.63.24:2781:2784:participant;2792
> server.3=125.23.63.25:2782:2785:participant;2793
>
> In what use case would you want to use entirely different ports for
each server? Or is this just a demonstration that this is possible?
>
> /Ryan
>
>
>
>
>
>
