Re: default value for quorum.auth.kerberos.servicePrincipal

[Rakesh Radhakrishnan]

As the name says, "quorum.auth.kerberos.servicePrincipal" property is
specifically for Kerberos based quorum authentication and no need to set
anything if you are enabling digest-md5.

Like mentioned earlier, its default value is "zkquorum/localhost" and it
will never be used if you configure/enable digest-md5.

Thanks,
Rakesh

On Mon, Dec 16, 2019 at 7:14 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> "quorum.auth.kerberos.servicePrincipal" this one
>
> On Sun, Dec 15, 2019, 9:33 PM Rakesh Radhakrishnan 
> wrote:
>
> > OK, got it.
> >
> >  Even if i enable sasl but md5-diget what should be this property set
> > to,
> > Could you please name the specific property you are referring.
> >
> > Hope you are talking about "DIGEST-MD5" mechanism ? String[] mechs = {
> > "DIGEST-MD5" };
> >
> > Presently the execution flow is that, if there is
> > no subject.getPrincipals() in jaas config then it must not be GSSAPI and
> > fallback to check DIGEST-MD5 details in jaas config.
> > Whenever user want to enable DIGEST-MD5, they have to define the JAAS
> > configuration file with DIGEST-MD5 configs like below and there is no
> > default value for this mechanism.
> >  QuorumServer {
> >org.apache.zookeeper.server.auth.DigestLoginModule required
> >user_test1="mypassword";
> >  };
> >
> > QuorumLearner {
> >org.apache.zookeeper.server.auth.DigestLoginModule required
> >user_test2=" mypassword";
> >  };
> >
> > Populate DIGEST-MD5 user -> password map for the "QuorumServer",
> > "QuorumLearner" section.
> > Usernames are distinguished from other options by prefixing the username
> > with a "user_" prefix.
> >
> > Hope its clear to you.
> >
> > Thanks,
> > Rakesh
> >
> > On Fri, Dec 13, 2019 at 9:45 PM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi Rakesh,
> > >
> > > Right now i am not enabling sasl but i am trying to define all default
> > > properties and should be able to use them once sasl is enabled with
> > > override values. So my question is for digest auth do we even need this
> > > property? i remember seeing i don't set that property it was using the
> > > default value "zkquorum/localhost".
> > >
> > > Thanks,
> > > Ram
> > >
> > > On Thu, Dec 12, 2019 at 11:06 PM Rakesh Radhakrishnan <
> > rake...@apache.org>
> > > wrote:
> > >
> > > > Hi Ram,
> > > >
> > > > ZooKeeper Quorum authentication support two schemes, Kerberos or
> > > > DIGEST-MD5. User has to configure either Kerb or digest configuration
> > > > values. Both together not required.
> > > >
> > > > I'd recommend you to go through Kerberos, digest simulation unit test
> > > cases
> > > > where we have valid and invalid scenarios. Hope this would get idea
> > about
> > > > the required configs.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
> > > >
> > > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
> > > >
> > > > Could you describe the issues that troubles you in setting up quorum
> > > auth,
> > > > if any.
> > > >
> > > > Thanks,
> > > > Rakesh
> > > >
> > > > On Fri, Dec 13, 2019 at 3:49 AM rammohan ganapavarapu <
> > > > rammohanga...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Even if i enable sasl but md5-diget what should be this property
> set
> > > to,
> > > > > this property only take effect for kerberos or for both?
> > > > >
> > > > > Ram
> > > > >
> > > > > On Fri, Dec 6, 2019 at 7:55 AM rammohan ganapavarapu <
> > > > > rammohanga...@gmail.com> wrote:
> > > > >
> > > > > > Mate,
> > > > > >
> > > > > > Thank you, I did search source code found the same, I am trying
> to
> > > > create
> > > > > > a zoo conf with all default properties.
> > > > > >
> > > > > > Ram
> > > > > >
> > > > > > On Fri, Dec 6, 2019, 2:44 AM Mate Szalay-Beko
> > > > > 
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Ram,
> > > > > >>
> > > > > >> this parameter is needed to be defined when you want to enable
> > > secure
> > > > > >> authentication in the communication between ZooKeeper servers.
> In
> > > > > general,
> > > > > >> the 'principal' is a 'username' what you want your ZooKeeper
> > servers
> > > > to
> > > > > >> use
> > > > > >> when they talk with each other. Ideally you have a central
> Kereros
> > > > > service
> > > > > >> somewhere where this principal is already registered.
> > > > > >> A kerberos principal is usually in the form of
> > > > > >> "user_or_service_name/host@realm" (some more explanation:
> > > > > >> https://ssimo.org/blog/id_016.html)
> > > > > >>
> > > > > >> According to the source code, the default value of
> > > > > >> quorum.auth.kerberos.servicePrincipal is "zkquorum/localhost".
> > But I
> > > > > think
> > > > > >> if you don't enable the quorum SASL in ZooKe

Re: Zookeeper 3.5 SSL and Kerberos authentication

[Jörn Franke]

Thanks a lot for the invesrtigation. I did not find again the time to
install it somewhere. I observed the issue with 3.5.5

On Tue, Dec 17, 2019 at 6:58 PM Andor Molnar  wrote:

> "We were using early 3.5.3 or something like that.”
>
> Netty stack had a major refactor in 3.5.5
>
> Andor
>
>
>
> > On 2019. Dec 17., at 16:40, Enrico Olivelli  wrote:
> >
> > Il giorno mar 17 dic 2019 alle ore 16:26 Szalay-Bekő Máté <
> > szalay.beko.m...@gmail.com> ha scritto:
> >
> >> I added a comment on Jira. This is something we will also need to fix
> in my
> >> company soon.
> >>
> >> @enrico you wrote:
> >>> in my company we set up some ZK with TLS and SASL, using TLS for
> >> encryption and SASL for auth. We were using early 3.5.3 or something
> like
> >> that.
> >>
> >
> > Unfortunately we do not have that setup anymore, we had to drop it
> because
> > at that time (and still nowadays) from the same JVMs we had also to
> connect
> > to an HBase cluster with ZK 3.4
> > that does not support TLS.
> >
> > Currently we are using only SASL and not TLS
> > Sorry
> >
> > Enrico
> >
> >
> >>
> >> According to this, the scenario should work. Maybe we just misconfigured
> >> something, or this was something got broken in a later version? Can you
> >> share the config you use? Maybe you are setting
> `zookeeper.ssl.clientAuth`
> >> and `zookeeper.ssl.quorum.clientAuth` to `none` or `want` ?
> >>
> >> Regards,
> >> Mate
> >>
> >> On Tue, Dec 17, 2019 at 10:48 AM Andor Molnar  wrote:
> >>
> >>> Hi Jorn,
> >>>
> >>> Sorry for coming back late to this. I’ve just validated the scenario on
> >> my
> >>> test cluster. Looks like the issue is valid: Kerberos auth and SSL are
> >>> mutually exclusive currently. When Kerberos is set up and trying to
> >> connect
> >>> to secure port I got an infinite loop on client side:
> >>>
> >>> 2019-12-17 01:43:30,984 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> WARN
> >>> [Thread-39:Login$1@197] - TGT renewal thread has been interrupted and
> >>> will exit.
> >>> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182):Login@302] -
> Client
> >>> successfully logged in.
> >>> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [Thread-40:Login$1@135] - TGT refresh thread started.
> >>> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> ):SecurityUtils$1@124
> >> ]
> >>> - Client will use GSSAPI as SASL mechanism.
> >>> 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> >>> ):ClientCnxn$SendThread@1112] - Opening socket connection to server
> >>> barbaresco-1.vpc.cloudera.com/10.65.25.98:2182. Will attempt to
> >>> SASL-authenticate using Login Context section 'Client'
> >>> 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> >>> ):ClientCnxn$SendThread@959] - Socket connection established,
> initiating
> >>> session, client: /10.65.25.98:45362, server:
> >>> barbaresco-1.vpc.cloudera.com/10.65.25.98:2182
> >>> 2019-12-17
> >>> 
> >>> 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> >>> [Thread-40:Login@320] - TGT valid starting at:Tue Dec 17
> >> 01:43:30
> >>> PST 2019
> >>> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [Thread-40:Login@321] - TGT expires:  Thu Jan 16
> >> 01:43:30
> >>> PST 2020
> >>> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [Thread-40:Login$1@193] - TGT refresh sleeping until: Fri Jan 10
> >> 20:23:33
> >>> PST 2020
> >>> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] -
> INFO
> >>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> >>> ):ClientCnxn$SendThread@1240] - Unable to read additional data from
> >>> server sessionid 0x0, likely server has closed socket, closing socket
> >>> connection and attempting reconnect
> >>>
> >>> And the following error on server side:
> >>>
> >>> 2019-12-17 01:43:33,002 INFO
> >>> org.apache.zookeeper.server.NettyServerCnxnFactory: SSL handler added
> for
> >>> channel: [id: 0xcf37c14b, L:/10.65.25.98:2182 - R:/10.65.25.98:45380]
> >>> 2019-12-17 01:43:33,003 ERROR
> >>> org.apache.zookeeper.server.NettyServerCnxnFactory: Unsuccessful
> >> handshake
> >>> with session 0x0
> >>> 2019-12-17 01:43:33,003 WARN
> >>> org.apache.zookeeper.server.NettyServerCnxnFactory: Exception caught
> >>> io.netty.handler.codec.DecoderException:
> >>> io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
> >>>
> >>
> 002d7530001000
> >>>at
> >>>
> >>
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(Byte

Re: Zookeeper 3.5 SSL and Kerberos authentication

[Andor Molnar]

"We were using early 3.5.3 or something like that.”

Netty stack had a major refactor in 3.5.5

Andor



> On 2019. Dec 17., at 16:40, Enrico Olivelli  wrote:
> 
> Il giorno mar 17 dic 2019 alle ore 16:26 Szalay-Bekő Máté <
> szalay.beko.m...@gmail.com> ha scritto:
> 
>> I added a comment on Jira. This is something we will also need to fix in my
>> company soon.
>> 
>> @enrico you wrote:
>>> in my company we set up some ZK with TLS and SASL, using TLS for
>> encryption and SASL for auth. We were using early 3.5.3 or something like
>> that.
>> 
> 
> Unfortunately we do not have that setup anymore, we had to drop it because
> at that time (and still nowadays) from the same JVMs we had also to connect
> to an HBase cluster with ZK 3.4
> that does not support TLS.
> 
> Currently we are using only SASL and not TLS
> Sorry
> 
> Enrico
> 
> 
>> 
>> According to this, the scenario should work. Maybe we just misconfigured
>> something, or this was something got broken in a later version? Can you
>> share the config you use? Maybe you are setting `zookeeper.ssl.clientAuth`
>> and `zookeeper.ssl.quorum.clientAuth` to `none` or `want` ?
>> 
>> Regards,
>> Mate
>> 
>> On Tue, Dec 17, 2019 at 10:48 AM Andor Molnar  wrote:
>> 
>>> Hi Jorn,
>>> 
>>> Sorry for coming back late to this. I’ve just validated the scenario on
>> my
>>> test cluster. Looks like the issue is valid: Kerberos auth and SSL are
>>> mutually exclusive currently. When Kerberos is set up and trying to
>> connect
>>> to secure port I got an infinite loop on client side:
>>> 
>>> 2019-12-17 01:43:30,984 [myid:barbaresco-1.vpc.cloudera.com:2182] - WARN
>>> [Thread-39:Login$1@197] - TGT renewal thread has been interrupted and
>>> will exit.
>>> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182):Login@302] - Client
>>> successfully logged in.
>>> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [Thread-40:Login$1@135] - TGT refresh thread started.
>>> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182):SecurityUtils$1@124
>> ]
>>> - Client will use GSSAPI as SASL mechanism.
>>> 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
>>> ):ClientCnxn$SendThread@1112] - Opening socket connection to server
>>> barbaresco-1.vpc.cloudera.com/10.65.25.98:2182. Will attempt to
>>> SASL-authenticate using Login Context section 'Client'
>>> 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
>>> ):ClientCnxn$SendThread@959] - Socket connection established, initiating
>>> session, client: /10.65.25.98:45362, server:
>>> barbaresco-1.vpc.cloudera.com/10.65.25.98:2182
>>> 2019-12-17
>>> 
>>> 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [Thread-40:Login@320] - TGT valid starting at:Tue Dec 17
>> 01:43:30
>>> PST 2019
>>> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [Thread-40:Login@321] - TGT expires:  Thu Jan 16
>> 01:43:30
>>> PST 2020
>>> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [Thread-40:Login$1@193] - TGT refresh sleeping until: Fri Jan 10
>> 20:23:33
>>> PST 2020
>>> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
>>> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
>>> ):ClientCnxn$SendThread@1240] - Unable to read additional data from
>>> server sessionid 0x0, likely server has closed socket, closing socket
>>> connection and attempting reconnect
>>> 
>>> And the following error on server side:
>>> 
>>> 2019-12-17 01:43:33,002 INFO
>>> org.apache.zookeeper.server.NettyServerCnxnFactory: SSL handler added for
>>> channel: [id: 0xcf37c14b, L:/10.65.25.98:2182 - R:/10.65.25.98:45380]
>>> 2019-12-17 01:43:33,003 ERROR
>>> org.apache.zookeeper.server.NettyServerCnxnFactory: Unsuccessful
>> handshake
>>> with session 0x0
>>> 2019-12-17 01:43:33,003 WARN
>>> org.apache.zookeeper.server.NettyServerCnxnFactory: Exception caught
>>> io.netty.handler.codec.DecoderException:
>>> io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
>>> 
>> 002d7530001000
>>>at
>>> 
>> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475)
>>>at
>>> 
>> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
>>>at
>>> 
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
>>>at
>>> 
>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
>>>at
>>> 
>> io.netty.c

Re: Zookeeper 3.5 SSL and Kerberos authentication

[Enrico Olivelli]

Il giorno mar 17 dic 2019 alle ore 16:26 Szalay-Bekő Máté <
szalay.beko.m...@gmail.com> ha scritto:

> I added a comment on Jira. This is something we will also need to fix in my
> company soon.
>
> @enrico you wrote:
> > in my company we set up some ZK with TLS and SASL, using TLS for
> encryption and SASL for auth. We were using early 3.5.3 or something like
> that.
>

Unfortunately we do not have that setup anymore, we had to drop it because
at that time (and still nowadays) from the same JVMs we had also to connect
to an HBase cluster with ZK 3.4
that does not support TLS.

Currently we are using only SASL and not TLS
Sorry

Enrico


>
> According to this, the scenario should work. Maybe we just misconfigured
> something, or this was something got broken in a later version? Can you
> share the config you use? Maybe you are setting `zookeeper.ssl.clientAuth`
> and `zookeeper.ssl.quorum.clientAuth` to `none` or `want` ?
>
> Regards,
> Mate
>
> On Tue, Dec 17, 2019 at 10:48 AM Andor Molnar  wrote:
>
> > Hi Jorn,
> >
> > Sorry for coming back late to this. I’ve just validated the scenario on
> my
> > test cluster. Looks like the issue is valid: Kerberos auth and SSL are
> > mutually exclusive currently. When Kerberos is set up and trying to
> connect
> > to secure port I got an infinite loop on client side:
> >
> > 2019-12-17 01:43:30,984 [myid:barbaresco-1.vpc.cloudera.com:2182] - WARN
> > [Thread-39:Login$1@197] - TGT renewal thread has been interrupted and
> > will exit.
> > 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [main-SendThread(barbaresco-1.vpc.cloudera.com:2182):Login@302] - Client
> > successfully logged in.
> > 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [Thread-40:Login$1@135] - TGT refresh thread started.
> > 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [main-SendThread(barbaresco-1.vpc.cloudera.com:2182):SecurityUtils$1@124
> ]
> > - Client will use GSSAPI as SASL mechanism.
> > 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> > ):ClientCnxn$SendThread@1112] - Opening socket connection to server
> > barbaresco-1.vpc.cloudera.com/10.65.25.98:2182. Will attempt to
> > SASL-authenticate using Login Context section 'Client'
> > 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> > ):ClientCnxn$SendThread@959] - Socket connection established, initiating
> > session, client: /10.65.25.98:45362, server:
> > barbaresco-1.vpc.cloudera.com/10.65.25.98:2182
> > 2019-12-17
> > 
> > 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [Thread-40:Login@320] - TGT valid starting at:Tue Dec 17
> 01:43:30
> > PST 2019
> > 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [Thread-40:Login@321] - TGT expires:  Thu Jan 16
> 01:43:30
> > PST 2020
> > 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [Thread-40:Login$1@193] - TGT refresh sleeping until: Fri Jan 10
> 20:23:33
> > PST 2020
> > 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> > [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> > ):ClientCnxn$SendThread@1240] - Unable to read additional data from
> > server sessionid 0x0, likely server has closed socket, closing socket
> > connection and attempting reconnect
> >
> > And the following error on server side:
> >
> > 2019-12-17 01:43:33,002 INFO
> > org.apache.zookeeper.server.NettyServerCnxnFactory: SSL handler added for
> > channel: [id: 0xcf37c14b, L:/10.65.25.98:2182 - R:/10.65.25.98:45380]
> > 2019-12-17 01:43:33,003 ERROR
> > org.apache.zookeeper.server.NettyServerCnxnFactory: Unsuccessful
> handshake
> > with session 0x0
> > 2019-12-17 01:43:33,003 WARN
> > org.apache.zookeeper.server.NettyServerCnxnFactory: Exception caught
> > io.netty.handler.codec.DecoderException:
> > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
> >
> 002d7530001000
> > at
> >
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475)
> > at
> >
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
> > at
> >
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> > at
> >
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> > at
> >
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> > at
> >
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
> > at
> >
> i

Re: Zookeeper 3.5 SSL and Kerberos authentication

[Szalay-Bekő Máté]

I added a comment on Jira. This is something we will also need to fix in my
company soon.

@enrico you wrote:
> in my company we set up some ZK with TLS and SASL, using TLS for
encryption and SASL for auth. We were using early 3.5.3 or something like
that.

According to this, the scenario should work. Maybe we just misconfigured
something, or this was something got broken in a later version? Can you
share the config you use? Maybe you are setting `zookeeper.ssl.clientAuth`
and `zookeeper.ssl.quorum.clientAuth` to `none` or `want` ?

Regards,
Mate

On Tue, Dec 17, 2019 at 10:48 AM Andor Molnar  wrote:

> Hi Jorn,
>
> Sorry for coming back late to this. I’ve just validated the scenario on my
> test cluster. Looks like the issue is valid: Kerberos auth and SSL are
> mutually exclusive currently. When Kerberos is set up and trying to connect
> to secure port I got an infinite loop on client side:
>
> 2019-12-17 01:43:30,984 [myid:barbaresco-1.vpc.cloudera.com:2182] - WARN
> [Thread-39:Login$1@197] - TGT renewal thread has been interrupted and
> will exit.
> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182):Login@302] - Client
> successfully logged in.
> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [Thread-40:Login$1@135] - TGT refresh thread started.
> 2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182):SecurityUtils$1@124]
> - Client will use GSSAPI as SASL mechanism.
> 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> ):ClientCnxn$SendThread@1112] - Opening socket connection to server
> barbaresco-1.vpc.cloudera.com/10.65.25.98:2182. Will attempt to
> SASL-authenticate using Login Context section 'Client'
> 2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> ):ClientCnxn$SendThread@959] - Socket connection established, initiating
> session, client: /10.65.25.98:45362, server:
> barbaresco-1.vpc.cloudera.com/10.65.25.98:2182
> 2019-12-17
> 
> 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [Thread-40:Login@320] - TGT valid starting at:Tue Dec 17 01:43:30
> PST 2019
> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [Thread-40:Login@321] - TGT expires:  Thu Jan 16 01:43:30
> PST 2020
> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [Thread-40:Login$1@193] - TGT refresh sleeping until: Fri Jan 10 20:23:33
> PST 2020
> 2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO
> [main-SendThread(barbaresco-1.vpc.cloudera.com:2182
> ):ClientCnxn$SendThread@1240] - Unable to read additional data from
> server sessionid 0x0, likely server has closed socket, closing socket
> connection and attempting reconnect
>
> And the following error on server side:
>
> 2019-12-17 01:43:33,002 INFO
> org.apache.zookeeper.server.NettyServerCnxnFactory: SSL handler added for
> channel: [id: 0xcf37c14b, L:/10.65.25.98:2182 - R:/10.65.25.98:45380]
> 2019-12-17 01:43:33,003 ERROR
> org.apache.zookeeper.server.NettyServerCnxnFactory: Unsuccessful handshake
> with session 0x0
> 2019-12-17 01:43:33,003 WARN
> org.apache.zookeeper.server.NettyServerCnxnFactory: Exception caught
> io.netty.handler.codec.DecoderException:
> io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
> 002d7530001000
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)
> at
> io.netty.channe

Re: Zookeeper 3.5 SSL and Kerberos authentication

[Andor Molnar]

Hi Jorn,

Sorry for coming back late to this. I’ve just validated the scenario on my test 
cluster. Looks like the issue is valid: Kerberos auth and SSL are mutually 
exclusive currently. When Kerberos is set up and trying to connect to secure 
port I got an infinite loop on client side:

2019-12-17 01:43:30,984 [myid:barbaresco-1.vpc.cloudera.com:2182] - WARN  
[Thread-39:Login$1@197] - TGT renewal thread has been interrupted and will exit.
2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):Login@302] - Client 
successfully logged in.
2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[Thread-40:Login$1@135] - TGT refresh thread started.
2019-12-17 01:43:30,987 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):SecurityUtils$1@124] - 
Client will use GSSAPI as SASL mechanism.
2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):ClientCnxn$SendThread@1112]
 - Opening socket connection to server 
barbaresco-1.vpc.cloudera.com/10.65.25.98:2182. Will attempt to 
SASL-authenticate using Login Context section 'Client'
2019-12-17 01:43:30,988 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):ClientCnxn$SendThread@959] 
- Socket connection established, initiating session, client: 
/10.65.25.98:45362, server: barbaresco-1.vpc.cloudera.com/10.65.25.98:2182
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[Thread-40:Login@320] - TGT valid starting at:Tue Dec 17 01:43:30 PST 
2019
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[Thread-40:Login@321] - TGT expires:  Thu Jan 16 01:43:30 PST 
2020
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[Thread-40:Login$1@193] - TGT refresh sleeping until: Fri Jan 10 20:23:33 PST 
2020
2019-12-17 01:43:30,989 [myid:barbaresco-1.vpc.cloudera.com:2182] - INFO  
[main-SendThread(barbaresco-1.vpc.cloudera.com:2182):ClientCnxn$SendThread@1240]
 - Unable to read additional data from server sessionid 0x0, likely server has 
closed socket, closing socket connection and attempting reconnect

And the following error on server side:

2019-12-17 01:43:33,002 INFO 
org.apache.zookeeper.server.NettyServerCnxnFactory: SSL handler added for 
channel: [id: 0xcf37c14b, L:/10.65.25.98:2182 - R:/10.65.25.98:45380]
2019-12-17 01:43:33,003 ERROR 
org.apache.zookeeper.server.NettyServerCnxnFactory: Unsuccessful handshake with 
session 0x0
2019-12-17 01:43:33,003 WARN 
org.apache.zookeeper.server.NettyServerCnxnFactory: Exception caught
io.netty.handler.codec.DecoderException: 
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 
002d7530001000
at 
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at 
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at 
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at 
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

I will update the Jira too.

Andor





> On 2019. Nov 8., at 20:31, Jörn Franke  wrote:
> 
> Thanks. Can you please share the configuration file?
> 
> I tried with 3.5.5 - without SSL Kerberos works, but once I configured client 
> ssl it said authentication fail (I have to check if I can dig up the log 
> files) and as far as I remember this was related to x509 authentication. The 
> cert