[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[
https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999663#comment-16999663
]
Masatake Iwasaki commented on HADOOP-16763:
---
[~elgoiri], I should have set
{{yarn.resourcemanager.ha.curator-leader-elector.enabled}} to {{true}} to
reproduce the issue. I got the error below with zookeeper-3.5.6.jar on the
classpath of RM:
{noformat}
2019-12-19 01:45:37,149 ERROR
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
ResourceManager
java.lang.NoSuchMethodError:
org.apache.zookeeper.server.quorum.flexible.QuorumMaj.(Ljava/util/Map;)V
at
org.apache.curator.framework.imps.EnsembleTracker.(EnsembleTracker.java:57)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.(CuratorFrameworkImpl.java:159)
at
org.apache.curator.framework.CuratorFrameworkFactory$Builder.build(CuratorFrameworkFactory.java:165)
at
org.apache.hadoop.util.curator.ZKCuratorManager.start(ZKCuratorManager.java:154)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndStartZKManager(ResourceManager.java:419)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createEmbeddedElector(ResourceManager.java:385)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:333)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1576)
{noformat}
If I replace the zookeeper (client) jar on the classpath, it worked.
{noformat}
$ docker exec hadoop01 /hadoop/bin/hadoop classpath --glob | sed -z -e
's/:/\n/g' | grep zookeeper
/zookeeper/zookeeper-3.4.14.jar
$ docker exec hadoop01 /hadoop/bin/yarn rmadmin -getServiceState rm1
2019-12-19 01:52:27,597 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
active
{noformat}
If both of zookeeper-3.4.14.jar and zookeeper-3.5.6.jar are on the classpath, I
got the error above.
{noformat}
$ docker exec hadoop01 cat /hadoop/etc/hadoop/hadoop-env.sh | grep
'^export.*CLASSPATH'
export HADOOP_CLASSPATH="/zookeeper/zookeeper-3.4.14.jar"
export HADOOP_USER_CLASSPATH_FIRST="yes"
$ docker exec hadoop01 /hadoop/bin/hadoop classpath --glob | sed -z -e
's/:/\n/g' | grep zookeeper
/zookeeper/zookeeper-3.4.14.jar
/hadoop/share/hadoop/common/lib/zookeeper-3.5.6.jar
/hadoop/share/hadoop/common/lib/zookeeper-jute-3.5.6.jar
/hadoop/share/hadoop/hdfs/lib/zookeeper-3.5.6.jar
/hadoop/share/hadoop/hdfs/lib/zookeeper-jute-3.5.6.jar
{noformat}
> Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
>
>
> Key: HADOOP-16763
> URL: https://issues.apache.org/jira/browse/HADOOP-16763
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Priority: Major
>
> HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5.
> This change relate to the client libraries used by the components.
> However, the ensemble in most deployments is 3.4 (default in Ubuntu for
> example).
> To allow this mode, there is a soft-compatibility mode described in
> http://curator.apache.org/zk-compatibility.html
> We should enable this soft-compatibility mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[ https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998442#comment-16998442 ] Íñigo Goiri commented on HADOOP-16763: -- [~iwasakims] I'm testing against a default Ubuntu 18.04 installation which seems to provide a 3.4.10 ZooKeeper ensemble. Let me see if I can get a newer ZooKeeper. > Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4 > > > Key: HADOOP-16763 > URL: https://issues.apache.org/jira/browse/HADOOP-16763 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Íñigo Goiri >Priority: Major > > HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5. > This change relate to the client libraries used by the components. > However, the ensemble in most deployments is 3.4 (default in Ubuntu for > example). > To allow this mode, there is a soft-compatibility mode described in > http://curator.apache.org/zk-compatibility.html > We should enable this soft-compatibility mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[
https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997904#comment-16997904
]
Masatake Iwasaki commented on HADOOP-16763:
---
[~elgoiri] I tried today's trunk against zookeeper-3.4.14 quorum (on 3-node
docker cluster on my laptop). Both zookeeper-3.5.6.jar (bundled with hadoop
tarball) and zookeeper-3.4.14.jar as a client library worked for me.
{noformat}
$ docker exec hadoop01 /hadoop/bin/hadoop classpath --glob | sed -z -e
's/:/\n/g' | grep zookeeper
/hadoop/share/hadoop/common/lib/zookeeper-3.5.6.jar
/hadoop/share/hadoop/common/lib/zookeeper-jute-3.5.6.jar
/hadoop/share/hadoop/hdfs/lib/zookeeper-3.5.6.jar
/hadoop/share/hadoop/hdfs/lib/zookeeper-jute-3.5.6.jar
$ docker exec hadoop01 /hadoop/bin/yarn rmadmin -getServiceState rm1
active
{noformat}
{noformat}
$ docker exec hadoop01 /hadoop/bin/hadoop classpath --glob | sed -z -e
's/:/\n/g' | grep zookeeper
/zookeeper/zookeeper-3.4.14.jar
$ docker exec hadoop01 /hadoop/bin/yarn rmadmin -getServiceState rm1
active
{noformat}
> Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
>
>
> Key: HADOOP-16763
> URL: https://issues.apache.org/jira/browse/HADOOP-16763
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Priority: Major
>
> HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5.
> This change relate to the client libraries used by the components.
> However, the ensemble in most deployments is 3.4 (default in Ubuntu for
> example).
> To allow this mode, there is a soft-compatibility mode described in
> http://curator.apache.org/zk-compatibility.html
> We should enable this soft-compatibility mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[
https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997761#comment-16997761
]
Íñigo Goiri commented on HADOOP-16763:
--
I tested after HADOOP-16765 got merge and I get:
{code}
org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
...
Caused by: org.apache.zookeeper.KeeperException$UnimplementedException:
KeeperErrorCode = Unimplemented for /rmstore
at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1637)
at
org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1180)
at
org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1156)
at
org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
at
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1153)
at
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:607)
at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:597)
at
org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:362)
at
org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:310)
at
org.apache.hadoop.util.curator.ZKCuratorManager.create(ZKCuratorManager.java:286)
at
org.apache.hadoop.util.curator.ZKCuratorManager.createRootDirRecursively(ZKCuratorManager.java:319)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.startInternal(ZKRMStateStore.java:391)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:825)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
... 15 more
{code}
This message is slightly different but still does not work.
If there is no solution for this, we need to document clearly that upgrading to
Hadoop 3.3+ requires an installation of ZooKeeper 3.5+.
> Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
>
>
> Key: HADOOP-16763
> URL: https://issues.apache.org/jira/browse/HADOOP-16763
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Priority: Major
>
> HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5.
> This change relate to the client libraries used by the components.
> However, the ensemble in most deployments is 3.4 (default in Ubuntu for
> example).
> To allow this mode, there is a soft-compatibility mode described in
> http://curator.apache.org/zk-compatibility.html
> We should enable this soft-compatibility mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[ https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997509#comment-16997509 ] Íñigo Goiri commented on HADOOP-16763: -- It looks like HADOOP-16765 is proposing to do this indirectly now. > Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4 > > > Key: HADOOP-16763 > URL: https://issues.apache.org/jira/browse/HADOOP-16763 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Íñigo Goiri >Priority: Major > > HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5. > This change relate to the client libraries used by the components. > However, the ensemble in most deployments is 3.4 (default in Ubuntu for > example). > To allow this mode, there is a soft-compatibility mode described in > http://curator.apache.org/zk-compatibility.html > We should enable this soft-compatibility mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[ https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997453#comment-16997453 ] Wei-Chiu Chuang commented on HADOOP-16763: -- [~nkalmar], https://issues.apache.org/jira/browse/HADOOP-16579?focusedCommentId=16945744&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16945744 bq. Quick analysis: the ZK test failing creates a TestingServer which is implemented in Curator. Curator supposedly decides based on ZK version on the classpath if it runs in 3.5 or 3.4 mode. Somehow it looks like it decides to run in 3.5 mode even though there is a 3.4 ZK on the classpath (that's why ZK is excluded in curator's dependency). For reason we didn't figure out, Curator decided to run in 3.5 mode. To get it going we updated ZK on 3.5 too. That is to say if we create a ZK 3.4 profile, it would also change Curator version. That is fine if compatibility is preserved. > Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4 > > > Key: HADOOP-16763 > URL: https://issues.apache.org/jira/browse/HADOOP-16763 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Íñigo Goiri >Priority: Major > > HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5. > This change relate to the client libraries used by the components. > However, the ensemble in most deployments is 3.4 (default in Ubuntu for > example). > To allow this mode, there is a soft-compatibility mode described in > http://curator.apache.org/zk-compatibility.html > We should enable this soft-compatibility mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[ https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997186#comment-16997186 ] Norbert Kalmár commented on HADOOP-16763: - Curator 4 should be able to identify that there is a ZK 3.4 on the classpath, so it will run in a compatible mode with 3.4. I don't remember any issues with it. And to use it, as stated above, it is required only to exclude ZK from curator-recipes. > Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4 > > > Key: HADOOP-16763 > URL: https://issues.apache.org/jira/browse/HADOOP-16763 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Íñigo Goiri >Priority: Major > > HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5. > This change relate to the client libraries used by the components. > However, the ensemble in most deployments is 3.4 (default in Ubuntu for > example). > To allow this mode, there is a soft-compatibility mode described in > http://curator.apache.org/zk-compatibility.html > We should enable this soft-compatibility mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[ https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996922#comment-16996922 ] Akira Ajisaka commented on HADOOP-16763: We hit the same issue and we are going to upgrade our ZooKeeper cluster to 3.5.6 before upgrading Hadoop to the latest trunk, therefore we don't have experience with the soft-compatibility mode. As the document says, ZK 3.4.x dependency is required for the soft-compatibility mode, so I would like to propose creating a profile to build Hadoop with ZK 3.4.x. > Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4 > > > Key: HADOOP-16763 > URL: https://issues.apache.org/jira/browse/HADOOP-16763 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Íñigo Goiri >Priority: Major > > HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5. > This change relate to the client libraries used by the components. > However, the ensemble in most deployments is 3.4 (default in Ubuntu for > example). > To allow this mode, there is a soft-compatibility mode described in > http://curator.apache.org/zk-compatibility.html > We should enable this soft-compatibility mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[ https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996601#comment-16996601 ] Íñigo Goiri commented on HADOOP-16763: -- [~symat], [~nkalmar], [~weichiu], you guys worked on HADOOP-16579. Any thoughts on pushing this? Any of you has experience with this "soft-compatibility" mode? > Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4 > > > Key: HADOOP-16763 > URL: https://issues.apache.org/jira/browse/HADOOP-16763 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Íñigo Goiri >Priority: Major > > HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5. > This change relate to the client libraries used by the components. > However, the ensemble in most deployments is 3.4 (default in Ubuntu for > example). > To allow this mode, there is a soft-compatibility mode described in > http://curator.apache.org/zk-compatibility.html > We should enable this soft-compatibility mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[
https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996075#comment-16996075
]
Íñigo Goiri commented on HADOOP-16763:
--
I don't quite get how this works but apparently is as easy as adding an
exclusion in the curator-recipes:
{code}
org.apache.curator
curator-recipes
${curator-version}
org.apache.zookeeper
zookeeper
{code}
> Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
>
>
> Key: HADOOP-16763
> URL: https://issues.apache.org/jira/browse/HADOOP-16763
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Priority: Major
>
> HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5.
> This change relate to the client libraries used by the components.
> However, the ensemble in most deployments is 3.4 (default in Ubuntu for
> example).
> To allow this mode, there is a soft-compatibility mode described in
> http://curator.apache.org/zk-compatibility.html
> We should enable this soft-compatibility mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
