Thank you, Christopher. My tests are with a cluster that uses HDFS in a Highly Available (HA) configuration. The HDFS ZKFC component (which is used in the HA configuration) has a dependency on ZK. I repeated my test with a snapshot build of Hadoop-3.3.1 and ZK 3.5.7 and it worked. I'm not sure what other impact we might expect with Hadoop 3.3 - has anyone been testing with that version?
Warm Regards, Arvind. -----Original Message----- From: Christopher <[email protected]> Sent: Thursday, May 14, 2020 10:45 AM To: fluo-dev <[email protected]> Subject: Re: [EXTERNAL] Re: JDK14 and Fluo Hadoop itself doesn't use ZooKeeper. I'm using Hadoop 3.2.1. The open ZK bug appears to be for 3.4, which is EOL. I have been actively contributing to ZK in the past few weeks, where I and others have been using and testing newer JDK versions with 3.5, 3.6, and 3.7, and fixing issues as they are observed. As far as I know, 3.5.8, 3.6.1, and 3.7.0-SNAPSHOT are fine with JDK14. On Thu, May 14, 2020 at 1:29 PM Arvind Shyamsundar <[email protected]> wrote: > > Thank you, Christopher. Can you comment on what version of Hadoop you are > using? We've been pointed by some JDK experts at Microsoft to this ZK issue: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FZOOKEEPER-3779&data=02%7C01%7Carvindsh%40microsoft.com%7C14e74746812045583ddd08d7f82e807e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637250750928449394&sdata=l88wA8zmn%2FN37mrxFNLg5vbGhidxnKaQLpoASOqziG0%3D&reserved=0. > As far as I can see, Hadoop 3.2.1 (which is what I was using) references ZK > 3.4.14, which would perhaps explain the issue. I'm going to repeat my tests > with a snapshot build of Hadoop branch-3.3 and report back. > > Stay safe everyone! > > Warm Regards, > > Arvind. > > -----Original Message----- > From: Christopher <[email protected]> > Sent: Thursday, May 14, 2020 4:42 AM > To: fluo-dev <[email protected]> > Subject: Re: [EXTERNAL] Re: JDK14 and Fluo > > Just a follow up from this. As far as I can tell, the problem I was having > was related to my system's hostname resolver, and that jdk14 works fine. > > On Wed, May 6, 2020, 18:10 Arvind Shyamsundar > <[email protected]> wrote: > > > FWIW, sharing my notes - here's what I see when attempting to setup > > an cluster (Accumulo-2.1.0-SNAPSHOT, Hadoop 3.2.1, ZooKeeper 3.5.7) > > with OpenJDK 14. The step where the Hadoop ZKFC (hdfs zkfc -formatZK > > -force) is configured fails with the below exception, which seemed > > vaguely similar to Christopher's report of Accumulo being unable to connect > > to ZK. > > > > 2020-05-06 22:00:47,518 INFO zookeeper.ClientCnxn: Opening socket > > connection to server leader-3/<unresolved>:2181. Will not attempt to > > authenticate using SASL (unknown error) > > 2020-05-06 22:00:47,518 WARN zookeeper.ClientCnxn: Session 0x0 for > > server leader-3/<unresolved>:2181, unexpected error, closing socket > > connection and attempting reconnect > > java.nio.channels.UnresolvedAddressException > > at java.base/sun.nio.ch.Net.checkAddress(Net.java:139) > > at java.base/sun.nio.ch > > .SocketChannelImpl.checkRemote(SocketChannelImpl.java:727) > > at java.base/sun.nio.ch > > .SocketChannelImpl.connect(SocketChannelImpl.java:741) > > at > > org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277) > > at > > org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287) > > at > > org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021) > > at > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064) > > > > (the above is repeated for the other 2 ZK servers configured in this > > cluster as well). The JDK item I found (again, this is all > > circumstantial, and I have not spent any time to dig deep) was > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbu > > gs > > .openjdk.java.net%2Fbrowse%2FJDK-8225499&data=02%7C01%7Carvindsh > > %4 > > 0microsoft.com%7Cf4c61036a9714c7045bb08d7f7fbe87b%7C72f988bf86f141af > > 91 > > ab2d7cd011db47%7C1%7C0%7C637250533625323572&sdata=TmSrpgDIlpxltWzH7GGduHqHARmonstwmorQ94yh0o0%3D&reserved=0. > > The associated note in JDK 14 release notes > > (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjdk.java.net%2F14%2Frelease-notes&data=02%7C01%7Carvindsh%40microsoft.com%7C14e74746812045583ddd08d7f82e807e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637250750928449394&sdata=DpfMByRjIF63FRbKn0%2F4gyyeiYqvGeMA1xgUZJmZKds%3D&reserved=0) > > has the below text, which seemed interesting as it kind of matches up with > > the symptoms above: > > > > Additionally, the string format for unresolved addresses has been changed. > > The method now represents the literal IP address with the token > > <unresolved>, for example: foo/<unresolved>:80 instead of foo:80. > > This is based on InetAddress::toString, which returns a string of > > the form "hostname / literal IP address". To retrieve a string > > representation of the hostname, or the string form of the address if > > it doesn't have a hostname, use InetSocketAddress::getHostString, > > rather than parsing the string representation. > > > > - Arvind > > > > From: Arvind Shyamsundar <[email protected]> > > Sent: Wednesday, May 6, 2020 1:09 AM > > To: [email protected] > > Subject: Re: [EXTERNAL] Re: JDK14 and Fluo > > > > Incidentally I did a brief test yesterday with Muchos and jdk 14. I > > saw similar issues with zk client (within Hadoop zkfc) being unable > > to connect to zk, complaining about an unresolved address. I did > > find a vaguely related documented change in jdk14 for ipv6 addresses > > but gave up after a while. I can send details, on Wednesday. > > Sent from Outlook Mobile<https://aka.ms/blhgte> > > > > ________________________________ > > From: Christopher <[email protected]<mailto:[email protected]>> > > Sent: Wednesday, May 6, 2020 12:50:39 AM > > To: fluo-dev <[email protected]<mailto:[email protected]>> > > Subject: [EXTERNAL] Re: JDK14 and Fluo > > > > So, a quick follow-up to this: > > > > The CMS flags seem to be ignored, so that's not really a problem: > > > > OpenJDK 64-Bit Server VM warning: Ignoring option > > UseConcMarkSweepGC; support was removed in 14.0 OpenJDK 64-Bit > > Server VM warning: Ignoring option CMSInitiatingOccupancyFraction; > > support was removed in 14.0 > > > > These are just warnings about the flags being ignored. > > > > I may be having other problems on my machine that prevent the tests > > from running. I'm not sure, but it seems that Accumulo's Initialize > > is failing to connect to the ZooKeeper server within the 30s timeout > > period. There does not appear to be any problems with ZooKeeper > > itself, and I can use telnet to send `ruok`. My machine might just > > be slow... I'll have to troubleshoot later. > > > > On Wed, May 6, 2020 at 2:52 AM Christopher <[email protected]<mailto: > > [email protected]>> wrote: > > > > > > I have switched my development environment to primarily run with > > > JDK14 and noticed some incompatibilities with Fluo. > > > > > > Some of these are now fixed in a PR: > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi > > th > > ub.com%2Fapache%2Ffluo%2Fpull%2F1093&data=02%7C01%7Carvindsh%40m > > ic > > rosoft.com%7Cf4c61036a9714c7045bb08d7f7fbe87b%7C72f988bf86f141af91ab > > 2d > > 7cd011db47%7C1%7C0%7C637250533625328563&sdata=TkVp97HUB3ugh%2FYq > > d1 > > 8asT8qJIukZxCoT8g4p8AT40M%3D&reserved=0 > > > > > > However, this doesn't completely fix JDK14 builds. Specifically, > > > JDK14 removed the Concurrent Mark Sweep garbage collector. This > > > garbage collector is hard-coded in MiniAccumuloCluster 2.0.0 and earlier. > > > Since accumulo2-maven-plugin uses MiniAccumuloCluster, the tests > > > cannot run in Fluo on JDK14. > > > > > > This issue has already been addressed in upstream Accumulo ( > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi > > th > > ub.com%2Fapache%2Faccumulo%2Fpull%2F1427&data=02%7C01%7Carvindsh > > %4 > > 0microsoft.com%7Cf4c61036a9714c7045bb08d7f7fbe87b%7C72f988bf86f141af > > 91 > > ab2d7cd011db47%7C1%7C0%7C637250533625328563&sdata=D8n09flrBYZVAw > > UV > > fYcOYNIni3Qgk%2BRuJkTK8OnX6xQ%3D&reserved=0) > > for Accumulo 2.1.0. > > > > > > My proposed resolution is that Fluo should use an updated version > > > of accumulo2-maven-plugin, after Accumulo 2.1.0 is released with > > > the MiniAccumuloCluster improvements. Of course, this requires > > > Accumulo > > > 2.1.0 to be released first. > >
