As for flume module - flume-ng is coming with commons-io 2.1 while hadoop & hbase require org.apache.commons.io.Charsets which was introduced in 2.3. Easy way is to move dependency on flume-ng after the dependencies on hbase/hadoop.
The last thing about ConcurrentHashMap - it definitely means that the code was compiled with 1.8 since 1.7 returns a simple Set while 1.8 returns KeySetView On Thu, Apr 28, 2016 at 4:08 PM, Josh Elser <[email protected]> wrote: > *tl;dr* > > * I'm removing ubuntu-us1 from all pools > * Phoenix-Flume ITs look busted > * UpsertValuesIT looks busted > * Something is weirdly wrong with Phoenix-4.x-HBase-1.1 in its entirety. > > Details below... > > It looks like we have a bunch of different reasons for the failures. > Starting with Phoenix-master: > >>>> > org.apache.phoenix.schema.NewerTableAlreadyExistsException: ERROR 1013 > (42M04): Table already exists. tableName=T > at > org.apache.phoenix.end2end.UpsertValuesIT.testBatchedUpsert(UpsertValuesIT.java:476) > <<< > > I've seen this coming out of a few different tests (I think I've also run > into it on my own, but that's another thing) > > Some of them look like the Jenkins build host is just over-taxed: > >>>> > Java HotSpot(TM) 64-Bit Server VM warning: INFO: > os::commit_memory(0x00000007e7600000, 331350016, 0) failed; error='Cannot > allocate memory' (errno=12) > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 331350016 bytes for > committing reserved memory. > # An error report file with more information is saved as: > # > /home/jenkins/jenkins-slave/workspace/Phoenix-master/phoenix-core/hs_err_pid26454.log > Java HotSpot(TM) 64-Bit Server VM warning: INFO: > os::commit_memory(0x00000007ea600000, 273678336, 0) failed; error='Cannot > allocate memory' (errno=12) > # > <<< > > and > >>>> > ------------------------------------------------------- > T E S T S > ------------------------------------------------------- > Build step 'Invoke top-level Maven targets' marked build as failure > <<< > > Both of these issues are limited to the host "ubuntu-us1". Let me just > remove him from the pool (on Phoenix-master) and see if that helps at all. > > I also see some sporadic failures of some Flume tests > >>>> > Running org.apache.phoenix.flume.PhoenixSinkIT > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 sec > <<< FAILURE! - in org.apache.phoenix.flume.PhoenixSinkIT > org.apache.phoenix.flume.PhoenixSinkIT Time elapsed: 0.004 sec <<< ERROR! > java.lang.RuntimeException: java.io.IOException: Failed to save in any > storage directories while saving namespace. > Caused by: java.io.IOException: Failed to save in any storage directories > while saving namespace. > > Running org.apache.phoenix.flume.RegexEventSerializerIT > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.005 sec > <<< FAILURE! - in org.apache.phoenix.flume.RegexEventSerializerIT > org.apache.phoenix.flume.RegexEventSerializerIT Time elapsed: 0.004 sec > <<< ERROR! > java.lang.RuntimeException: java.io.IOException: Failed to save in any > storage directories while saving namespace. > Caused by: java.io.IOException: Failed to save in any storage directories > while saving namespace. > <<< > > I'm not sure what the error message means at a glance. > > For Phoenix-HBase-1.1: > >>>> > org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError: > java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NoSuchMethodError: > java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; > at > org.apache.hadoop.hbase.master.ServerManager.findServerWithSameHostnamePortWithLock(ServerManager.java:432) > at > org.apache.hadoop.hbase.master.ServerManager.checkAndRecordNewServer(ServerManager.java:346) > at > org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:264) > at > org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:318) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) > ... 4 more > 2016-04-28 22:54:35,497 WARN [RS:0;hemera:41302] > org.apache.hadoop.hbase.regionserver.HRegionServer(2279): error telling > master we are up > com.google.protobuf.ServiceException: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): > org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError: > java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NoSuchMethodError: > java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; > at > org.apache.hadoop.hbase.master.ServerManager.findServerWithSameHostnamePortWithLock(ServerManager.java:432) > at > org.apache.hadoop.hbase.master.ServerManager.checkAndRecordNewServer(ServerManager.java:346) > at > org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:264) > at > org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:318) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) > ... 4 more > > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:318) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2269) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:356) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): > org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError: > java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NoSuchMethodError: > java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; > at > org.apache.hadoop.hbase.master.ServerManager.findServerWithSameHostnamePortWithLock(ServerManager.java:432) > at > org.apache.hadoop.hbase.master.ServerManager.checkAndRecordNewServer(ServerManager.java:346) > at > org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:264) > at > org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:318) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) > ... 4 more > > at > org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1235) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217) > ... 13 more > <<< > > We have hit-or-miss on this error message which keeps hbase:namespace from > being assigned (as the RS's can never report into the hmaster). This is > happening across a couple of the nodes (ubuntu-[3,4,6]). I had tried to look > into this one over the weekend (and was lead to a JDK8 built jar, running on > JDK7), but if I look at META-INF/MANIFEST.mf in the hbase-server-1.1.3.jar > from central, I see it was built with 1.7.0_80 (which I think means the JDK8 > thought is a red-herring). I'm really confused by this one, actually. > Something must be amiss here. > > For Phoenix-HBase-1.0: > > We see the same Phoenix-Flume failures, UpsertValuesIT failure, and timeouts > on ubuntu-us1. There is one crash on H10, but that might just be bad luck. > > For Phoenix-HBase-0.98: > > Same UpsertValuesIT failure and failures on ubuntu-us1. > > > James Taylor wrote: >> >> Anyone know why our Jenkins builds keep failing? Is it environmental and >> is >> there anything we can do about it? >> >> Thanks, >> James >> >
