[ https://issues.apache.org/jira/browse/PHOENIX-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Geoffrey Jacoby resolved PHOENIX-4216. -------------------------------------- Fix Version/s: (was: 4.17.0) (was: 4.16.2) Resolution: Done Associated HBase JIRA has been worked, and I don't think this error happens anymore > Figure out why tests randomly fail with master not able to initialize in 200 > seconds > ------------------------------------------------------------------------------------ > > Key: PHOENIX-4216 > URL: https://issues.apache.org/jira/browse/PHOENIX-4216 > Project: Phoenix > Issue Type: Bug > Affects Versions: 5.0.0, 4.15.0, 4.14.3 > Reporter: Samarth Jain > Priority: Major > Labels: phoenix-hardening, precommit, quality-improvement > Fix For: 5.2.0 > > Attachments: Precommit-3849.log > > > Sample failure: > [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1450//testReport/] > [~apurtell] - Looking at the thread dump in the above link, do you see why > master startup failed? I couldn't see any obvious deadlocks > > Exception stacktrace: > org.apache.hadoop.hbase.regionserver.HRegionServer(2414): Master rejected > startup because clock is out of > syncorg.apache.hadoop.hbase.regionserver.HRegionServer(2414): Master rejected > startup because clock is out of > syncorg.apache.hadoop.hbase.ClockOutOfSyncException: > org.apache.hadoop.hbase.ClockOutOfSyncException: Server > 2a3b1691db3a,42899,1590685404919 has been rejected; Reported time is too far > out of sync with master. Time difference of 1590685396313ms > max allowed of > 30000ms at > org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:411) > at > org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:277) > at > org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:368) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2417) at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:186) at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:166) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:95) > at > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:85) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:372) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:331) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2412) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:960) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:360) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1744) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:334) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:139) > at java.lang.Thread.run(Thread.java:748)Caused by: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): > org.apache.hadoop.hbase.ClockOutOfSyncException: Server > 2a3b1691db3a,42899,1590685404919 has been rejected; Reported time is too far > out of sync with master. Time difference of 1590685396313ms > max allowed of > 30000ms at > org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:411) > at > org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:277) > at > org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:368) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2417) at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:186) at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:166) > at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1291) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:231) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340) > at > org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2410) > ... 10 more -- This message was sent by Atlassian Jira (v8.20.7#820007)