[
https://issues.apache.org/jira/browse/HBASE-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128475#comment-17128475
]
Michael Stack commented on HBASE-24493:
---------------------------------------
Here is some more detail:
Fails because:
{code}
java.util.concurrent.TimeoutException: Timed out waiting for condition. Thread
diagnostics:
Timestamp: 2020-06-08 12:12:57,319
{code}
The 'Thread diagnostics' and 'Timestamp:' from above is from the Hadoop
GenericTestUtils#waitFor method.
>From around the above noted time 2020-06-08 12:12:57,319 in the logs, we see
>yarn doing jetty extraction... of jar, a bunch of guice injection, before
>starting an httpserver2 instance followed near immediately by shutdown but no
>statement on why the shutdown.
Context seems to be start of the history server in yarn. The mini yarn cluster
starts a thread to bring it up.
You cannot disable the test yarn history server. The waitFor that is waiting on
job history server to come up has hard coded 60 seconds:
GenericTestUtils.waitFor(new Supplier<Boolean>() {
@Override
public Boolean get() {
return jhsStarted;
}
}, 1500, 60_000);
... from v2/MiniMRYarnCluster.java.
We log when we start the mini cluster and when we think it up but the history
server is off starting in its own thread....
2020-06-08 12:11:26,935 INFO [Listener at localhost/38777]
hbase.HBaseTestingUtility(2886): Starting mini mapreduce cluster...
.,..
A minute could expire in here....
{code}
2020-06-08 12:12:17,008 INFO [Thread-230] log.Slf4jLog(67): Extract
jar:file:/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@3/yetus-m2/hbase-branch-2.3-full-0/org/apache/hadoop/hadoop-yarn-common/2.10.0/
hadoop-yarn-common-2.10.0.jar!/webapps/jobhistory to
/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@3/component/hbase-mapreduce/target/test-data/d16a8e72-4121-1766-eb7e-0a7e08beff63/java.io.tmpdir/
Jetty_074df066d9de_39485_jobhistory____.793npv/webapp
Jun 08, 2020 12:12:29 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices as a
root resource class
Jun 08, 2020 12:12:29 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.hs.webapp.JAXBContextResolver
as a provider class
Jun 08, 2020 12:12:29 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a
provider class
Jun 08, 2020 12:12:29 PM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
Jun 08, 2020 12:12:33 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.hs.webapp.JAXBContextResolver to
GuiceManagedComponentProvider with the scope "Singleton"
Jun 08, 2020 12:12:47 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
GuiceManagedComponentProvider with the scope "Singleton"
Jun 08, 2020 12:12:55 PM
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices to
GuiceManagedComponentProvider with the scope "PerRequest"
2020-06-08 12:12:57,160 INFO [Thread-230] log.Slf4jLog(67): Started
HttpServer2$SelectChannelConnectorWithSafeStartup@074df066d9de:39485
2020-06-08 12:12:58,289 INFO [Listener at 074df066d9de/37739]
log.Slf4jLog(67): Stopped
HttpServer2$SelectChannelConnectorWithSafeStartup@074df066d9de:0
2020-06-08 12:12:58,454 ERROR [Thread[Thread-231,5,FailOnTimeoutGroup]]
delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover(696):
ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2020-06-08 12:12:58,471 INFO [Listener at 074df066d9de/37739]
hbase.HBaseTestingUtility(1339): Shutting down minicluster
{code}
So, disabling these tests until we can disable history server or configure
things so it can have more time to startup.
> [flakey test] TestExportSnapshot family of tests failing due to timeout in
> AbstractDelegationTokenSecretManager$ExpiredTokenRemover
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-24493
> URL: https://issues.apache.org/jira/browse/HBASE-24493
> Project: HBase
> Issue Type: Test
> Components: test
> Affects Versions: 2.3.0
> Reporter: Nick Dimiduk
> Priority: Major
> Attachments:
> TEST-org.apache.hadoop.hbase.snapshot.TestExportSnapshot.xml
>
>
> I've observed another occurrence of this test timing out, over on
> https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/3/
> The failure message is cryptic, but I think i found the issue. This isn't the
> {{HBaseClassTestRule}} invoking the timeout, it's something in the test, I
> think in the mini-cluster.
> There appears to be a timeout set in some kind of secret manager, which is
> too aggressive for this mini-cluster test. The last component of the
> mini-cluster, MapReduce, is finally available at T+273501ms -- ~4.5 minutes
> after process launch. This is how I interpret the log line
> {noformat}
> 2020-06-02 03:20:49,252 INFO [Thread-223] server.Server(419): Started
> @273501ms
> {noformat}
> a scant 20ms later we get
> {noformat}
> 2020-06-02 03:20:50,274 ERROR [Thread[Thread-224,5,FailOnTimeoutGroup]]
> delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover(700):
> ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
> 2020-06-02 03:20:50,351 INFO [Time-limited test]
> hbase.HBaseTestingUtility(1272): Shutting down minicluster
> {noformat}
> These thread group names have no meaning to me.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
