[
https://issues.apache.org/jira/browse/HBASE-23999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061099#comment-17061099
]
Nick Dimiduk commented on HBASE-23999:
--------------------------------------
Marking this as possibly related to HBASE-23957. The file path in question is
the same, and if it really is a race between tests, then the symptoms align.
> [flakey test] TestTableOutputFormatConnectionExhaust
> ----------------------------------------------------
>
> Key: HBASE-23999
> URL: https://issues.apache.org/jira/browse/HBASE-23999
> Project: HBase
> Issue Type: Test
> Components: test
> Affects Versions: 2.3.0
> Reporter: Nick Dimiduk
> Priority: Major
>
> Hit this during master startup sequence in the test.
> {noformat}
> 2020-03-16 23:40:37,298 ERROR [StoreOpener-1588230740-1]
> conf.Configuration(2980): error parsing conf hbase-site.xml
> com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
> at [row,col,system-id]:
> [1,0,"file:/home/vagrant/repos/hbase/hbase-mapreduce/target/test-classes/hbase-site.xml"]
> at
> com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOF(StreamScanner.java:687)
> at
> com.ctc.wstx.sr.BasicStreamReader.handleEOF(BasicStreamReader.java:2220)
> at
> com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2126)
> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1181)
> at
> org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3277)
> at
> org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3071)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2964)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2930)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2805)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1199)
> at
> org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1253)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1659)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:70)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:84)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at org.apache.hadoop.hbase.io.crypto.Context.<init>(Context.java:44)
> at
> org.apache.hadoop.hbase.io.crypto.Encryption$Context.<init>(Encryption.java:64)
> at
> org.apache.hadoop.hbase.io.crypto.Encryption$Context.<clinit>(Encryption.java:61)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:228)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5890)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1096)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1093)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-03-16 23:40:37,301 ERROR [master/bionic:0:becomeActiveMaster]
> regionserver.HRegion(1137): Could not initialize all stores for the
> region=hbase:meta,,1.1588230740
> {noformat}
> Looking at the file under {{target/test-classes}}, it looks like this is a
> file written by YARN.
> {noformat}
> <?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
> <property><name>yarn.log-aggregation.file-formats</name><value>TFile</value><final>false</final><source>yarn-default.xml</source></property>
> <property><name>hbase.master.mob.ttl.cleaner.period</name><value>86400</value><final>false</final><source>hbase-default.xml</source></property>
> <property><name>dfs.namenode.resource.check.interval</name><value>5000</value><final>false</final><source>hdfs-default.xml</source></property>
> <property><name>mapreduce.jobhistory.client.thread-count</name><value>10</value><final>false</final><source>mapred-default.xml</source></property>
> ...
> {noformat}
> My guess is that we have something in the MR framework unconfigured, it's
> writing these temporary job files to some default (like the first class path
> location or something??) and parallel test runs are stomping on each other.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
