I found this while looking at Apache Lucene/Solr and Hadoop 3. Hadoop
uses FastDateFormat to format the current timestamp. Apache
Lucene/Solr randomizes locales to ensure that things behave correctly
even when there are different locales being used. There have been a
few failures that have the following stack trace:
ava.lang.ArrayIndexOutOfBoundsException: 4
[junit4] 2> at
org.apache.commons.lang3.time.FastDatePrinter$TextField.appendTo(FastDatePrinter.java:901)
~[commons-lang3-3.7.jar:3.7]
[junit4] 2> at
org.apache.commons.lang3.time.FastDatePrinter.applyRules(FastDatePrinter.java:573)
~[commons-lang3-3.7.jar:3.7]
[junit4] 2> at
org.apache.commons.lang3.time.FastDatePrinter.applyRulesToString(FastDatePrinter.java:455)
~[commons-lang3-3.7.jar:3.7]
[junit4] 2> at
org.apache.commons.lang3.time.FastDatePrinter.format(FastDatePrinter.java:446)
~[commons-lang3-3.7.jar:3.7]
[junit4] 2> at
org.apache.commons.lang3.time.FastDateFormat.format(FastDateFormat.java:428)
~[commons-lang3-3.7.jar:3.7]
[junit4] 2> at
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.start(DirectoryScanner.java:281)
~[hadoop-hdfs-3.2.0.jar:?]
[junit4] 2> at
org.apache.hadoop.hdfs.server.datanode.DataNode.initDirectoryScanner(DataNode.java:1090)
~[hadoop-hdfs-3.2.0.jar:?]
[junit4] 2> at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1686)
~[hadoop-hdfs-3.2.0.jar:?]
[junit4] 2> at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:390)
~[hadoop-hdfs-3.2.0.jar:?]
[junit4] 2> at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280)
~[hadoop-hdfs-3.2.0.jar:?]
[junit4] 2> at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:819)
[hadoop-hdfs-3.2.0.jar:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
I was also able to reproduce this with a simple test case:
long timestamp = System.currentTimeMillis();
Locale.setDefault(Locale.forLanguageTag("ja-JP-u-ca-japanese-x-lvariant-JP"));
Assert.assertEquals(SimpleDateFormat.getInstance().format(timestamp),
FastDateFormat.getInstance().format(timestamp));
Showing that the issue isn't with Hadoop but with commons-lang3
specifically. SimpleDateFormat has no issue formatting the timestamp
with the given locale. The FastDateFormat javadoc doesn't state any
issues with locales.
Is this to be expected?
Kevin Risden
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
