[
https://issues.apache.org/jira/browse/HDFS-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115099#comment-14115099
]
Steve Loughran commented on HDFS-6974:
--------------------------------------
Core stack
{code}
4-192.168.1.138-1409307851902:blk_1073741825_1001]] ERROR datanode.DataNode
(DataXceiver.java:run(243)) - 127.0.0.1:52841:DataXceiver
error processing WRITE_BLOCK operation src: /127.0.0.1:52858 dst:
/127.0.0.1:52841
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
at
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native
Method)
at
org.apache.hadoop.util.NativeCrc32.verifyChunkedSumsByteArray(NativeCrc32.java:67)
at
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:344)
at
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:292)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.verifyChunks(BlockReceiver.java:416)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:551)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:771)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:718)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:225)
at java.lang.Thread.run(Thread.java:745)
2014-08-29 11:24:15,800 [ResponseProcessor for block
BP-126326924-192.168.1.138-1409307851902:blk_1073741825_1001] WARN hdfs.DFSClie
nt (DFSOutputStream.java:run(880)) - DFSOutputStream ResponseProcessor
exception for block BP-126326924-192.168.1.138-1409307851902:
blk_1073741825_1001
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2081)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:79
{code}
I can work around this in my tests by simply disabling JNI load, but it could
be a problem that re-occurs elsewhere. The native lib is loading (so the flag
is set), but the operations downstream are failing in more obscure ways.
What to do?
# nothing. This JIRA and stack trace can be used to identify the problem and
fix (get rid of that lib/change the {{"java.library.path"}})
# do some post-load tests ... are all the methods there? But if not: what to do?
# include some version numbering on the library, so the first version of
{{hadoop.dll}}, or {{hadoop.lib}} isn't blindly picked up.
I'd argue for action 3, numbering the native lib with the Hadoop version built.
> MiniHDFScluster breaks if there is an out of date hadoop.lib on the lib path
> -----------------------------------------------------------------------------
>
> Key: HDFS-6974
> URL: https://issues.apache.org/jira/browse/HDFS-6974
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.6.0
> Environment: Windows with a version of Hadoop (HDP2.1) installed
> somewhere via an MSI
> Reporter: Steve Loughran
> Priority: Minor
>
> SLIDER-377 shows the trace of a MiniHDFSCluster test failing on native
> library calls ... the root cause appears to be the 2.4.1 hadoop lib on the
> path doesn't have all the methods needed by branch-2.
> When this situation arises, MiniHDFS cluster fails to work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
