dlmarion commented on PR #8348:
URL: https://github.com/apache/hadoop/pull/8348#issuecomment-4143287447
I added a test that causes an NPE in DFSStripedInputStream.useDirectBuffer.
On trunk, the test fails with the following in the log:
```
[main] INFO hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:testReadWithXceiverExhaustion(164))
- 7 exceptions occurred
[main] ERROR hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:lambda$testReadWithXceiverExhaustion$3(165))
- Exception details
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.preferDirectBuffer()"
because "this.decoder" is null
at
org.apache.hadoop.hdfs.DFSStripedInputStream.useDirectBuffer(DFSStripedInputStream.java:119)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:124)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:299)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:421)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:960)
at java.base/java.io.DataInputStream.read(DataInputStream.java:158)
at
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyStatefulRead(StripedFileTestUtil.java:128)
at
org.apache.hadoop.hdfs.TestDFSStripedInputStreamReadFailures.lambda$testReadWithXceiverExhaustion$0(TestDFSStripedInputStreamReadFailures.java:146)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
The other 6 exceptions have the same stack trace. When this test is run on
this branch it still fails, but in a different way which is probably handled
better by a client because an IOException is expected exception. The failure in
this branch looks like:
```
[main] INFO hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:testReadWithXceiverExhaustion(164))
- 9 exceptions occurred
[main] ERROR hdfs.TestDFSStripedInputStreamReadFailures
(TestDFSStripedInputStreamReadFailures.java:lambda$testReadWithXceiverExhaustion$3(165))
- Exception details
java.io.IOException: 6 missing blocks, the stripe is:
AlignedStripe(Offset=0, length=1048576, fetchedChunksNum=0,
missingChunksNum=6); locatedBlocks is: LocatedBlocks{; fileLength=6291579;
underConstruction=false;
blocks=[LocatedStripedBlock{BP-1063483206-10.113.15.120-1774623121364:blk_-9223372036854775728_1005;
getBlockSize()=6291579; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:38777,DS-5d4bafbc-f78a-4608-b4d7-635c26439ef6,DISK],
DatanodeInfoWithStorage[127.0.0.1:33155,DS-b94250d0-39ca-47b9-ae38-14b0eccc3e2a,DISK],
DatanodeInfoWithStorage[127.0.0.1:33469,DS-0ef39e40-8f52-497f-bcc4-7b74f5359200,DISK],
DatanodeInfoWithStorage[127.0.0.1:40287,DS-346798b6-b6c5-4f85-ba85-84a701ced827,DISK],
DatanodeInfoWithStorage[127.0.0.1:32995,DS-9be75b23-9ac5-4278-a46b-ee4a52f4ca2f,DISK],
DatanodeInfoWithStorage[127.0.0.1:33789,DS-e6ff79de-2b1b-48da-a84d-a8da0fba80ff,DISK],
DatanodeInfoWithStorage[127.0.0.1:46867,DS-dfa3ee1d-0964-4e1d-818d-77bdcae7c4e0,DISK],
DatanodeInfoW
ithStorage[127.0.0.1:44495,DS-8d4dd57f-c417-4010-9278-253236ea4ff3,DISK],
DatanodeInfoWithStorage[127.0.0.1:34009,DS-1af2db95-5652-4bdf-b6a8-99ef2eea080f,DISK]];
indices=[0, 1, 2, 3, 4, 5, 6, 7, 8]}];
lastLocatedBlock=LocatedStripedBlock{BP-1063483206-10.113.15.120-1774623121364:blk_-9223372036854775728_1005;
getBlockSize()=6291579; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:38777,DS-5d4bafbc-f78a-4608-b4d7-635c26439ef6,DISK],
DatanodeInfoWithStorage[127.0.0.1:33155,DS-b94250d0-39ca-47b9-ae38-14b0eccc3e2a,DISK],
DatanodeInfoWithStorage[127.0.0.1:33469,DS-0ef39e40-8f52-497f-bcc4-7b74f5359200,DISK],
DatanodeInfoWithStorage[127.0.0.1:40287,DS-346798b6-b6c5-4f85-ba85-84a701ced827,DISK],
DatanodeInfoWithStorage[127.0.0.1:32995,DS-9be75b23-9ac5-4278-a46b-ee4a52f4ca2f,DISK],
DatanodeInfoWithStorage[127.0.0.1:33789,DS-e6ff79de-2b1b-48da-a84d-a8da0fba80ff,DISK],
DatanodeInfoWithStorage[127.0.0.1:46867,DS-dfa3ee1d-0964-4e1d-818d-77bdcae7c4e0,DISK],
DatanodeInfoWithStora
ge[127.0.0.1:44495,DS-8d4dd57f-c417-4010-9278-253236ea4ff3,DISK],
DatanodeInfoWithStorage[127.0.0.1:34009,DS-1af2db95-5652-4bdf-b6a8-99ef2eea080f,DISK]];
indices=[0, 1, 2, 3, 4, 5, 6, 7, 8]}; isLastBlockComplete=true;
ecPolicy=ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs,
numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1]}
at
org.apache.hadoop.hdfs.StripeReader.checkMissingBlocks(StripeReader.java:180)
at
org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:350)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:322)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:421)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:960)
at java.base/java.io.DataInputStream.read(DataInputStream.java:158)
at
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyStatefulRead(StripedFileTestUtil.java:128)
at
org.apache.hadoop.hdfs.TestDFSStripedInputStreamReadFailures.lambda$testReadWithXceiverExhaustion$0(TestDFSStripedInputStreamReadFailures.java:146)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]