dlmarion commented on PR #8348:
URL: https://github.com/apache/hadoop/pull/8348#issuecomment-4143287447

   I added a test that causes an NPE in DFSStripedInputStream.useDirectBuffer. 
On trunk, the test fails with the following in the log:
   ```
   [main] INFO  hdfs.TestDFSStripedInputStreamReadFailures 
(TestDFSStripedInputStreamReadFailures.java:testReadWithXceiverExhaustion(164)) 
- 7 exceptions occurred
   [main] ERROR hdfs.TestDFSStripedInputStreamReadFailures 
(TestDFSStripedInputStreamReadFailures.java:lambda$testReadWithXceiverExhaustion$3(165))
 - Exception details
   java.lang.NullPointerException: Cannot invoke 
"org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.preferDirectBuffer()"
 because "this.decoder" is null
           at 
org.apache.hadoop.hdfs.DFSStripedInputStream.useDirectBuffer(DFSStripedInputStream.java:119)
           at 
org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:124)
           at 
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:299)
           at 
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:421)
           at 
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:960)
           at java.base/java.io.DataInputStream.read(DataInputStream.java:158)
           at 
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyStatefulRead(StripedFileTestUtil.java:128)
           at 
org.apache.hadoop.hdfs.TestDFSStripedInputStreamReadFailures.lambda$testReadWithXceiverExhaustion$0(TestDFSStripedInputStreamReadFailures.java:146)
           at java.base/java.lang.Thread.run(Thread.java:1583)
   ```
   
   The other 6 exceptions have the same stack trace. When this test is run on 
this branch it still fails, but in a different way which is probably handled 
better by a client because an IOException is expected exception. The failure in 
this branch looks like:
   
   ```
   [main] INFO  hdfs.TestDFSStripedInputStreamReadFailures 
(TestDFSStripedInputStreamReadFailures.java:testReadWithXceiverExhaustion(164)) 
- 9 exceptions occurred
   [main] ERROR hdfs.TestDFSStripedInputStreamReadFailures 
(TestDFSStripedInputStreamReadFailures.java:lambda$testReadWithXceiverExhaustion$3(165))
 - Exception details
   java.io.IOException: 6 missing blocks, the stripe is: 
AlignedStripe(Offset=0, length=1048576, fetchedChunksNum=0, 
missingChunksNum=6); locatedBlocks is: LocatedBlocks{;  fileLength=6291579;  
underConstruction=false;  
blocks=[LocatedStripedBlock{BP-1063483206-10.113.15.120-1774623121364:blk_-9223372036854775728_1005;
 getBlockSize()=6291579; corrupt=false; offset=0; 
locs=[DatanodeInfoWithStorage[127.0.0.1:38777,DS-5d4bafbc-f78a-4608-b4d7-635c26439ef6,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:33155,DS-b94250d0-39ca-47b9-ae38-14b0eccc3e2a,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:33469,DS-0ef39e40-8f52-497f-bcc4-7b74f5359200,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:40287,DS-346798b6-b6c5-4f85-ba85-84a701ced827,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:32995,DS-9be75b23-9ac5-4278-a46b-ee4a52f4ca2f,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:33789,DS-e6ff79de-2b1b-48da-a84d-a8da0fba80ff,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:46867,DS-dfa3ee1d-0964-4e1d-818d-77bdcae7c4e0,DISK],
 DatanodeInfoW
 ithStorage[127.0.0.1:44495,DS-8d4dd57f-c417-4010-9278-253236ea4ff3,DISK], 
DatanodeInfoWithStorage[127.0.0.1:34009,DS-1af2db95-5652-4bdf-b6a8-99ef2eea080f,DISK]];
 indices=[0, 1, 2, 3, 4, 5, 6, 7, 8]}];  
lastLocatedBlock=LocatedStripedBlock{BP-1063483206-10.113.15.120-1774623121364:blk_-9223372036854775728_1005;
 getBlockSize()=6291579; corrupt=false; offset=0; 
locs=[DatanodeInfoWithStorage[127.0.0.1:38777,DS-5d4bafbc-f78a-4608-b4d7-635c26439ef6,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:33155,DS-b94250d0-39ca-47b9-ae38-14b0eccc3e2a,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:33469,DS-0ef39e40-8f52-497f-bcc4-7b74f5359200,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:40287,DS-346798b6-b6c5-4f85-ba85-84a701ced827,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:32995,DS-9be75b23-9ac5-4278-a46b-ee4a52f4ca2f,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:33789,DS-e6ff79de-2b1b-48da-a84d-a8da0fba80ff,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:46867,DS-dfa3ee1d-0964-4e1d-818d-77bdcae7c4e0,DISK],
 DatanodeInfoWithStora
 ge[127.0.0.1:44495,DS-8d4dd57f-c417-4010-9278-253236ea4ff3,DISK], 
DatanodeInfoWithStorage[127.0.0.1:34009,DS-1af2db95-5652-4bdf-b6a8-99ef2eea080f,DISK]];
 indices=[0, 1, 2, 3, 4, 5, 6, 7, 8]};  isLastBlockComplete=true;  
ecPolicy=ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1]}
           at 
org.apache.hadoop.hdfs.StripeReader.checkMissingBlocks(StripeReader.java:180)
           at 
org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:350)
           at 
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:322)
           at 
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:421)
           at 
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:960)
           at java.base/java.io.DataInputStream.read(DataInputStream.java:158)
           at 
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyStatefulRead(StripedFileTestUtil.java:128)
           at 
org.apache.hadoop.hdfs.TestDFSStripedInputStreamReadFailures.lambda$testReadWithXceiverExhaustion$0(TestDFSStripedInputStreamReadFailures.java:146)
           at java.base/java.lang.Thread.run(Thread.java:1583)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to