Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1899717945 Thanks @ctrezzo, @li-leyang and @mccormickt12 for reviewing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

ctrezzo merged PR #6446: URL: https://github.com/apache/hadoop/pull/6446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoo

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1898272516 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1897926271 We also added "Retry with the next available datanode." explicitly to the WARN log line. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1897924023 > > > Ok so it seems we didn't remove any exception logging right? We are just additionally keeping track of them and logging them again at the end? > > > > > > correct. store al

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

mccormickt12 commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1897870432 > > Ok so it seems we didn't remove any exception logging right? We are just additionally keeping track of them and logging them again at the end? > > correct. store all excepti

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1456687729 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -786,6 +786,7 @@ private synchronized int readBuffer(ReaderStra

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1897575748 > Ok so it seems we didn't remove any exception logging right? We are just additionally keeping track of them and logging them again at the end? correct. store all exceptions in a ma

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

mccormickt12 commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1456635545 ## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRead.java: ## @@ -25,24 +25,49 @@ import java.nio.channels.ClosedByInterruptException;

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

mccormickt12 commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1456619911 ## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRead.java: ## @@ -25,24 +25,49 @@ import java.nio.channels.ClosedByInterruptException;

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

mccormickt12 commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1456618755 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -880,6 +914,8 @@ protected synchronized int readWithStrate

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

mccormickt12 commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1456617867 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -786,6 +786,7 @@ private synchronized int readBuffer(Reade

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

mccormickt12 commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1897448148 @xinglin isn't it useful to have the stacktrace for all the failures? My understanding is its 3 failures before the block missing exception. The stacktraces for each is valuable to

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1892631648 @simbadzina / @ctrezzo / @goiri / @Hexiaoqiao, we have a clean build now (most previous builds were failed due to TestDirectoryScanner). Could I get a review? -- This is an automated m

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1892196856 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890812356 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890462891 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890455591 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890444134 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1451271051 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -1270,17 +1314,16 @@ protected LocatedBlock refreshLocatedBlock

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1451270678 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -1142,15 +1178,15 @@ protected void fetchBlockByteRange(Located

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1451261032 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -1494,19 +1533,39 @@ private int pread(long position, ByteBuffe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1451226414 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -796,11 +797,18 @@ private synchronized int readBuffer(ReaderSt

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890293245 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890290464 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890275728 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

hadoop-yetus commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890273989 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

li-leyang commented on PR #6446: URL: https://github.com/apache/hadoop/pull/6446#issuecomment-1890058512 Add more reviewers. @simbadzina @goiri -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

li-leyang commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1450967849 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -1494,19 +1533,39 @@ private int pread(long position, ByteBuf

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

li-leyang commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1450962488 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -796,11 +797,18 @@ private synchronized int readBuffer(Reader

Re: [PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

li-leyang commented on code in PR #6446: URL: https://github.com/apache/hadoop/pull/6446#discussion_r1450961197 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java: ## @@ -796,11 +797,18 @@ private synchronized int readBuffer(Reader

[PR] HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException [hadoop]

xinglin opened a new pull request, #6446: URL: https://github.com/apache/hadoop/pull/6446 ### Description of PR In DFSInputStream#actualGetFromOneDataNode() and DFSInputStream#readBuffer(), it would send the exception stacktrace to the dfsClient.LOG whenever we fail on a DN.