[jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client

[ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825665#comment-17825665 ] Bryan Beaudreault commented on HDFS-3702: - I can't find any evidence that this was ever used in

[jira] [Created] (HDFS-17364) Use WeakReferencedElasticByteBufferPool in DFSStripedInputStream

Bryan Beaudreault created HDFS-17364: Summary: Use WeakReferencedElasticByteBufferPool in DFSStripedInputStream Key: HDFS-17364 URL: https://issues.apache.org/jira/browse/HDFS-17364 Project:

[jira] [Updated] (HDFS-17284) Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery

[ https://issues.apache.org/jira/browse/HDFS-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-17284: - Component/s: ec > Fix int overflow in calculating numEcReplicatedTasks and

[jira] [Commented] (HDFS-16874) Improve DataNode decommission for Erasure Coding

[ https://issues.apache.org/jira/browse/HDFS-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804427#comment-17804427 ] Bryan Beaudreault commented on HDFS-16874: -- [~jingzhao] is there any update here? > Improve

[jira] [Created] (HDFS-17319) Downgrade noisy InvalidToken log in ShortCircuitCache

Bryan Beaudreault created HDFS-17319: Summary: Downgrade noisy InvalidToken log in ShortCircuitCache Key: HDFS-17319 URL: https://issues.apache.org/jira/browse/HDFS-17319 Project: Hadoop HDFS

[jira] [Commented] (HDFS-17262) Transfer rate metric warning log is too verbose

[ https://issues.apache.org/jira/browse/HDFS-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788504#comment-17788504 ] Bryan Beaudreault commented on HDFS-17262: -- cc [~rdingankar]  > Transfer rate metric warning

[jira] [Created] (HDFS-17262) Transfer rate metric warning log is too verbose

Bryan Beaudreault created HDFS-17262: Summary: Transfer rate metric warning log is too verbose Key: HDFS-17262 URL: https://issues.apache.org/jira/browse/HDFS-17262 Project: Hadoop HDFS

[jira] [Commented] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections

[ https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783449#comment-17783449 ] Bryan Beaudreault commented on HDFS-15413: -- [~qinyuren] thank you for the suggestion! In our

[jira] [Commented] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections

[ https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782303#comment-17782303 ] Bryan Beaudreault commented on HDFS-15413: -- I was also having this issue when trying out Erasure

[jira] [Commented] (HDFS-17179) DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode

[ https://issues.apache.org/jira/browse/HDFS-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762377#comment-17762377 ] Bryan Beaudreault commented on HDFS-17179: -- Digging further into this, I’m realizing that

[jira] [Updated] (HDFS-17179) DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode

[ https://issues.apache.org/jira/browse/HDFS-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-17179: - Description: We've been running into some data corruption issues recently. When a

[jira] [Updated] (HDFS-17179) DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode

[ https://issues.apache.org/jira/browse/HDFS-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-17179: - Description: We've been running into some data corruption issues recently. When a

[jira] [Commented] (HDFS-17179) DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode

[ https://issues.apache.org/jira/browse/HDFS-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762160#comment-17762160 ] Bryan Beaudreault commented on HDFS-17179: -- This is only a problem with ShortCircuitReads. With

[jira] [Updated] (HDFS-17179) DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode

[ https://issues.apache.org/jira/browse/HDFS-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-17179: - Description: We've been running into some data corruption issues recently. When a

[jira] [Updated] (HDFS-17179) DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode

[ https://issues.apache.org/jira/browse/HDFS-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-17179: - Description: We've been running into some data corruption issues recently. When a

[jira] [Created] (HDFS-17179) DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode

Bryan Beaudreault created HDFS-17179: Summary: DFSInputStream should report CorruptMetaHeaderException as corruptBlock to NameNode Key: HDFS-17179 URL: https://issues.apache.org/jira/browse/HDFS-17179

[jira] [Commented] (HDFS-16262) Async refresh of cached locations in DFSInputStream

[ https://issues.apache.org/jira/browse/HDFS-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680883#comment-17680883 ] Bryan Beaudreault commented on HDFS-16262: -- Hey [~mccormickt12]. It's not exactly that it was

[jira] [Commented] (HDFS-16261) Configurable grace period around invalidation of replaced blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457637#comment-17457637 ] Bryan Beaudreault commented on HDFS-16261: -- [~hexiaoqiao] thank you very much for the feedback!

[jira] [Commented] (HDFS-16261) Configurable grace period around invalidation of replaced blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456729#comment-17456729 ] Bryan Beaudreault commented on HDFS-16261: -- I would love to get a review here if someone has

[jira] [Comment Edited] (HDFS-16262) Async refresh of cached locations in DFSInputStream

[ https://issues.apache.org/jira/browse/HDFS-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456727#comment-17456727 ] Bryan Beaudreault edited comment on HDFS-16262 at 12/9/21, 8:57 PM: I

[jira] [Commented] (HDFS-16262) Async refresh of cached locations in DFSInputStream

[ https://issues.apache.org/jira/browse/HDFS-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456727#comment-17456727 ] Bryan Beaudreault commented on HDFS-16262: -- I would love to get a review here if someone has

[jira] [Commented] (HDFS-16261) Configurable grace period around invalidation of replaced blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434941#comment-17434941 ] Bryan Beaudreault commented on HDFS-16261: -- Just to close the loop on that last comment, it did

[jira] [Updated] (HDFS-16261) Configurable grace period around invalidation of replaced blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-16261: - Summary: Configurable grace period around invalidation of replaced blocks (was:

[jira] [Commented] (HDFS-16261) Configurable grace period around deletion of invalidated blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426391#comment-17426391 ] Bryan Beaudreault commented on HDFS-16261: -- Despite how well tuning

[jira] [Commented] (HDFS-16261) Configurable grace period around deletion of invalidated blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425866#comment-17425866 ] Bryan Beaudreault commented on HDFS-16261: -- I've verified that setting

[jira] [Commented] (HDFS-16261) Configurable grace period around deletion of invalidated blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425805#comment-17425805 ] Bryan Beaudreault commented on HDFS-16261: -- I'm looking at this now. I don't have much

[jira] [Commented] (HDFS-16262) Async refresh of cached locations in DFSInputStream

[ https://issues.apache.org/jira/browse/HDFS-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425784#comment-17425784 ] Bryan Beaudreault commented on HDFS-16262: -- PR submitted:

[jira] [Created] (HDFS-16262) Async refresh of cached locations in DFSInputStream

Bryan Beaudreault created HDFS-16262: Summary: Async refresh of cached locations in DFSInputStream Key: HDFS-16262 URL: https://issues.apache.org/jira/browse/HDFS-16262 Project: Hadoop HDFS

[jira] [Updated] (HDFS-16261) Configurable grace period around deletion of invalidated blocks

[ https://issues.apache.org/jira/browse/HDFS-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-16261: - Description: When a block is moved with REPLACE_BLOCK, the new location is recorded in

[jira] [Created] (HDFS-16261) Configurable grace period around deletion of invalidated blocks

Bryan Beaudreault created HDFS-16261: Summary: Configurable grace period around deletion of invalidated blocks Key: HDFS-16261 URL: https://issues.apache.org/jira/browse/HDFS-16261 Project:

[jira] [Commented] (HDFS-15119) Allow expiration of cached locations in DFSInputStream

[ https://issues.apache.org/jira/browse/HDFS-15119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421490#comment-17421490 ] Bryan Beaudreault commented on HDFS-15119: -- Did anything come of the benchmarks [~ahussein]?

[jira] [Commented] (HDFS-15910) Replace bzero with explicit_bzero for better safety

[ https://issues.apache.org/jira/browse/HDFS-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419988#comment-17419988 ] Bryan Beaudreault commented on HDFS-15910: -- The linked issue has been backported to branch 3.3,

[jira] [Commented] (HDFS-15977) Call explicit_bzero only if it is available

[ https://issues.apache.org/jira/browse/HDFS-15977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419987#comment-17419987 ] Bryan Beaudreault commented on HDFS-15977: -- Just to close the loop, with that fix in place I was

[jira] [Commented] (HDFS-15977) Call explicit_bzero only if it is available

[ https://issues.apache.org/jira/browse/HDFS-15977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419741#comment-17419741 ] Bryan Beaudreault commented on HDFS-15977: -- Thank you! > Call explicit_bzero only if it is

[jira] [Commented] (HDFS-15977) Call explicit_bzero only if it is available

[ https://issues.apache.org/jira/browse/HDFS-15977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418876#comment-17418876 ] Bryan Beaudreault commented on HDFS-15977: -- Any chance this can be backported to branch-3.3 for

[jira] [Commented] (HDFS-15910) Replace bzero with explicit_bzero for better safety

[ https://issues.apache.org/jira/browse/HDFS-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418875#comment-17418875 ] Bryan Beaudreault commented on HDFS-15910: -- FYI, this breaks the ability to build hadoop 3.3 on

[jira] [Commented] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations

[ https://issues.apache.org/jira/browse/HDFS-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403140#comment-17403140 ] Bryan Beaudreault commented on HDFS-16155: -- [~hexiaoqiao] any chance you could review this? >

[jira] [Commented] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations

[ https://issues.apache.org/jira/browse/HDFS-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394872#comment-17394872 ] Bryan Beaudreault commented on HDFS-16155: -- [~brahma] any chance this could be included in the

[jira] [Commented] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations

[ https://issues.apache.org/jira/browse/HDFS-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394719#comment-17394719 ] Bryan Beaudreault commented on HDFS-16155: -- Not sure why the above comment was not edited or a

[jira] [Commented] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations

[ https://issues.apache.org/jira/browse/HDFS-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394237#comment-17394237 ] Bryan Beaudreault commented on HDFS-16155: -- If this PR is approved, I'd appreciate it included

[jira] [Commented] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations

[ https://issues.apache.org/jira/browse/HDFS-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394236#comment-17394236 ] Bryan Beaudreault commented on HDFS-16155: -- I've submitted the linked PR to solve this jira, but

[jira] [Created] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations

Bryan Beaudreault created HDFS-16155: Summary: Allow configurable exponential backoff in DFSInputStream refetchLocations Key: HDFS-16155 URL: https://issues.apache.org/jira/browse/HDFS-16155

[jira] [Commented] (HDFS-5837) dfs.namenode.replication.considerLoad does not consider decommissioned nodes

[ https://issues.apache.org/jira/browse/HDFS-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13896080#comment-13896080 ] Bryan Beaudreault commented on HDFS-5837: - Thanks!

[jira] [Created] (HDFS-5837) dfs.namenode.replication.considerLoad does not consider decommissioned nodes

Bryan Beaudreault created HDFS-5837: --- Summary: dfs.namenode.replication.considerLoad does not consider decommissioned nodes Key: HDFS-5837 URL: https://issues.apache.org/jira/browse/HDFS-5837

[jira] [Updated] (HDFS-5837) dfs.namenode.replication.considerLoad does not consider decommissioned nodes

[ https://issues.apache.org/jira/browse/HDFS-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-5837: Description: In DefaultBlockPlacementPolicy, there is a setting