[jira] [Commented] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014703#comment-16014703 ] Kihwal Lee commented on HDFS-11817: --- *Details of the NPE:* The JVM did produce a stacktrace on the very

[jira] [Updated] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11641: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.2

[jira] [Commented] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012686#comment-16012686 ] Kihwal Lee commented on HDFS-11641: --- Please provide a patch for branch-2. With the EC-related portion

[jira] [Commented] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012671#comment-16012671 ] Kihwal Lee commented on HDFS-11641: --- +1 the latest patch looks good. > Reduce cost of audit logging by

[jira] [Updated] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11641: -- Target Version/s: 2.8.2 (was: 2.8.1) > Reduce cost of audit logging by using FileStatus instead of

[jira] [Commented] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011498#comment-16011498 ] Kihwal Lee commented on HDFS-11820: --- bq. Thread #2 catches up, and acquires same instance of RenameOp,

[jira] [Commented] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011457#comment-16011457 ] Kihwal Lee commented on HDFS-11820: --- Write lock ensures there is only one thread calling logEdit() at

[jira] [Resolved] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11820. --- Resolution: Invalid > Thread safety in logEdit? > - > > Key:

[jira] [Commented] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010962#comment-16010962 ] Kihwal Lee commented on HDFS-11820: --- Jira is for bug report. Please use relevant mailing list for

[jira] [Commented] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010646#comment-16010646 ] Kihwal Lee commented on HDFS-11817: --- There are two possible immediate fixes that can be implemented. -

[jira] [Assigned] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11817: - Assignee: Kihwal Lee > A faulty node can cause a lease leak and NPE on accessing data >

[jira] [Created] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-12 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11817: - Summary: A faulty node can cause a lease leak and NPE on accessing data Key: HDFS-11817 URL: https://issues.apache.org/jira/browse/HDFS-11817 Project: Hadoop HDFS

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11755: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.2

[jira] [Commented] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005219#comment-16005219 ] Kihwal Lee commented on HDFS-11755: --- +1 looks good. > Underconstruction blocks can be considered

[jira] [Commented] (HDFS-8789) Block Placement policy migrator

2017-05-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004804#comment-16004804 ] Kihwal Lee commented on HDFS-8789: -- What happens on NN restart (to be precise, during the replication

[jira] [Updated] (HDFS-11765) Fix:Performance regression due to incorrect use of DataChecksum

2017-05-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11765: -- Target Version/s: 2.8.1 > Fix:Performance regression due to incorrect use of DataChecksum >

[jira] [Commented] (HDFS-8789) Block Placement policy migrator

2017-05-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001021#comment-16001021 ] Kihwal Lee commented on HDFS-8789: -- This can be fatal to namenode. {code} blocksWlocs =

[jira] [Updated] (HDFS-11702) Remove indefinite caching of key provider uri in DFSClient

2017-05-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11702: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.2

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999105#comment-15999105 ] Kihwal Lee commented on HDFS-11742: --- bq. I probably still not. I invite everyone to run Balancer

[jira] [Comment Edited] (HDFS-11742) Improve balancer usability after HDFS-8818

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999071#comment-15999071 ] Kihwal Lee edited comment on HDFS-11742 at 5/5/17 11:24 PM: bq. Do you think

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8818

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Summary: Improve balancer usability after HDFS-8818 (was: Improve balancer usability after HDFS-8188)

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999071#comment-15999071 ] Kihwal Lee commented on HDFS-11742: --- bq. Do you think that this is still a blocker of the release? -

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998564#comment-15998564 ] Kihwal Lee commented on HDFS-8818: -- Posted a patch to HDFS-11742. Hopefully, this will prevent users from

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Attachment: HDFS-11742.v2.trunk.patch > Improve balancer usability after HDFS-8188 >

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998536#comment-15998536 ] Kihwal Lee commented on HDFS-11742: --- Setting HDFS-11377 as a dependency. In most cases, the proposed

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998534#comment-15998534 ] Kihwal Lee commented on HDFS-11742: --- Instead of reverting, I am making a simple change to make it more

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Description: We ran 2.8 balancer with HDFS-8818 on a 280-node and a 2,400-node cluster. In both cases,

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Target Version/s: 2.7.4, 2.8.1 (was: 2.8.1) > Improve balancer usability after HDFS-8188 >

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Summary: Improve balancer usability after HDFS-8188 (was: Revert the core changes from HDFS-8818) >

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997774#comment-15997774 ] Kihwal Lee commented on HDFS-8818: -- I will post a patch for improvement tomorrow. I am letting it run

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996738#comment-15996738 ] Kihwal Lee commented on HDFS-8818: -- bq. Do you have HDFS-8824 in your runs? I suspect the first run has

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995477#comment-15995477 ] Kihwal Lee commented on HDFS-8818: -- I set the dfs.datanode.balance.max.concurrent.moves to 15, which is

[jira] [Comment Edited] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995477#comment-15995477 ] Kihwal Lee edited comment on HDFS-8818 at 5/3/17 7:17 PM: -- I set

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994840#comment-15994840 ] Kihwal Lee commented on HDFS-8818: -- We see this problem in EVERY single cluster we tried this on. The

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: bal1.png bal2.png > Allow Balancer to run faster >

[jira] [Comment Edited] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994001#comment-15994001 ] Kihwal Lee edited comment on HDFS-11742 at 5/2/17 11:26 PM:

[jira] [Commented] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994001#comment-15994001 ] Kihwal Lee commented on HDFS-11742: ---

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Attachment: balancer2.8.png > Revert the core changes from HDFS-8818 >

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Description: This is to revert the core changes made by HDFS-8818. The reason is explained in the jira

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Assignee: Kihwal Lee Status: Patch Available (was: Open) > Revert the core changes from

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Attachment: HDFS-11742.trunk.patch HDFS-11742.branch-2.patch

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.branch-2.patch) > Allow Balancer to run faster >

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.trunk.patch) > Allow Balancer to run faster >

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.branch-2.8.patch) > Allow Balancer to run faster >

[jira] [Created] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11742: - Summary: Revert the core changes from HDFS-8818 Key: HDFS-11742 URL: https://issues.apache.org/jira/browse/HDFS-11742 Project: Hadoop HDFS Issue Type: Bug

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: bal_revert.trunk.patch bal_revert.branch-2.patch

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993840#comment-15993840 ] Kihwal Lee commented on HDFS-8818: -- I will open a new one and post the patch there. > Allow Balancer to

[jira] [Reopened] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-8818: -- Reopening for revert. git won't revert it cleanly since there have been other changes since the commit. I

[jira] [Assigned] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-8818: Assignee: Kihwal Lee (was: Tsz Wo Nicholas Sze) > Allow Balancer to run faster >

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993723#comment-15993723 ] Kihwal Lee commented on HDFS-8818: -- Thanks for the update, [~jojochuang]. That fixes the hang, but the

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993654#comment-15993654 ] Kihwal Lee commented on HDFS-8818: -- -1. Proposing revert of this untested change. We've run it on a small

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1

[jira] [Comment Edited] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991670#comment-15991670 ] Kihwal Lee edited comment on HDFS-11714 at 5/1/17 10:24 PM: the branch-2 patch

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991670#comment-15991670 ] Kihwal Lee commented on HDFS-11714: --- the branch-2 patch is identical except for the variables in the

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991665#comment-15991665 ] Kihwal Lee commented on HDFS-11714: --- You beat me. Thanks. > Newly added NN storage directory won't get

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991664#comment-15991664 ] Kihwal Lee commented on HDFS-11714: --- TestPipelinesFailover, TestBalancer are passing. The rest is

[jira] [Commented] (HDFS-7541) Upgrade Domains in HDFS

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991533#comment-15991533 ] Kihwal Lee commented on HDFS-7541: -- Just found out I somehow clicked the evil "Assign to me" link 6 days

[jira] [Assigned] (HDFS-7541) Upgrade Domains in HDFS

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-7541: Assignee: Ming Ma (was: Kihwal Lee) > Upgrade Domains in HDFS > --- > >

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991469#comment-15991469 ] Kihwal Lee commented on HDFS-11714: --- Filed HDFS-11729. > Newly added NN storage directory won't get

[jira] [Created] (HDFS-11729) Improve NNStorageRetentionManager failure handling.

2017-05-01 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11729: - Summary: Improve NNStorageRetentionManager failure handling. Key: HDFS-11729 URL: https://issues.apache.org/jira/browse/HDFS-11729 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Attachment: HDFS-11714.v3.trunk.patch HDFS-11714.v3.branch-2.patch Attaching new

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991420#comment-15991420 ] Kihwal Lee commented on HDFS-11714: --- I will file a jira for the retention manager improvement. > Newly

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991339#comment-15991339 ] Kihwal Lee commented on HDFS-11714: --- bq. What if a VERSION file already exists in the directory for some

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991104#comment-15991104 ] Kihwal Lee commented on HDFS-11609: --- Only updated the comment in the new patch. Test failures are not

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991045#comment-15991045 ] Kihwal Lee commented on HDFS-11714: --- These are passing. {noformat}

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Attachment: HDFS-11714.v2.trunk.patch HDFS-11714.v2.branch-2.patch Attaching updated

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Target Version/s: 2.7.4, 2.8.1 (was: 2.8.1) > Newly added NN storage directory won't get initialized

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Attachment: HDFS-11609_v3.trunk.patch HDFS-11609_v3.branch-2.patch

[jira] [Assigned] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11714: - Assignee: Kihwal Lee > Newly added NN storage directory won't get initialized and cause space >

[jira] [Commented] (HDFS-11396) TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988813#comment-15988813 ] Kihwal Lee commented on HDFS-11396: --- It could be due to interactions with other tests in

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988809#comment-15988809 ] Kihwal Lee commented on HDFS-11714: --- None of the findbugs warning was introduced by the patch.

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Status: Patch Available (was: Open) > Newly added NN storage directory won't get initialized and cause

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Attachment: HDFS-11714.trunk.patch The version file can be created in an empty storage diectory in

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987756#comment-15987756 ] Kihwal Lee commented on HDFS-11714: --- The original design pre-HA was to just create directory structure

[jira] [Created] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11714: - Summary: Newly added NN storage directory won't get initialized and cause space exhaustion Key: HDFS-11714 URL: https://issues.apache.org/jira/browse/HDFS-11714 Project:

[jira] [Commented] (HDFS-11709) StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986836#comment-15986836 ] Kihwal Lee commented on HDFS-11709: --- +1 lgtm > StandbyCheckpointer should handle an non-existing

[jira] [Commented] (HDFS-5042) Completed files lost after power failure

2017-04-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985460#comment-15985460 ] Kihwal Lee commented on HDFS-5042: -- We've seen that too. Unless there is other bug in Hadoop, I think

[jira] [Commented] (HDFS-10495) Block should be marked as missing if the all the replicas are on Decommissioned nodes.

2017-04-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985457#comment-15985457 ] Kihwal Lee commented on HDFS-10495: --- I've analyzed this and discussed potential solutions in HDFS-11609.

[jira] [Commented] (HDFS-5042) Completed files lost after power failure

2017-04-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984824#comment-15984824 ] Kihwal Lee commented on HDFS-5042: -- bq. Can we recover this kind of block from namenode after getting

[jira] [Assigned] (HDFS-7541) Upgrade Domains in HDFS

2017-04-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-7541: Assignee: Kihwal Lee (was: Ming Ma) > Upgrade Domains in HDFS > --- > >

[jira] [Commented] (HDFS-7541) Upgrade Domains in HDFS

2017-04-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981993#comment-15981993 ] Kihwal Lee commented on HDFS-7541: -- [~mingma] Since you guys have been using it in production for a while,

[jira] [Commented] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981295#comment-15981295 ] Kihwal Lee commented on HDFS-11691: --- bq. this way datanode link field on UI will also display

[jira] [Assigned] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11691: - Assignee: Kihwal Lee > Add a proper scheme to the datanode links in NN web UI >

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Status: Patch Available (was: Open) > Add a proper scheme to the datanode links in NN web UI >

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Attachment: HDFS-11691.patch > Add a proper scheme to the datanode links in NN web UI >

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Description: On the datanodes page of the namenode web UI, the datanode links may not be correct if

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Description: On the datanodes page of the namenode web UI, the datanode links may not be correct if

[jira] [Created] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11691: - Summary: Add a proper scheme to the datanode links in NN web UI Key: HDFS-11691 URL: https://issues.apache.org/jira/browse/HDFS-11691 Project: Hadoop HDFS Issue

[jira] [Comment Edited] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978923#comment-15978923 ] Kihwal Lee edited comment on HDFS-9922 at 4/21/17 3:58 PM: --- Don't we need this in

[jira] [Commented] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978923#comment-15978923 ] Kihwal Lee commented on HDFS-9922: -- Don't we need this in 2.8? > Upgrade Domain placement policy status

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977517#comment-15977517 ] Kihwal Lee commented on HDFS-11609: --- [~jojochuang], would you look at the updated patch? > Some blocks

[jira] [Commented] (HDFS-11661) GetContentSummary uses excessive amounts of memory

2017-04-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974623#comment-15974623 ] Kihwal Lee commented on HDFS-11661: --- [~xiaochen], [~jingzhao], any thoughts? We've internally reverted

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973539#comment-15973539 ] Kihwal Lee commented on HDFS-11609: --- Unit test failures are not related to the patch. I ran all of them

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Attachment: HDFS-11609_v2.trunk.patch HDFS-11609_v2.branch-2.patch Attaching the

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971133#comment-15971133 ] Kihwal Lee commented on HDFS-11609: --- Agree it is a blocker. I will update the patch today. > Some

[jira] [Updated] (HDFS-11648) Lazy construct the IIP pathname

2017-04-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11648: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1

[jira] [Commented] (HDFS-11648) Lazy construct the IIP pathname

2017-04-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966351#comment-15966351 ] Kihwal Lee commented on HDFS-11648: --- +1 simple, yet effective. > Lazy construct the IIP pathname >

[jira] [Commented] (HDFS-11626) Deprecate oiv_legacy tool

2017-04-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960871#comment-15960871 ] Kihwal Lee commented on HDFS-11626: --- bq. oiv_legacy only works for fsimage at or before Hadoop 2.4. We

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Attachment: HDFS-11609.branch-2.patch Attaching branch-2 version of the patch, as the trunk version was

<    2   3   4   5   6   7   8   9   10   11   >