[jira] [Comment Edited] (HDFS-12749) DN may not send block report to NN after NN restart

2019-10-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956462#comment-16956462 ] Kihwal Lee edited comment on HDFS-12749 at 10/21/19 9:55 PM: - I am very sorry

[jira] [Updated] (HDFS-12749) DN may not send block report to NN after NN restart

2019-10-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-12749: -- Fix Version/s: 2.10.1 3.2.2 3.1.4 2.9.3

[jira] [Updated] (HDFS-12749) DN may not send block report to NN after NN restart

2019-10-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-12749: -- Hadoop Flags: Reviewed > DN may not send block report to NN after NN restart >

[jira] [Commented] (HDFS-12749) DN may not send block report to NN after NN restart

2019-10-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956462#comment-16956462 ] Kihwal Lee commented on HDFS-12749: --- I am very sorry this has fell through a crack. +1 on the version

[jira] [Commented] (HDFS-14894) Add balancer parameter to balance top N used nodes

2019-10-07 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946117#comment-16946117 ] Kihwal Lee commented on HDFS-14894: --- We could sort nodes based on the utilization, so that highly

[jira] [Created] (HDFS-14892) Close the output stream if createWrappedOutputStream() fails

2019-10-04 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-14892: - Summary: Close the output stream if createWrappedOutputStream() fails Key: HDFS-14892 URL: https://issues.apache.org/jira/browse/HDFS-14892 Project: Hadoop HDFS

[jira] [Commented] (HDFS-14878) DataStreamer's ResponseProceesor#run() should log with Warn loglevel

2019-10-01 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942346#comment-16942346 ] Kihwal Lee commented on HDFS-14878: --- Do you want to change pipeline recovery related log messages to

[jira] [Commented] (HDFS-14878) DataStreamer's ResponseProceesor#run() should log with Warn loglevel

2019-10-01 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942112#comment-16942112 ] Kihwal Lee commented on HDFS-14878: --- The default log level is INFO, so this is already visible to users

[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues

2019-09-25 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937771#comment-16937771 ] Kihwal Lee commented on HDFS-14852: --- bq. Add a proper deletion check (block.isDeleted()) to

[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues

2019-09-24 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937203#comment-16937203 ] Kihwal Lee commented on HDFS-14852: --- I don't think it is simply reproduced by deleting any missing

[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues

2019-09-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933450#comment-16933450 ] Kihwal Lee commented on HDFS-14852: --- Deleted blocks are removed in {{scheduleReconstruction()}} when

[jira] [Commented] (HDFS-13959) TestUpgradeDomainBlockPlacementPolicy is flaky

2019-09-18 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932878#comment-16932878 ] Kihwal Lee commented on HDFS-13959: --- We were seeing failures in branch-2. Cherry-picked it to

[jira] [Updated] (HDFS-13959) TestUpgradeDomainBlockPlacementPolicy is flaky

2019-09-18 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-13959: -- Fix Version/s: 3.2.2 3.1.4 2.10.0 >

[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues

2019-09-18 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932687#comment-16932687 ] Kihwal Lee commented on HDFS-14852: --- bq. maybe one block in two or more queues. That is caused by

[jira] [Commented] (HDFS-14848) ACE for Non Existing Paths

2019-09-13 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929261#comment-16929261 ] Kihwal Lee commented on HDFS-14848: --- Throwing FileNotFoundException may not always be correct thing to

[jira] [Commented] (HDFS-12119) Inconsistent "Number of Under-Replicated Blocks" shown on HDFS web UI and fsck report

2019-09-05 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923634#comment-16923634 ] Kihwal Lee commented on HDFS-12119: --- Is it same as HDFS-3457? > Inconsistent "Number of

[jira] [Commented] (HDFS-14758) Decrease lease hard limit

2019-09-04 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922757#comment-16922757 ] Kihwal Lee commented on HDFS-14758: --- Make it as a namenode config with a default of 20 minutes.

[jira] [Commented] (HDFS-14758) Decrease lease hard limit

2019-08-29 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918839#comment-16918839 ] Kihwal Lee commented on HDFS-14758: --- Unconditionally calling recoverLease() on a failure will work when

[jira] [Commented] (HDFS-14771) Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index)

2019-08-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916780#comment-16916780 ] Kihwal Lee commented on HDFS-14771: --- [~sodonnell], thanks for the explanation. Sounds reasonable. >

[jira] [Commented] (HDFS-14782) Add a new fsck command to get detailed decommission information.

2019-08-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916734#comment-16916734 ] Kihwal Lee commented on HDFS-14782: --- Doesn't the metasave command tell you enough information already?

[jira] [Commented] (HDFS-14771) Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index)

2019-08-26 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915947#comment-16915947 ] Kihwal Lee commented on HDFS-14771: --- FSImage format is changing. What is the compatibility story here?

[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-07-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883317#comment-16883317 ] Kihwal Lee commented on HDFS-14563: --- Maintenance mode is different from other admin state. It has

[jira] [Commented] (HDFS-14588) Client retries Standby NN continuously even if Active NN is available (WebHDFS)

2019-06-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869595#comment-16869595 ] Kihwal Lee commented on HDFS-14588: --- bq.The issue we had in the past is the namenoderpcaddress being

[jira] [Comment Edited] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867975#comment-16867975 ] Kihwal Lee edited comment on HDFS-14563 at 6/19/19 8:27 PM: {quote} It may

[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867975#comment-16867975 ] Kihwal Lee commented on HDFS-14563: --- bq. It may make sense to have an API like \{{setNodeStatus(nodeId,

[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867925#comment-16867925 ] Kihwal Lee commented on HDFS-14563: --- In your patch, the command can be undone if {{refreshNodes}} is

[jira] [Issue Comment Deleted] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14563: -- Comment: was deleted (was: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Updated] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14563: -- Status: Open (was: Patch Available) > Enhance interface about recommissioning/decommissioning >

[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867919#comment-16867919 ] Kihwal Lee commented on HDFS-14563: --- I am attaching an internal implementation of maintenance command

[jira] [Updated] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14563: -- Attachment: mt_mode-2.txt > Enhance interface about recommissioning/decommissioning >

[jira] [Updated] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14563: -- Attachment: (was: mt_mode-2.txt) > Enhance interface about recommissioning/decommissioning >

[jira] [Updated] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14563: -- Attachment: mt_mode-2.txt > Enhance interface about recommissioning/decommissioning >

[jira] [Commented] (HDFS-14579) In refreshNodes, avoid performing a DNS lookup while holding the write lock

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867808#comment-16867808 ] Kihwal Lee commented on HDFS-14579: --- Loading include/exclude causes lookups unless the host entries in

[jira] [Commented] (HDFS-14570) Bring back ability to totally disable webhdfs by bringing dfs.webhdfs.enabled property back into the hdfs-site.xml

2019-06-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867675#comment-16867675 ] Kihwal Lee commented on HDFS-14570: --- As another workaround, one can build a simple filter that rejects

[jira] [Commented] (HDFS-14579) In refreshNodes, avoid performing a DNS lookup while holding the write lock

2019-06-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866612#comment-16866612 ] Kihwal Lee commented on HDFS-14579: --- {{node.getResolvedAddress()}} does not cause an actual look up.

[jira] [Commented] (HDFS-14575) LeaseRenewer#daemon threads leak in DFSClient

2019-06-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865593#comment-16865593 ] Kihwal Lee commented on HDFS-14575: --- Which branch did you see this? We've seen this before. >

[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning

2019-06-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862241#comment-16862241 ] Kihwal Lee commented on HDFS-14563: --- In our experience, decommissioning and adding new nodes are

[jira] [Commented] (HDFS-14532) Datanode's BlockSender checksum buffer is too big

2019-06-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858722#comment-16858722 ] Kihwal Lee commented on HDFS-14532: --- Assuming 4 byte checksum per 512 byte data chunk, a 128kB checksum

[jira] [Commented] (HDFS-14531) Datanode's ScanInfo requires excessive memory

2019-06-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858706#comment-16858706 ] Kihwal Lee commented on HDFS-14531: --- One workaround is to disable DirectoryScanner by setting

[jira] [Updated] (HDFS-14521) Suppress setReplication logging.

2019-05-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14521: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.3 2.9.3

[jira] [Comment Edited] (HDFS-14521) Suppress setReplication logging.

2019-05-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852138#comment-16852138 ] Kihwal Lee edited comment on HDFS-14521 at 5/30/19 6:05 PM: The logging was

[jira] [Updated] (HDFS-14521) Suppress setReplication logging.

2019-05-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14521: -- Summary: Suppress setReplication logging. (was: Suppress setReplication logging while replaying

[jira] [Commented] (HDFS-14521) Suppress setReplication logging while replaying edits

2019-05-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852138#comment-16852138 ] Kihwal Lee commented on HDFS-14521: --- Changed it to debug. > Suppress setReplication logging while

[jira] [Updated] (HDFS-14521) Suppress setReplication logging while replaying edits

2019-05-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14521: -- Attachment: HDFS-14521-01.patch > Suppress setReplication logging while replaying edits >

[jira] [Commented] (HDFS-14521) Suppress setReplication logging while replaying edits

2019-05-29 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851335#comment-16851335 ] Kihwal Lee commented on HDFS-14521: --- No test. It is a log line change. Failed tests are not related. >

[jira] [Updated] (HDFS-14521) Suppress setReplication logging while replaying edits

2019-05-29 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14521: -- Assignee: Kihwal Lee Status: Patch Available (was: Open) > Suppress setReplication logging

[jira] [Updated] (HDFS-14521) Suppress setReplication logging while replaying edits

2019-05-29 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14521: -- Attachment: HDFS-14521.patch > Suppress setReplication logging while replaying edits >

[jira] [Created] (HDFS-14521) Suppress setReplication logging while replaying edits

2019-05-29 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-14521: - Summary: Suppress setReplication logging while replaying edits Key: HDFS-14521 URL: https://issues.apache.org/jira/browse/HDFS-14521 Project: Hadoop HDFS Issue

[jira] [Commented] (HDFS-6708) StorageType should be encoded in the block token

2019-05-23 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846874#comment-16846874 ] Kihwal Lee commented on HDFS-6708: -- Namenode should support both format. The old format should be used

[jira] [Commented] (HDFS-6708) StorageType should be encoded in the block token

2019-05-23 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846710#comment-16846710 ] Kihwal Lee commented on HDFS-6708: -- A user reports this issue in HDFS-14509. > StorageType should be

[jira] [Commented] (HDFS-14509) DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x

2019-05-23 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846709#comment-16846709 ] Kihwal Lee commented on HDFS-14509: --- It is caused by HDFS-6708.  Are you seeing failures? > DN throws

[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster

2019-05-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842303#comment-16842303 ] Kihwal Lee commented on HDFS-14186: --- We probably want an ability to tell whether a block report has

[jira] [Commented] (HDFS-14496) Backport HDFS-13056 into branch-2.8/2.9/2

2019-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841519#comment-16841519 ] Kihwal Lee commented on HDFS-14496: --- If the bridge release is really happening, it makes sense to pull

[jira] [Created] (HDFS-14464) Remove unnecessary log message from DFSInputStream

2019-05-02 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-14464: - Summary: Remove unnecessary log message from DFSInputStream Key: HDFS-14464 URL: https://issues.apache.org/jira/browse/HDFS-14464 Project: Hadoop HDFS Issue Type:

[jira] [Commented] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter

2019-04-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822024#comment-16822024 ] Kihwal Lee commented on HDFS-14434: --- We can have the client selectively change the query param to

[jira] [Commented] (HDFS-14443) Throwing RemoteException in the time of Read Operation

2019-04-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821926#comment-16821926 ] Kihwal Lee commented on HDFS-14443: --- {{msync()}} is a write op and it looks like read requests can

[jira] [Assigned] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter

2019-04-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-14434: - Assignee: KWON BYUNGCHANG > webhdfs that connect secure hdfs should not use user.name parameter

[jira] [Commented] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter

2019-04-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821436#comment-16821436 ] Kihwal Lee commented on HDFS-14434: --- The changes in {{JspHelper}} looks good. IMO, it is better to

[jira] [Commented] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter

2019-04-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820458#comment-16820458 ] Kihwal Lee commented on HDFS-14434: --- I should have clarified that the current code does validation

[jira] [Comment Edited] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter

2019-04-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820082#comment-16820082 ] Kihwal Lee edited comment on HDFS-14434 at 4/17/19 1:42 PM: I think server

[jira] [Commented] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter

2019-04-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820082#comment-16820082 ] Kihwal Lee commented on HDFS-14434: --- I think server should ignore the parameter, "user.name", if in

[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer

2019-02-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777303#comment-16777303 ] Kihwal Lee commented on HDFS-14292: --- There was also an effort to make data transfer use http (by

[jira] [Commented] (HDFS-14158) Checkpointer ignores configured time period > 5 minutes

2019-01-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744109#comment-16744109 ] Kihwal Lee commented on HDFS-14158: --- [~tiwalter], I suggest take a look at the patch in HDFS-1572 and

[jira] [Comment Edited] (HDFS-14158) Checkpointer ignores configured time period > 5 minutes

2019-01-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744104#comment-16744104 ] Kihwal Lee edited comment on HDFS-14158 at 1/16/19 2:44 PM: Are you running

[jira] [Commented] (HDFS-14158) Checkpointer ignores configured time period > 5 minutes

2019-01-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744104#comment-16744104 ] Kihwal Lee commented on HDFS-14158: --- Are you running BackupNode? I has received little to no attention

[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster

2019-01-14 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742297#comment-16742297 ] Kihwal Lee commented on HDFS-14186: --- bq. Do you know what's the issue with lifeline? Lifeline is more

[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster

2019-01-14 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742293#comment-16742293 ] Kihwal Lee commented on HDFS-14186: --- [~hexiaoqiao], one thing to note is that the rpc processing time

[jira] [Comment Edited] (HDFS-14200) Add emptyTrash option to purge trash immediately

2019-01-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740740#comment-16740740 ] Kihwal Lee edited comment on HDFS-14200 at 1/11/19 8:05 PM: "expunge" is

[jira] [Commented] (HDFS-14200) Add emptyTrash option to purge trash immediately

2019-01-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740740#comment-16740740 ] Kihwal Lee commented on HDFS-14200: --- "expunge" is simply doing what trash emptier is already doing. The

[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster

2019-01-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739493#comment-16739493 ] Kihwal Lee commented on HDFS-14186: --- I think the safe mode extension was intended for this.  Instead of

[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster

2019-01-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735986#comment-16735986 ] Kihwal Lee commented on HDFS-14186: --- I think nodes are already not marked "dead" in the startup safe

[jira] [Comment Edited] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster

2019-01-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735986#comment-16735986 ] Kihwal Lee edited comment on HDFS-14186 at 1/7/19 3:57 PM: --- I think nodes are

[jira] [Commented] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE

2018-12-06 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712008#comment-16712008 ] Kihwal Lee commented on HDFS-14081: --- {{metaSave()}} needs to be made to skip the parts that are

[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly

2018-12-06 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711810#comment-16711810 ] Kihwal Lee commented on HDFS-14109: --- This has less to do with federation itself. Rather, it is more

[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly

2018-11-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701969#comment-16701969 ] Kihwal Lee commented on HDFS-14109: --- I don't think it is a good idea. NN audit log is already huge and

[jira] [Updated] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock

2018-11-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14101: -- Labels: newbie (was: ) > Random failure of testListCorruptFilesCorruptedBlock >

[jira] [Updated] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock

2018-11-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-14101: -- Component/s: test > Random failure of testListCorruptFilesCorruptedBlock >

[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock

2018-11-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699563#comment-16699563 ] Kihwal Lee commented on HDFS-14101: --- The test is very old, pre-2011, pre-git. The source of this bug

[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock

2018-11-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699560#comment-16699560 ] Kihwal Lee commented on HDFS-14101: --- The test creates two files with random sizes. In a failed run, one

[jira] [Created] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock

2018-11-26 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-14101: - Summary: Random failure of testListCorruptFilesCorruptedBlock Key: HDFS-14101 URL: https://issues.apache.org/jira/browse/HDFS-14101 Project: Hadoop HDFS Issue

[jira] [Commented] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE

2018-11-14 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687097#comment-16687097 ] Kihwal Lee commented on HDFS-14081: --- This is not a problem. metasave is meaningful only on active

[jira] [Commented] (HDFS-13100) Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs.

2018-09-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626528#comment-16626528 ] Kihwal Lee commented on HDFS-13100: --- Is the patch working properly? We have a user who is running 2.7.3

[jira] [Comment Edited] (HDFS-13904) ContentSummary does not always respect processing limit, resulting in long lock acquisitions

2018-09-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607629#comment-16607629 ] Kihwal Lee edited comment on HDFS-13904 at 9/7/18 8:35 PM: --- Your proposal

[jira] [Commented] (HDFS-13904) ContentSummary does not always respect processing limit, resulting in long lock acquisitions

2018-09-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607629#comment-16607629 ] Kihwal Lee commented on HDFS-13904: --- Your proposal sounds reasonable, although I wonder what is

[jira] [Commented] (HDFS-13889) The hadoop3.x client have compatible problem with hadoop2.x cluster

2018-09-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603037#comment-16603037 ] Kihwal Lee commented on HDFS-13889: --- Looks like a dupe of HDFS-12920 > The hadoop3.x client have

[jira] [Resolved] (HDFS-13889) The hadoop3.x client have compatible problem with hadoop2.x cluster

2018-09-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-13889. --- Resolution: Duplicate > The hadoop3.x client have compatible problem with hadoop2.x cluster >

[jira] [Commented] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

2018-08-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581525#comment-16581525 ] Kihwal Lee commented on HDFS-13671: --- I've started working on the revert. So far in BlockManager, I had

[jira] [Commented] (HDFS-13814) Remove super user privilege requirement for getServiceStatus in HAServiceProtocol

2018-08-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576300#comment-16576300 ] Kihwal Lee commented on HDFS-13814: --- +1. We have done the same thing internally for a custom monitoring

[jira] [Commented] (HDFS-13775) Add a conf property to retry dns reverse lookups during datanode registration

2018-08-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568303#comment-16568303 ] Kihwal Lee commented on HDFS-13775: --- It could be meaningless if the host is running a caching daemon

[jira] [Commented] (HDFS-13769) Namenode gets stuck when deleting large dir in trash

2018-07-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558665#comment-16558665 ] Kihwal Lee commented on HDFS-13769: --- bq. This seems to apply not only for trash dir, but also any

[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports

2018-07-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558382#comment-16558382 ] Kihwal Lee commented on HDFS-9260: -- I propose revert of this. HDFS-13671 also reports about 4x slower

[jira] [Resolved] (HDFS-13751) "hdfs dfs -du -h /*" shows local dir instead of HDFS dir

2018-07-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-13751. --- Resolution: Invalid > "hdfs dfs -du -h /*" shows local dir instead of HDFS dir >

[jira] [Commented] (HDFS-13306) NullPoint when launch org.apache.hadoop.fs.http.server.HttpFSServerWebServer

2018-07-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536464#comment-16536464 ] Kihwal Lee commented on HDFS-13306: --- {{HttpFSServerWebApp}} implements {{ServerWebApp}}, which blows up

[jira] [Commented] (HDFS-13306) NullPoint when launch org.apache.hadoop.fs.http.server.HttpFSServerWebServer

2018-07-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536462#comment-16536462 ] Kihwal Lee commented on HDFS-13306: --- I've seen httpFs server not starting with the exactly same error.

[jira] [Comment Edited] (HDFS-13236) Standby NN down with error encountered while tailing edits

2018-07-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531436#comment-16531436 ] Kihwal Lee edited comment on HDFS-13236 at 7/3/18 1:58 PM: --- The "restart fails

[jira] [Updated] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x

2018-07-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-13596: -- Target Version/s: 3.2.0, 3.1.1, 3.0.4 > NN restart fails after RollingUpgrade from 2.x to 3.x >

[jira] [Commented] (HDFS-13236) Standby NN down with error encountered while tailing edits

2018-07-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531436#comment-16531436 ] Kihwal Lee commented on HDFS-13236: --- The restart after upgrade issue is being addressed in HDFS-13596. 

[jira] [Commented] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

2018-06-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513915#comment-16513915 ] Kihwal Lee commented on HDFS-13671: --- One more (big) reason to revert HDFS-9260? > Namenode deletes

[jira] [Commented] (HDFS-8865) Improve quota initialization performance

2018-06-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497961#comment-16497961 ] Kihwal Lee commented on HDFS-8865: -- The memory overhead is very small compared to the memory required to

[jira] [Commented] (HDFS-12749) DN may not send block report to NN after NN restart

2018-05-22 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484503#comment-16484503 ] Kihwal Lee commented on HDFS-12749: --- The patch looks good, but a test case would still be good. We can

<    1   2   3   4   5   6   7   8   9   10   >