[jira] [Comment Edited] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979494#comment-15979494 ] Junping Du edited comment on HDFS-10797 at 4/21/17 10:28 PM: - Hi [~mackrorysd] and [~xiaochen], can you take a look at HDFS-11661 which could be caused by improvement here? Thx! was (Author: djp): Hi [~mackrorysd], can you take a look at HDFS-11661 which could be caused by improvement here? Thx! > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.8.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch, > HDFS-10797.009.patch, HDFS-10797.010.patch, HDFS-10797.010.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979494#comment-15979494 ] Junping Du commented on HDFS-10797: --- Hi [~mackrorysd], can you take a look at HDFS-11661 which could be caused by improvement here? Thx! > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 2.8.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch, > HDFS-10797.009.patch, HDFS-10797.010.patch, HDFS-10797.010.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11538) Move ClientProtocol HA proxies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979115#comment-15979115 ] Junping Du commented on HDFS-11538: --- Cool. Thanks Andrew! > Move ClientProtocol HA proxies into hadoop-hdfs-client > -- > > Key: HDFS-11538 > URL: https://issues.apache.org/jira/browse/HDFS-11538 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Huafeng Wang >Priority: Blocker > Fix For: 3.0.0-alpha3, 2.8.1 > > Attachments: HDFS-11538.001.patch, HDFS-11538.002.patch, > HDFS-11538.003.patch, HDFS-11538-branch-2.001.patch > > > Follow-up for HDFS-11431. We should move this missing class over rather than > pulling in the whole hadoop-hdfs dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11558) BPServiceActor thread name is too long
[
https://issues.apache.org/jira/browse/HDFS-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966725#comment-15966725
]
Junping Du commented on HDFS-11558:
---
No worry. Mingliang. I will keep tracking. :)
> BPServiceActor thread name is too long
> --
>
> Key: HDFS-11558
> URL: https://issues.apache.org/jira/browse/HDFS-11558
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
>Priority: Minor
> Fix For: 3.0.0-alpha3, 2.8.1
>
> Attachments: HDFS-11558.000.patch, HDFS-11558.001.patch,
> HDFS-11558.002.patch, HDFS-11558.003.patch, HDFS-11558.004.patch,
> HDFS-11558.005.patch, HDFS-11558.006.patch, HDFS-11558-branch-2.006.patch,
> HDFS-11558-branch-2.8.006.patch
>
>
> Currently, the thread name looks like
> {code}
> 2017-03-20 18:32:22,022 [DataNode:
> [[[DISK]file:/Users/szetszwo/hadoop/t2/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn1_data0,
>
> [DISK]file:/Users/szetszwo/hadoop/t2/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn1_data1]]
> heartbeating to localhost/127.0.0.1:51772] INFO ...
> {code}
> which contains the full path for each storage dir. It is unnecessarily long.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11163) Mover should move the file blocks to default storage once policy is unset
[
https://issues.apache.org/jira/browse/HDFS-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966719#comment-15966719
]
Junping Du commented on HDFS-11163:
---
You are welcome, [~cnauroth]! :)
> Mover should move the file blocks to default storage once policy is unset
> -
>
> Key: HDFS-11163
> URL: https://issues.apache.org/jira/browse/HDFS-11163
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 3.0.0-alpha3, 2.8.1
>
> Attachments: HDFS-11163-001.patch, HDFS-11163-002.patch,
> HDFS-11163-003.patch, HDFS-11163-004.patch, HDFS-11163-005.patch,
> HDFS-11163-006.patch, HDFS-11163-007.patch, HDFS-11163-branch-2.001.patch,
> HDFS-11163-branch-2.002.patch, HDFS-11163-branch-2.003.patch,
> temp-YARN-6278.HDFS-11163.patch
>
>
> HDFS-9534 added new API in FileSystem to unset the storage policy. Once
> policy is unset blocks should move back to the default storage policy.
> Currently mover is not moving file blocks which have zero storage ID
> {code}
> // currently we ignore files with unspecified storage policy
> if (policyId == HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11558) BPServiceActor thread name is too long
[
https://issues.apache.org/jira/browse/HDFS-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966628#comment-15966628
]
Junping Du commented on HDFS-11558:
---
Hi [~liuml07], thanks for review and commit. As my email to hadoop dev list, we
have 2.8.1 branch get cut-off for release since yesterday. Just merge the
commit to branch-2.8.1 assume it is supposed to land in 2.8.1 release. Isn't it?
> BPServiceActor thread name is too long
> --
>
> Key: HDFS-11558
> URL: https://issues.apache.org/jira/browse/HDFS-11558
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
>Priority: Minor
> Fix For: 3.0.0-alpha3, 2.8.1
>
> Attachments: HDFS-11558.000.patch, HDFS-11558.001.patch,
> HDFS-11558.002.patch, HDFS-11558.003.patch, HDFS-11558.004.patch,
> HDFS-11558.005.patch, HDFS-11558.006.patch, HDFS-11558-branch-2.006.patch,
> HDFS-11558-branch-2.8.006.patch
>
>
> Currently, the thread name looks like
> {code}
> 2017-03-20 18:32:22,022 [DataNode:
> [[[DISK]file:/Users/szetszwo/hadoop/t2/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn1_data0,
>
> [DISK]file:/Users/szetszwo/hadoop/t2/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn1_data1]]
> heartbeating to localhost/127.0.0.1:51772] INFO ...
> {code}
> which contains the full path for each storage dir. It is unnecessarily long.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11648) Lazy construct the IIP pathname
[ https://issues.apache.org/jira/browse/HDFS-11648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966624#comment-15966624 ] Junping Du commented on HDFS-11648: --- Hi [~kihwal], thanks for review and commit. As my email to hadoop dev list, we have 2.8.1 branch get cut-off for release since yesterday. Just merge the commit to branch-2.8.1 assume it is supposed to land in 2.8.1 release. Isn't it? > Lazy construct the IIP pathname > > > Key: HDFS-11648 > URL: https://issues.apache.org/jira/browse/HDFS-11648 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha3, 2.8.1 > > Attachments: HDFS-11648.patch > > > The IIP pathname is a string constructed from the byte[][] components. If > the pathname will never be accessed, ex. processing listStatus children, > building the path is unnecessarily expensive. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11163) Mover should move the file blocks to default storage once policy is unset
[
https://issues.apache.org/jira/browse/HDFS-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966620#comment-15966620
]
Junping Du commented on HDFS-11163:
---
Hi [~cnauroth], thanks for review and commit. As my email to hadoop dev list,
we have 2.8.1 branch get cut-off for release since yesterday. Just merge the
commit to branch-2.8.1 assume it is supposed to land in 2.8.1 release. Isn't it?
> Mover should move the file blocks to default storage once policy is unset
> -
>
> Key: HDFS-11163
> URL: https://issues.apache.org/jira/browse/HDFS-11163
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 3.0.0-alpha3, 2.8.1
>
> Attachments: HDFS-11163-001.patch, HDFS-11163-002.patch,
> HDFS-11163-003.patch, HDFS-11163-004.patch, HDFS-11163-005.patch,
> HDFS-11163-006.patch, HDFS-11163-007.patch, HDFS-11163-branch-2.001.patch,
> HDFS-11163-branch-2.002.patch, HDFS-11163-branch-2.003.patch,
> temp-YARN-6278.HDFS-11163.patch
>
>
> HDFS-9534 added new API in FileSystem to unset the storage policy. Once
> policy is unset blocks should move back to the default storage policy.
> Currently mover is not moving file blocks which have zero storage ID
> {code}
> // currently we ignore files with unspecified storage policy
> if (policyId == HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959711#comment-15959711 ] Junping Du commented on HDFS-11609: --- Sounds like a blocker for 2.8.1. [~kihwal] and [~jojochuang], what do you guys think? > Some blocks can be permanently lost if nodes are decommissioned while dead > -- > > Key: HDFS-11609 > URL: https://issues.apache.org/jira/browse/HDFS-11609 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-11609.branch-2.patch, HDFS-11609.trunk.patch > > > When all the nodes containing a replica of a block are decommissioned while > they are dead, they get decommissioned right away even if there are missing > blocks. This behavior was introduced by HDFS-7374. > The problem starts when those decommissioned nodes are brought back online. > The namenode no longer shows missing blocks, which creates a false sense of > cluster health. When the decommissioned nodes are removed and reformatted, > the block data is permanently lost. The namenode will report missing blocks > after the heartbeat recheck interval (e.g. 10 minutes) from the moment the > last node is taken down. > There are multiple issues in the code. As some cause different behaviors in > testing vs. production, it took a while to reproduce it in a unit test. I > will present analysis and proposal soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-11609: -- Priority: Blocker (was: Critical) > Some blocks can be permanently lost if nodes are decommissioned while dead > -- > > Key: HDFS-11609 > URL: https://issues.apache.org/jira/browse/HDFS-11609 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-11609.branch-2.patch, HDFS-11609.trunk.patch > > > When all the nodes containing a replica of a block are decommissioned while > they are dead, they get decommissioned right away even if there are missing > blocks. This behavior was introduced by HDFS-7374. > The problem starts when those decommissioned nodes are brought back online. > The namenode no longer shows missing blocks, which creates a false sense of > cluster health. When the decommissioned nodes are removed and reformatted, > the block data is permanently lost. The namenode will report missing blocks > after the heartbeat recheck interval (e.g. 10 minutes) from the moment the > last node is taken down. > There are multiple issues in the code. As some cause different behaviors in > testing vs. production, it took a while to reproduce it in a unit test. I > will present analysis and proposal soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11538) Move ClientProtocol HA proxies into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957608#comment-15957608 ] Junping Du commented on HDFS-11538: --- Thanks [~andrew.wang] and all who work on this JIRA. +1 on get this fix in 2.8.1 and revert HDFS-11431. > Move ClientProtocol HA proxies into hadoop-hdfs-client > -- > > Key: HDFS-11538 > URL: https://issues.apache.org/jira/browse/HDFS-11538 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Huafeng Wang >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11538.001.patch, HDFS-11538.002.patch, > HDFS-11538.003.patch, HDFS-11538-branch-2.001.patch > > > Follow-up for HDFS-11431. We should move this missing class over rather than > pulling in the whole hadoop-hdfs dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11302) Improve Logging for SSLHostnameVerifier
[
https://issues.apache.org/jira/browse/HDFS-11302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-11302:
--
Target Version/s: 2.8.1 (was: 2.8.0)
> Improve Logging for SSLHostnameVerifier
> ---
>
> Key: HDFS-11302
> URL: https://issues.apache.org/jira/browse/HDFS-11302
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-11302.001.patch
>
>
> SSLHostnameVerifier interface/class was copied from other projects without
> any logging to help troubleshooting SSL certificate related issues. For a
> misconfigured SSL truststore, we may get some very confusing error message
> like
> {code}
> >hdfs dfs -cat swebhdfs://NNl/tmp/test1.txt
> ...
> cause:java.io.IOException: DN2:50475: HTTPS hostname wrong: should be
> cat: DN2:50475: HTTPS hostname wrong: should be
> {code}
> This ticket is opened to add tracing to give more useful context information
> around SSL certificate verification failures inside the following code.
> {code}AbstractVerifier#check(String[] host, X509Certificate cert) {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9651) All web UIs should include a robots.txt file
[ https://issues.apache.org/jira/browse/HDFS-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946042#comment-15946042 ] Junping Du commented on HDFS-9651: -- Patch LGTM. Will commit it in next couple of days if no objections. > All web UIs should include a robots.txt file > > > Key: HDFS-9651 > URL: https://issues.apache.org/jira/browse/HDFS-9651 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-9651.1.patch, HDFS-9651.2.patch > > > Similar to HDFS-330. So that public UIs don't get crawled. > I can provide a patch that includes a simple robots.txt. Another alternative > is probably a Filter that provides one automatically for all UIs but I don't > have time to do that. > If anyone wants to take over please go ahead. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7967) Reduce the performance impact of the balancer
[ https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7967: - Target Version/s: 2.8.1 (was: 2.8.0) > Reduce the performance impact of the balancer > - > > Key: HDFS-7967 > URL: https://issues.apache.org/jira/browse/HDFS-7967 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-7967.branch-2.001.patch, > HDFS-7967.branch-2.002.patch, HDFS-7967.branch-2-1.patch, > HDFS-7967.branch-2.8.001.patch, HDFS-7967.branch-2.8.002.patch, > HDFS-7967.branch-2.8.003.patch, HDFS-7967.branch-2.8-1.patch, > HDFS-7967-branch-2.8.patch, HDFS-7967-branch-2.patch > > > The balancer needs to query for blocks to move from overly full DNs. The > block lookup is extremely inefficient. An iterator of the node's blocks is > created from the iterators of its storages' blocks. A random number is > chosen corresponding to how many blocks will be skipped via the iterator. > Each skip requires costly scanning of triplets. > The current design also only considers node imbalances while ignoring > imbalances within the nodes's storages. A more efficient and intelligent > design may eliminate the costly skipping of blocks via round-robin selection > of blocks from the storages based on remaining capacity. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11431) hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929210#comment-15929210
]
Junping Du commented on HDFS-11431:
---
bq. I want to target HDFS-11538 at 2.9.0 and 3.0.0-alpha3
Sure. Add back 2.9.0 to HDFS-11538.
bq. but if HDFS-11431 stays in branch-2, then committing HDFS-11538 to branch-2
also requires reverting HDFS-11431, and it wouldn't for trunk. It makes
tracking what's where more complicated.
We want to revert HDFS-11431 from trunk because it cause build failure. We
don't want to revert HDFS-11431 from branch-2 because it works (even like a
hack way as your said). I would like branch-2 to keep in a safe place even
adding a bit more effort to tracking differences between branch-2 and trunk.
bq. That's not possible here since HDFS-11431 doesn't work for trunk. Which is
why I suggested the above course of action.
Agree. That's why we should have one patch for branch-2/branch-2.8 and have a
different patch for trunk later.
bq. Like I said before too, since 2.9.0 isn't imminently being released, I'd
prefer the default action be "fix HDFS-13715" than "maintain the hack of
HDFS-11431". It's also easy to revisit this when 2.9.0 is closer to an RC.
I don't see HDFS-13715 will get immediately fixed in short term also - it even
haven't get any assignee yet.
My key points here:
1. HDFS-13715 is somethings TBD, for branch-2. better to have HDFS-11431 patch
than nothing.
2. HDFS-13715 is not a blocker but something nice to have for 2.9. As I
mentioned earlier, the whole feature to make hdfs-client jar thinner is not a
must given many features on 2.9 are also in pipeline.
3. If you really think tracking the revert of this patch (when we have
HDFS-13715) is a big problem, then we could file a separated JIRA and mark that
one as a blocker for 2.9 to revisit reverting patch here when we are in RC
stage.
Make sense?
> hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
> ---
>
> Key: HDFS-11431
> URL: https://issues.apache.org/jira/browse/HDFS-11431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Blocker
> Labels: maven
> Fix For: 2.8.0
>
> Attachments: HDFS-11431-branch-2.8.0.001.patch,
> HDFS-11431-branch-2.8.0.002.patch
>
>
> The {{hadoop-hdfs-client-2.8.0.jar}} file does include the
> {{ConfiguredFailoverProxyProvider}} class. This breaks client applications
> that use this class to communicate with the active NameNode in an HA
> deployment of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11538) Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929180#comment-15929180 ] Junping Du commented on HDFS-11538: --- Add back 2.9.0 given discussion in HDFS-11431. However, it is not a blocker for 2.9.0 as workaround of HDFS-11431 still works for branch-2. If we have patch here before 2.9.0 going out, then we should revert HDFS-11431. > Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client > > > Key: HDFS-11538 > URL: https://issues.apache.org/jira/browse/HDFS-11538 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > > Follow-up for HDFS-11431. We should move this missing class over rather than > pulling in the whole hadoop-hdfs dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11538) Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-11538: -- Target Version/s: 2.9.0, 3.0.0-alpha3 (was: 3.0.0-alpha3) > Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client > > > Key: HDFS-11538 > URL: https://issues.apache.org/jira/browse/HDFS-11538 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > > Follow-up for HDFS-11431. We should move this missing class over rather than > pulling in the whole hadoop-hdfs dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11431) hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928927#comment-15928927
]
Junping Du commented on HDFS-11431:
---
bq. Why not try and fix it properly for 2.9?
+1 on fixing it more properly for 2.9. However, we shouldn't be risky if the
missing class are not moved by then or other classes are found missing. I
checked the code with some HDFS guy it seems ConfiguredFailoverProxyProvider is
not very clean to move as include some server side logic. So, I think keeping
this patch on branch-2 is benign which is a different case for trunk where
build will be broken by the patch.
bq. I think it also makes the tracking easier, since otherwise the fix versions
don't reflect where the code is.
I don't understand your point. In our current practice, all 2.8.x patches
should be in branch-2 first. I think that's easier for track.
bq. The current fix thus falls in the "hack" category, and I'd rather we not
default to carrying it forward to future bramch-2 releases.
If we have elegant fixes, I am OK with get fixes in. Otherwise, HDFS-6200
doesn't achieve its goal. However, I would rather exclude one feature which
could cause regression rather than stopping the whole branch-2 release trains.
In this sense, the patch here is still benign for branch-2.
> hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
> ---
>
> Key: HDFS-11431
> URL: https://issues.apache.org/jira/browse/HDFS-11431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Blocker
> Labels: maven
> Fix For: 2.8.0
>
> Attachments: HDFS-11431-branch-2.8.0.001.patch,
> HDFS-11431-branch-2.8.0.002.patch
>
>
> The {{hadoop-hdfs-client-2.8.0.jar}} file does include the
> {{ConfiguredFailoverProxyProvider}} class. This breaks client applications
> that use this class to communicate with the active NameNode in an HA
> deployment of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11431) hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928808#comment-15928808
]
Junping Du commented on HDFS-11431:
---
Reverted.
> hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
> ---
>
> Key: HDFS-11431
> URL: https://issues.apache.org/jira/browse/HDFS-11431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Blocker
> Labels: maven
> Fix For: 2.8.0
>
> Attachments: HDFS-11431-branch-2.8.0.001.patch,
> HDFS-11431-branch-2.8.0.002.patch
>
>
> The {{hadoop-hdfs-client-2.8.0.jar}} file does include the
> {{ConfiguredFailoverProxyProvider}} class. This breaks client applications
> that use this class to communicate with the active NameNode in an HA
> deployment of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11431) hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928720#comment-15928720
]
Junping Du commented on HDFS-11431:
---
Hi [~andrew.wang], do you have more concern for patch here landing on branch-2?
If not, I will revert the previous revert on branch-2.
> hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
> ---
>
> Key: HDFS-11431
> URL: https://issues.apache.org/jira/browse/HDFS-11431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Blocker
> Labels: maven
> Fix For: 2.8.0
>
> Attachments: HDFS-11431-branch-2.8.0.001.patch,
> HDFS-11431-branch-2.8.0.002.patch
>
>
> The {{hadoop-hdfs-client-2.8.0.jar}} file does include the
> {{ConfiguredFailoverProxyProvider}} class. This breaks client applications
> that use this class to communicate with the active NameNode in an HA
> deployment of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11538) Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928708#comment-15928708 ] Junping Du commented on HDFS-11538: --- Drop 2.9 as HDFS-11431 works well for branch-2. > Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client > > > Key: HDFS-11538 > URL: https://issues.apache.org/jira/browse/HDFS-11538 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > > Follow-up for HDFS-11431. We should move this missing class over rather than > pulling in the whole hadoop-hdfs dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11538) Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-11538: -- Target Version/s: 3.0.0-alpha3 (was: 2.9.0, 3.0.0-alpha3) > Move ConfiguredFailoverProxyProvider into hadoop-hdfs-client > > > Key: HDFS-11538 > URL: https://issues.apache.org/jira/browse/HDFS-11538 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > > Follow-up for HDFS-11431. We should move this missing class over rather than > pulling in the whole hadoop-hdfs dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11431) hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928707#comment-15928707
]
Junping Du commented on HDFS-11431:
---
Branch-2 work well. HDFS-11538 should only for 3.0.
> hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
> ---
>
> Key: HDFS-11431
> URL: https://issues.apache.org/jira/browse/HDFS-11431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Blocker
> Labels: maven
> Fix For: 2.8.0
>
> Attachments: HDFS-11431-branch-2.8.0.001.patch,
> HDFS-11431-branch-2.8.0.002.patch
>
>
> The {{hadoop-hdfs-client-2.8.0.jar}} file does include the
> {{ConfiguredFailoverProxyProvider}} class. This breaks client applications
> that use this class to communicate with the active NameNode in an HA
> deployment of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11431) hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928613#comment-15928613
]
Junping Du commented on HDFS-11431:
---
bq. Did this run against trunk precommit?
I don't think so. The patch should only run against branch-2.8 given the name
marked.
I verified that branch-2 and branch-2.8 are running well. May be we should
revert the patch from trunk and file a separated JIRA to track trunk effort? -
Given the fixes for trunk/branch-2 should be significantly different.
> hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
> ---
>
> Key: HDFS-11431
> URL: https://issues.apache.org/jira/browse/HDFS-11431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Blocker
> Labels: maven
> Fix For: 2.8.0, 3.0.0-alpha3
>
> Attachments: HDFS-11431-branch-2.8.0.001.patch,
> HDFS-11431-branch-2.8.0.002.patch
>
>
> The {{hadoop-hdfs-client-2.8.0.jar}} file does include the
> {{ConfiguredFailoverProxyProvider}} class. This breaks client applications
> that use this class to communicate with the active NameNode in an HA
> deployment of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928568#comment-15928568 ] Junping Du commented on HDFS-6200: -- Due to HDFS-11431, I just updated release note here with adding "Please note that hadoop-hdfs-client module could miss class like ConfiguredFailoverProxyProvider. So if a cluster is in HA deployment, we should still use hadoop-hdfs instead." HDFS folks, please check if this is proper notes. Thanks! > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch, > HDFS-6200.005.patch, HDFS-6200.006.patch, HDFS-6200.007.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6200: - Release Note: Projects that access HDFS can depend on the hadoop-hdfs-client module instead of the hadoop-hdfs module to avoid pulling in unnecessary dependency. Please note that hadoop-hdfs-client module could miss class like ConfiguredFailoverProxyProvider. So if a cluster is in HA deployment, we should still use hadoop-hdfs instead. was:Projects that access HDFS can depend on the hadoop-hdfs-client module instead of the hadoop-hdfs module to avoid pulling in unnecessary dependency. > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch, > HDFS-6200.005.patch, HDFS-6200.006.patch, HDFS-6200.007.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11431) hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-11431:
--
Fix Version/s: 3.0.0-alpha3
> hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider
> ---
>
> Key: HDFS-11431
> URL: https://issues.apache.org/jira/browse/HDFS-11431
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Blocker
> Labels: maven
> Fix For: 2.8.0, 3.0.0-alpha3
>
> Attachments: HDFS-11431-branch-2.8.0.001.patch,
> HDFS-11431-branch-2.8.0.002.patch
>
>
> The {{hadoop-hdfs-client-2.8.0.jar}} file does include the
> {{ConfiguredFailoverProxyProvider}} class. This breaks client applications
> that use this class to communicate with the active NameNode in an HA
> deployment of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11441) Add escaping to error message in KMS web UI
[ https://issues.apache.org/jira/browse/HDFS-11441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897862#comment-15897862 ] Junping Du commented on HDFS-11441: --- How serious the issue here could be? If it belongs to minor as it claim to be, I would suggest better to leave it to 2.8.1. Otherwise, please bump up to critical and leave comments for justification. > Add escaping to error message in KMS web UI > --- > > Key: HDFS-11441 > URL: https://issues.apache.org/jira/browse/HDFS-11441 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.8.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha3, 2.8.1 > > Attachments: HDFS-11441-branch-2.6.patch, HDFS-11441.patch, > HDFS-11441.patch > > > There's a handful of places where web UIs don't escape error messages. We > should add escaping in these places. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11466) Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms
[ https://issues.apache.org/jira/browse/HDFS-11466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886988#comment-15886988 ] Junping Du commented on HDFS-11466: --- Thanks [~andrew.wang] for pinging me on this. Yes. 2.8.0 is pending on HADOOP-13866 decision which should get resolved soon. About patch here, if it is really important (seems unlikely at first glance), we can get it to 2.8.0. Otherwise, my suggestion is better to leave it to 2.8.1. > Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to > 5000ms > --- > > Key: HDFS-11466 > URL: https://issues.apache.org/jira/browse/HDFS-11466 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-11466.001.patch > > > Per discussion on HDFS-10798, it might make sense to change the default value > for long write lock holds to 5000ms like the read threshold, to avoid > spamming the log. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862289#comment-15862289 ] Junping Du commented on HDFS-11379: --- bq. Junping Du, this looks to be important enough to put in 2.8.0. Is it okay to commit this to the release branch? Hi [~kihwal], there is another blocker for 2.8.0 - YARN-6147 to fix an incompatible issues. So I am ok to put this patch to 2.8.0 if this is really important. > DFSInputStream may infinite loop requesting block locations > --- > > Key: HDFS-11379 > URL: https://issues.apache.org/jira/browse/HDFS-11379 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 3.0.0-alpha3, 2.8.1 > > Attachments: HDFS-11379.branch-2.patch, HDFS-11379.trunk.patch > > > DFSInputStream creation caches file size and initial range of locations. If > the file is truncated (or replaced) and the client attempts to read outside > the initial range, the client goes into a tight infinite looping requesting > locations for the nonexistent range. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9433) DFS getEZForPath API on a non-existent file should throw FileNotFoundException
[
https://issues.apache.org/jira/browse/HDFS-9433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-9433:
-
Release Note: Unify the behavior of dfs.getEZForPath() API when getting a
non-existent normal file and non-existent ezone file
> DFS getEZForPath API on a non-existent file should throw FileNotFoundException
> --
>
> Key: HDFS-9433
> URL: https://issues.apache.org/jira/browse/HDFS-9433
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: encryption
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9433-00.patch, HDFS-9433-00.patch
>
>
> Presently {{dfs.getEZForPath()}} API is behaving differently for a
> non-existent normal file and non-existent ezone file:
> - If user pass a normal non-existent file then it will return null value. For
> example, {{Path("/nonexistentfile")}}
> - If user pass a non-existent file but which is under an existing encryption
> zone then it is returning the parent's encryption zone info. For example,
> {{Path("/ezone/nonexistentfile")}}
> Here the proposed idea is to unify the behavior by throwing
> FileNotFoundException. Please refer the discussion
> [thread|https://issues.apache.org/jira/browse/HDFS-9348?focusedCommentId=14983301=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14983301].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9433) DFS getEZForPath API on a non-existent file should throw FileNotFoundException
[
https://issues.apache.org/jira/browse/HDFS-9433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-9433:
-
Release Note: Unify the behavior of dfs.getEZForPath() API when getting a
non-existent normal file and non-existent ezone file by throwing
FileNotFoundException (was: Unify the behavior of dfs.getEZForPath() API when
getting a non-existent normal file and non-existent ezone file)
> DFS getEZForPath API on a non-existent file should throw FileNotFoundException
> --
>
> Key: HDFS-9433
> URL: https://issues.apache.org/jira/browse/HDFS-9433
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: encryption
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-9433-00.patch, HDFS-9433-00.patch
>
>
> Presently {{dfs.getEZForPath()}} API is behaving differently for a
> non-existent normal file and non-existent ezone file:
> - If user pass a normal non-existent file then it will return null value. For
> example, {{Path("/nonexistentfile")}}
> - If user pass a non-existent file but which is under an existing encryption
> zone then it is returning the parent's encryption zone info. For example,
> {{Path("/ezone/nonexistentfile")}}
> Here the proposed idea is to unify the behavior by throwing
> FileNotFoundException. Please refer the discussion
> [thread|https://issues.apache.org/jira/browse/HDFS-9348?focusedCommentId=14983301=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14983301].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11312) Fix incompatible tag number change for nonDfsUsed in DatanodeInfoProto
[ https://issues.apache.org/jira/browse/HDFS-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819626#comment-15819626 ] Junping Du commented on HDFS-11312: --- Merge to branch-2.8.0. > Fix incompatible tag number change for nonDfsUsed in DatanodeInfoProto > -- > > Key: HDFS-11312 > URL: https://issues.apache.org/jira/browse/HDFS-11312 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Blocker > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11312.001.patch > > > The patches for HDFS-9038 had a discrepancy between trunk and branch-2.7: in > one message type, nonDfsUsed is given 2 different indices. This is a minor > wire incompatibility that is easy to fix... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9558) Replication requests from datanode always blames the source datanode in case of Checksum Exception.
[
https://issues.apache.org/jira/browse/HDFS-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-9558:
-
Target Version/s: (was: 2.8.0)
> Replication requests from datanode always blames the source datanode in case
> of Checksum Exception.
> ---
>
> Key: HDFS-9558
> URL: https://issues.apache.org/jira/browse/HDFS-9558
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Rushabh S Shah
>
> Replication requests from datanode (in case of rack failure event) always
> blames the source datanode if any of the downstream nodes encounters
> ChecksumException.
> We saw this case recently in our cluster.
> We lost 7 nodes in a rack.
> There was only one replica of the block (say on dnA).
> The namenode asks dnA to replicate to dnB and dnC.
> {noformat}
> 2015-12-13 21:09:41,798 [DataNode: heartbeating to NN:8020] INFO
> datanode.DataNode: DatanodeRegistration(dnA,
> datanodeUuid=bc1f183d-b74a-49c9-ab1a-d1d496ab77e9, infoPort=1006,
> infoSecurePort=0, ipcPort=8020,
> storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571)
> Starting thread to transfer
> BP-1620678153--1351096255769:blk_3065507810_1107476861617 to dnB:1004
> dnC:1004
> {noformat}
> All the packets going out from dnB's interface were getting corrupted.
> So dnC received corrupt block and it reported bad block (from dnA) to
> namenode.
> Following are the logs from dnC:
> {noformat}
> 2015-12-13 21:09:43,444 [DataXceiver for client at /dnB:34879 [Receiving
> block BP-1620678153--1351096255769:blk_3065507810_1107476861617]] WARN
> datanode.DataNode: Checksum error in block
> BP-1620678153--1351096255769:blk_3065507810_1107476861617 from /dnB:34879
> org.apache.hadoop.fs.ChecksumException: Checksum error: at 58368 exp:
> -1657951272 got: 856104973
> at
> org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSumsByteArray(NativeCrc32.java:69)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:347)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:294)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.verifyChunks(BlockReceiver.java:416)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:550)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:853)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:761)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:237)
> at java.lang.Thread.run(Thread.java:745)
> 2015-12-13 21:09:43,445 [DataXceiver for client at dnB:34879 [Receiving
> block BP-1620678153--1351096255769:blk_3065507810_1107476861617]] INFO
> datanode.DataNode: report corrupt
> BP-1620678153--1351096255769:blk_3065507810_1107476861617 from datanode
> dnA:1004 to namenode
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9020) Support hflush/hsync in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-9020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9020: - Target Version/s: (was: 2.8.0) > Support hflush/hsync in WebHDFS > --- > > Key: HDFS-9020 > URL: https://issues.apache.org/jira/browse/HDFS-9020 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: Chris Douglas > Attachments: HDFS-9020-alt.txt > > > In the current implementation, hflush/hsync have no effect on WebHDFS > streams, particularly w.r.t. visibility to other clients. This proposes to > extend the protocol and implementation to enable this functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11246) FSNameSystem#logAuditEvent should be called outside the read or write locks during operations like getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-11246:
--
Target Version/s: (was: 2.8.0)
> FSNameSystem#logAuditEvent should be called outside the read or write locks
> during operations like getContentSummary
>
>
> Key: HDFS-11246
> URL: https://issues.apache.org/jira/browse/HDFS-11246
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>
> {code}
> readLock();
> boolean success = true;
> ContentSummary cs;
> try {
> checkOperation(OperationCategory.READ);
> cs = FSDirStatAndListingOp.getContentSummary(dir, src);
> } catch (AccessControlException ace) {
> success = false;
> logAuditEvent(success, operationName, src);
> throw ace;
> } finally {
> readUnlock(operationName);
> }
> {code}
> It would be nice to have audit logging outside the lock esp. in scenarios
> where applications hammer a given operation several times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6661) Use BlockList instead of double linked list i.e BlockInfo triplets
[ https://issues.apache.org/jira/browse/HDFS-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6661: - Target Version/s: (was: 2.8.0) > Use BlockList instead of double linked list i.e BlockInfo triplets > > > Key: HDFS-6661 > URL: https://issues.apache.org/jira/browse/HDFS-6661 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.4.1 >Reporter: Amir Langer >Assignee: Amir Langer > Attachments: > 0003-use-BlockList-instead-of-a-double-linked-list-of-blo.patch > > > Replace the triplets data structure with a BlockList instance per > DatanodeStorageInfo. This requires to keep only two integers per replica in > any BlockInfo object. (One is the index for the DatanodeStorageInfo and the > other is the position of the block in that DatanodeStorageInfo BlockList). > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10506) OIV's ReverseXML processor cannot reconstruct some snapshot details
[ https://issues.apache.org/jira/browse/HDFS-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-10506: -- Target Version/s: (was: 2.8.0) > OIV's ReverseXML processor cannot reconstruct some snapshot details > --- > > Key: HDFS-10506 > URL: https://issues.apache.org/jira/browse/HDFS-10506 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Colin P. McCabe > > OIV's ReverseXML processor cannot reconstruct some snapshot details. > Specifically, should contain a and field, > but does not. should contain a field. OIV also > needs to be changed to emit these fields into the XML (they are currently > missing). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7304) TestFileCreation#testOverwriteOpenForWrite hangs
[ https://issues.apache.org/jira/browse/HDFS-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7304: - Target Version/s: 2.9.0 (was: 2.8.0, 2.9.0) > TestFileCreation#testOverwriteOpenForWrite hangs > > > Key: HDFS-7304 > URL: https://issues.apache.org/jira/browse/HDFS-7304 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Akira Ajisaka > Attachments: HDFS-7304.patch, HDFS-7304.patch > > > The test case times out. It has been observed in multiple pre-commit builds. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10770) Namenode shouldn't change storage info if client reports different storage. (via reportBadBlock)
[
https://issues.apache.org/jira/browse/HDFS-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-10770:
--
Target Version/s: (was: 2.8.0)
> Namenode shouldn't change storage info if client reports different storage.
> (via reportBadBlock)
>
>
> Key: HDFS-10770
> URL: https://issues.apache.org/jira/browse/HDFS-10770
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rushabh S Shah
>
> From HDFS-10348 [comment |
> https://issues.apache.org/jira/browse/HDFS-10348?focusedCommentId=15271522=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271522].
> This jira is created to discuss whether we should change the storageInfo in
> triplets if the client reports bad block with storage which is different than
> what the namenode thinks.
> Here is the relevant code:
> {code:title=DatanodeStorageInfo.java|borderStyle=solid}
> public AddBlockResult addBlock(BlockInfo b, Block reportedBlock) {
> // First check whether the block belongs to a different storage
> // on the same DN.
> AddBlockResult result = AddBlockResult.ADDED;
> DatanodeStorageInfo otherStorage =
> b.findStorageInfo(getDatanodeDescriptor());
> if (otherStorage != null) {
> if (otherStorage != this) {
> // The block belongs to a different storage. Remove it first.
> otherStorage.removeBlock(b);
> result = AddBlockResult.REPLACED;
> } else {
> // The block is already associated with this storage.
> return AddBlockResult.ALREADY_EXIST;
> }
> }
> b.addStorage(this, reportedBlock);
> blocks.add(b);
> return result;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11090) Leave safemode immediately if all blocks have reported in
[ https://issues.apache.org/jira/browse/HDFS-11090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-11090: -- Target Version/s: (was: 2.8.0) > Leave safemode immediately if all blocks have reported in > - > > Key: HDFS-11090 > URL: https://issues.apache.org/jira/browse/HDFS-11090 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.3 >Reporter: Andrew Wang >Assignee: Yiqun Lin > Attachments: HDFS-11090.001.patch > > > Startup safemode is triggered by two thresholds: % blocks reported in, and > min # datanodes. It's extended by an interval (default 30s) until these two > thresholds are met. > Safemode extension is helpful when the cluster has data, and the default % > blocks threshold (0.99) is used. It gives DNs a little extra time to report > in and thus avoid unnecessary replication work. > However, we can leave startup safemode early if 100% of blocks have reported > in. > Note that operators sometimes change the % blocks threshold to > 1 to never > automatically leave safemode. We should maintain this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11260) Slow writer threads are not stopped
[
https://issues.apache.org/jira/browse/HDFS-11260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-11260:
--
Target Version/s: 3.0.0-beta1 (was: 2.8.0, 3.0.0-beta1)
> Slow writer threads are not stopped
> ---
>
> Key: HDFS-11260
> URL: https://issues.apache.org/jira/browse/HDFS-11260
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> If a DataNode receives a transferred block, it tries to stop writer to the
> same block. However, this may not work, and we saw the following error
> message and stacktrace.
> Fundamentally, the assumption of {{ReplicaInPipeline#stopWriter}} is wrong.
> It assumes the writer thread must be a DataXceiver thread, which it can be
> interrupted and terminates afterwards. However, IPC threads may also be the
> writer thread by calling initReplicaRecovery, and which ignores interrupt and
> do not terminate.
> {noformat}
> 2016-12-16 19:58:56,167 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Join on writer thread Thread[IPC Server handler 6 on 50020,5,main] timed out
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:135)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2052)
> 2016-12-16 19:58:56,167 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver constructor. Cause is
> 2016-12-16 19:58:56,168 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> sj1dra082.corp.adobe.com:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /10.10.0.80:44105 dst: /10.10.0.82:50010
> java.io.IOException: Join on writer thread Thread[IPC Server handler 6 on
> 50020,5,main] timed out
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:212)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:1579)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:195)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:669)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There is also a logic error in FsDatasetImpl#createTemporary, in which if the
> code in the synchronized block executes for more than 60 seconds (in theory),
> it could throw an exception, without trying to stop the existing slow writer.
> We saw a FsDatasetImpl#createTemporary failed after nearly 10 minutes, and
> it's unclear why yet. It's my understanding that the code intends to stop
> slow writers after 1 minute by default. Some code rewrite is probably needed
> to get the logic right.
> {noformat}
> 2016-12-16 23:12:24,636 WARN
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Unable
> to stop existing writer for block
> BP-1527842723-10.0.0.180-1367984731269:blk_4313782210_1103780331023 after
> 568320 miniseconds.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6122) Rebalance cached replicas between datanodes
[ https://issues.apache.org/jira/browse/HDFS-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6122: - Target Version/s: (was: 2.8.0) > Rebalance cached replicas between datanodes > --- > > Key: HDFS-6122 > URL: https://issues.apache.org/jira/browse/HDFS-6122 > Project: Hadoop HDFS > Issue Type: Improvement > Components: caching >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > It'd be nice if the NameNode was able to rebalance cache usage among > datanodes. This would help avoid situations where the only three DNs with a > replica are full and there is still cache space on the rest of the cluster. > It'll also probably help for heterogeneous node sizes and when adding new > nodes to the cluster or doing a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10406) Test failure on trunk: TestReconstructStripedBlocks
[ https://issues.apache.org/jira/browse/HDFS-10406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-10406: -- Target Version/s: (was: 2.8.0) > Test failure on trunk: TestReconstructStripedBlocks > --- > > Key: HDFS-10406 > URL: https://issues.apache.org/jira/browse/HDFS-10406 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou > > It's been noticed there are some test failures: TestEditLog and > TestReconstructStripedBlocks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6708) StorageType should be encoded in the block token
[ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6708: - Target Version/s: (was: 2.8.0) > StorageType should be encoded in the block token > > > Key: HDFS-6708 > URL: https://issues.apache.org/jira/browse/HDFS-6708 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.4.1 >Reporter: Arpit Agarwal >Assignee: Pieter Reuse > > HDFS-6702 is adding support for file creation based on StorageType. > The block token is used as a tamper-proof channel for communicating block > parameters from the NN to the DN during block creation. The StorageType > should be included in this block token. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9499) Fix typos in DFSAdmin.java
[ https://issues.apache.org/jira/browse/HDFS-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9499: - Target Version/s: (was: 2.8.0) > Fix typos in DFSAdmin.java > -- > > Key: HDFS-9499 > URL: https://issues.apache.org/jira/browse/HDFS-9499 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Arpit Agarwal >Assignee: Daniel Green > > There are multiple instances of 'snapshot' spelled as 'snaphot' in > DFSAdmin.java and TestSnapshotCommands.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6212) Deprecate the BackupNode and CheckpointNode from branch-2
[ https://issues.apache.org/jira/browse/HDFS-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6212: - Target Version/s: (was: 2.8.0) > Deprecate the BackupNode and CheckpointNode from branch-2 > - > > Key: HDFS-6212 > URL: https://issues.apache.org/jira/browse/HDFS-6212 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.3.0 >Reporter: Jing Zhao > > As per discussion in HDFS-4114, this jira tries to deprecate BackupNode from > branch-2 and change the hadoop start/stop scripts to print deprecation > warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6220) Webhdfs should differentiate remote exceptions
[ https://issues.apache.org/jira/browse/HDFS-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6220: - Target Version/s: (was: 2.8.0) > Webhdfs should differentiate remote exceptions > -- > > Key: HDFS-6220 > URL: https://issues.apache.org/jira/browse/HDFS-6220 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > > Webhdfs's validateResponse should use a distinct exception for json decoded > exceptions than it does for servlet exceptions. Ex. A servlet generated 404 > with a json payload should be distinguishable from a http proxy or jetty > generated 404. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-8982:
-
Target Version/s: (was: 2.8.0)
> Consolidate getFileReplication and getPreferredBlockReplication in INodeFile
>
>
> Key: HDFS-8982
> URL: https://issues.apache.org/jira/browse/HDFS-8982
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>
> Currently {{INodeFile}} provides both {{getFileReplication}} and
> {{getPreferredBlockReplication}} interfaces. At the very least they should be
> renamed (e.g. {{getCurrentFileReplication}} and
> {{getMaxConfiguredFileReplication}}), with clearer Javadoc.
> I also suspect we are not using them correctly in all places right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6884) Include the hostname in HTTPFS log filenames
[
https://issues.apache.org/jira/browse/HDFS-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-6884:
-
Target Version/s: (was: 2.8.0)
> Include the hostname in HTTPFS log filenames
>
>
> Key: HDFS-6884
> URL: https://issues.apache.org/jira/browse/HDFS-6884
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.5.0
>Reporter: Andrew Wang
>Assignee: Alejandro Abdelnur
>
> It'd be good to include the hostname in the httpfs log filenames. Right now
> we have httpfs.log and httpfs-audit.log, it'd be nice to have e.g.
> "httpfs-${hostname}.log".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6891) Follow-on work for transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6891: - Target Version/s: (was: 2.8.0) > Follow-on work for transparent data at rest encryption > -- > > Key: HDFS-6891 > URL: https://issues.apache.org/jira/browse/HDFS-6891 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 2.5.0 >Reporter: Andrew Wang >Assignee: Charles Lamb > > This is an umbrella JIRA to track remaining subtasks from HDFS-6134. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6935) RawLocalFileSystem::supportsSymLinks is confusing
[
https://issues.apache.org/jira/browse/HDFS-6935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-6935:
-
Target Version/s: (was: 2.8.0)
> RawLocalFileSystem::supportsSymLinks is confusing
> --
>
> Key: HDFS-6935
> URL: https://issues.apache.org/jira/browse/HDFS-6935
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: symlinks
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Gopal V
>
> {code}
> ...
> @Override
> public boolean supportsSymlinks() {
> return true;
> }
> @SuppressWarnings("deprecation")
> @Override
> public void createSymlink(Path target, Path link, boolean createParent)
> throws IOException {
> if (!FileSystem.areSymlinksEnabled()) {
> throw new UnsupportedOperationException("Symlinks not supported");
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6691) The message on NN UI can be confusing during a rolling upgrade
[ https://issues.apache.org/jira/browse/HDFS-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6691: - Target Version/s: (was: 2.8.0) > The message on NN UI can be confusing during a rolling upgrade > --- > > Key: HDFS-6691 > URL: https://issues.apache.org/jira/browse/HDFS-6691 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: ha1.png, ha2.png > > > On ANN, it says rollback image was created. On SBN, it says otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6846) NetworkTopology#sortByDistance should give nodes higher priority, which cache the block.
[ https://issues.apache.org/jira/browse/HDFS-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6846: - Target Version/s: (was: 2.8.0) > NetworkTopology#sortByDistance should give nodes higher priority, which cache > the block. > > > Key: HDFS-6846 > URL: https://issues.apache.org/jira/browse/HDFS-6846 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Yi Liu >Assignee: Yi Liu > > Currently there are 3 weights: > * local > * same rack > * off rack > But if some nodes cache the block, then it's faster if client read block from > these nodes. So we should have some more weights as following: > * local > * cached & same rack > * same rack > * cached & off rack > * off rack -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6919) Enforce a single limit for RAM disk usage and replicas cached via locking
[ https://issues.apache.org/jira/browse/HDFS-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6919: - Target Version/s: (was: 2.8.0) > Enforce a single limit for RAM disk usage and replicas cached via locking > - > > Key: HDFS-6919 > URL: https://issues.apache.org/jira/browse/HDFS-6919 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > The DataNode can have a single limit for memory usage which applies to both > replicas cached via CCM and replicas on RAM disk. > See comments > [1|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106025=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106025], > > [2|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106245=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106245] > and > [3|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106575=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106575] > for discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6764) DN heartbeats may become clumped together
[ https://issues.apache.org/jira/browse/HDFS-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6764: - Target Version/s: (was: 2.8.0) > DN heartbeats may become clumped together > - > > Key: HDFS-6764 > URL: https://issues.apache.org/jira/browse/HDFS-6764 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Daryn Sharp > Attachments: Screen Shot 2014-07-28 at 11.12.06 AM.png > > > DNs send heartbeats on a fixed schedule based on the last time a heartbeat > was sent. If the NN takes longer to respond than the heartbeat interval then > DNs do not sleep until the next interval. Instead, another heartbeat is > immediately sent and all DNs begin heartbeating on the same schedule. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6333) REST API for fetching directory listing file from NN
[ https://issues.apache.org/jira/browse/HDFS-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6333: - Target Version/s: (was: 2.8.0) > REST API for fetching directory listing file from NN > > > Key: HDFS-6333 > URL: https://issues.apache.org/jira/browse/HDFS-6333 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.4.0 >Reporter: Andrew Wang > > It'd be convenient if the NameNode supported fetching the directory listing > via HTTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6128) Implement libhdfs bindings for HDFS ACL APIs.
[ https://issues.apache.org/jira/browse/HDFS-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6128: - Target Version/s: (was: 2.8.0) > Implement libhdfs bindings for HDFS ACL APIs. > - > > Key: HDFS-6128 > URL: https://issues.apache.org/jira/browse/HDFS-6128 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs >Affects Versions: 2.4.0 >Reporter: Chris Nauroth > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8510) Provide different timeout settings for hdfs dfsadmin -getDatanodeInfo.
[
https://issues.apache.org/jira/browse/HDFS-8510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-8510:
-
Target Version/s: (was: 2.8.0)
> Provide different timeout settings for hdfs dfsadmin -getDatanodeInfo.
> --
>
> Key: HDFS-8510
> URL: https://issues.apache.org/jira/browse/HDFS-8510
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>
> During a rolling upgrade, an administrator runs {{hdfs dfsadmin
> -getDatanodeInfo}} to check if a DataNode has stopped. Currently, this
> operation is subject to the RPC connection retries defined in
> {{ipc.client.connect.max.retries}} and {{ipc.client.connect.retry.interval}}.
> This issue proposes adding separate configuration properties to control the
> retries for this operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8555) Random read support on HDFS files using Indexed Namenode feature
[ https://issues.apache.org/jira/browse/HDFS-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8555: - Target Version/s: (was: 2.8.0) > Random read support on HDFS files using Indexed Namenode feature > > > Key: HDFS-8555 > URL: https://issues.apache.org/jira/browse/HDFS-8555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Affects Versions: 2.5.2 > Environment: Linux >Reporter: amit sehgal >Assignee: Afzal Saan > Original Estimate: 720h > Remaining Estimate: 720h > > Currently Namenode does not provide support to do random reads. With so many > tools built on top of HDFS solving the use case of Exploratory BI and > providing SQL over HDFS. The need of hour is to reduce the number of blocks > read for a Random read. > E.g. extracting say 10 lines worth of information out of 100GB files should > be reading only those block which can potentially have those 10 lines. > This can be achieved by adding a tagging feature per block in name node, each > block written to HDFS will have tags associated to it stored in index. > Namednode when access via the Indexing feature will use this index native to > reduce the no. of block returned to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6151) HDFS should refuse to cache blocks >=2GB
[
https://issues.apache.org/jira/browse/HDFS-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-6151:
-
Target Version/s: (was: 2.8.0)
> HDFS should refuse to cache blocks >=2GB
>
>
> Key: HDFS-6151
> URL: https://issues.apache.org/jira/browse/HDFS-6151
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching, datanode
>Affects Versions: 2.4.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>
> If you try to cache a block that's >=2GB, the DN will silently fail to cache
> it since {{MappedByteBuffer}} uses a signed int to represent size. Blocks
> this large are rare, but we should log or alert the user somehow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6332) Support protobuf-based directory listing output suitable for OIV
[ https://issues.apache.org/jira/browse/HDFS-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6332: - Target Version/s: (was: 2.8.0) > Support protobuf-based directory listing output suitable for OIV > > > Key: HDFS-6332 > URL: https://issues.apache.org/jira/browse/HDFS-6332 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.4.0 >Reporter: Andrew Wang > > The initial listing output may be in JSON, but protobuf would be a better > serialization format down the road. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6292) Display HDFS per user and per group usage on the webUI
[ https://issues.apache.org/jira/browse/HDFS-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6292: - Target Version/s: (was: 2.8.0) > Display HDFS per user and per group usage on the webUI > -- > > Key: HDFS-6292 > URL: https://issues.apache.org/jira/browse/HDFS-6292 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Attachments: HDFS-6292.01.patch, HDFS-6292.patch, HDFS-6292.png > > > It would be nice to show HDFS usage per user and per group on a web ui. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6624) TestBlockTokenWithDFS#testEnd2End fails sometimes
[
https://issues.apache.org/jira/browse/HDFS-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-6624:
-
Target Version/s: (was: 2.8.0)
> TestBlockTokenWithDFS#testEnd2End fails sometimes
> -
>
> Key: HDFS-6624
> URL: https://issues.apache.org/jira/browse/HDFS-6624
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
> Attachments: PreCommit-HDFS-Build #7274 test - testEnd2End
> [Jenkins].html
>
>
> On a recent test-patch.sh run, saw this error which did not repro locally:
> {noformat}
> Error Message
> Rebalancing expected avg utilization to become 0.2, but on datanode
> 127.0.0.1:57889 it remains at 0.08 after more than 4 msec.
> Stacktrace
> java.util.concurrent.TimeoutException: Rebalancing expected avg utilization
> to become 0.2, but on datanode 127.0.0.1:57889 it remains at 0.08 after more
> than 4 msec.
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForBalancer(TestBalancer.java:284)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:382)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:359)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:403)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.integrationTest(TestBalancer.java:416)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.testEnd2End(TestBlockTokenWithDFS.java:588)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8691) Cleanup BlockScanner initialization and add test for configuration contract
[
https://issues.apache.org/jira/browse/HDFS-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-8691:
-
Target Version/s: (was: 2.8.0)
> Cleanup BlockScanner initialization and add test for configuration contract
> ---
>
> Key: HDFS-8691
> URL: https://issues.apache.org/jira/browse/HDFS-8691
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Arpit Agarwal
>Assignee: Jagadesh Kiran N
>
> The initialization of the BlockScanner can be simplified by moving out test
> hooks. Tests can be modified to use configuration only.
> Also we need an additional test case to verify the behavior with positive,
> negative and zero values of {{dfs.datanode.scan.period.hours}} for
> compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10294) Balancer configures DN properties
[
https://issues.apache.org/jira/browse/HDFS-10294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-10294:
--
Target Version/s: (was: 2.8.0)
> Balancer configures DN properties
> -
>
> Key: HDFS-10294
> URL: https://issues.apache.org/jira/browse/HDFS-10294
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> Balancer configures the 2 balance-related properties (bandwidthPerSec and
> concurrentMoves) using NN API {{get/setBalancerBandwidth}} and the new
> {{get/setBalancerConcurrentMoves}}.
> Details:
> * Upon the start of the balancer, set the DN properties.
> * Use NN API to query and set the 2 properties. There might be a slight delay
> for the property changes to be propagated to all DNs.
> * The DN property changes will not survive restart.
> * Balancer gets the property values from command line or its config file.
> * No need to edit the config file on each DN or run {{hdfs dfsadmin
> -setBalancerBandwidth}} to configure every DN in the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8820) Simplify enabling NameNode RPC congestion control and FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8820: - Target Version/s: (was: 2.8.0) > Simplify enabling NameNode RPC congestion control and FairCallQueue > --- > > Key: HDFS-8820 > URL: https://issues.apache.org/jira/browse/HDFS-8820 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8820.01.patch, HDFS-8820.02.patch, > HDFS-8820.03.patch > > > Enabling RPC Congestion control and FairCallQueue settings can be simplified > with HDFS-specific configuration keys. Currently the configuration requires > knowing the exact RPC port number and also whether the service RPC port is > enabled or not separately. If a separate service RPC endpoint is not defined > then RPC congestion control must be enabled ([see > comment|https://issues.apache.org/jira/browse/HDFS-8820?focusedCommentId=14987848=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14987848] > from [~mingma] below. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8655) Refactor accesses to INodeFile#blocks
[
https://issues.apache.org/jira/browse/HDFS-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-8655:
-
Target Version/s: (was: 2.8.0)
> Refactor accesses to INodeFile#blocks
> -
>
> Key: HDFS-8655
> URL: https://issues.apache.org/jira/browse/HDFS-8655
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8655.00.patch, HDFS-8655.01.patch
>
>
> When enabling INodeFile support for striped blocks (mainly in HDFS-7749),
> HDFS-7285 branch generalized the concept of blocks under an inode. Now
> {{INodeFile#blocks}} only contains contiguous blocks of an inode. This JIRA
> separates out code refactors for this purpose. Two main changes:
> # Rename {{setBlocks}} to {{setContiguousBlocks}}
> # Replace direct accesses to {{INodeFile#blocks}} to {{getBlocks}}
> It also contains some code cleanups introduced in the branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10174) Add HTrace support to the Balancer
[ https://issues.apache.org/jira/browse/HDFS-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-10174: -- Target Version/s: (was: 2.8.0) > Add HTrace support to the Balancer > -- > > Key: HDFS-10174 > URL: https://issues.apache.org/jira/browse/HDFS-10174 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe > > The HDFS Balancer should have HTrace support so that we can see what's > happening across the cluster during a balancer run. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8776) Decom manager should not be active on standby
[ https://issues.apache.org/jira/browse/HDFS-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8776: - Target Version/s: (was: 2.8.0) > Decom manager should not be active on standby > - > > Key: HDFS-8776 > URL: https://issues.apache.org/jira/browse/HDFS-8776 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > > The decommission manager should not be actively processing on the standby. > The decomm manager goes through the costly computation for determining every > block on the node requires replication yet doesn't queue them for replication > - because it's in standby. The decomm manager is holding the namesystem write > lock, causing DNs to timeout on heartbeats or IBRs, NN purges the call queue > of timed out clients, NN processes some heartbeats/IBRs before the decomm > manager locks up the namesystem again. Nodes attempting to register will be > sending full BRs which are more costly to send and discard than a heartbeat. > If a failover is required, the standby will likely have to struggle very hard > to not GC while "catching up" on its queued IBRs while DNs continue to fill > the call queue and time out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8636) Tolerate disk errors during block pool initialization
[ https://issues.apache.org/jira/browse/HDFS-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8636: - Target Version/s: (was: 2.8.0) > Tolerate disk errors during block pool initialization > - > > Key: HDFS-8636 > URL: https://issues.apache.org/jira/browse/HDFS-8636 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > > If the DN hits an IO error during block pool initialization it aborts since > it does not respect the tolerated volumes configuration key. We saw this > error in particular when initializing a new block pool slice, but would also > apply to the scan to populate the replica map. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8584) Support using ramfs partitions on Linux
[
https://issues.apache.org/jira/browse/HDFS-8584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-8584:
-
Target Version/s: (was: 2.8.0)
> Support using ramfs partitions on Linux
> ---
>
> Key: HDFS-8584
> URL: https://issues.apache.org/jira/browse/HDFS-8584
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Now that the bulk of work for HDFS-6919 is complete the memory limit
> enforcement uses the {{dfs.datanode.max.locked.memory}} setting and not the
> RAM disk free space availability.
> We can now use ramfs partitions. This will require fixing the free space
> computation and reservation logic for transient volumes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9913) DispCp doesn't use Trash with -delete option
[ https://issues.apache.org/jira/browse/HDFS-9913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9913: - Target Version/s: (was: 2.8.0) > DispCp doesn't use Trash with -delete option > > > Key: HDFS-9913 > URL: https://issues.apache.org/jira/browse/HDFS-9913 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Affects Versions: 2.6.0 >Reporter: Konstantin Shaposhnikov >Assignee: John Zhuge > > Documentation for DistCp -delete option says > ([http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html]): > | The deletion is done by FS Shell. So the trash will be used, if it is > enable. > However it seems to be no longer the case. The latest source code > (https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java) > uses `FileSystem.delete` and trash options seems to be not applied. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3296) Running libhdfs tests in mac fails
[
https://issues.apache.org/jira/browse/HDFS-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-3296:
-
Target Version/s: (was: 2.8.0)
> Running libhdfs tests in mac fails
> --
>
> Key: HDFS-3296
> URL: https://issues.apache.org/jira/browse/HDFS-3296
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Reporter: Amareshwari Sriramadasu
>Assignee: Chris Nauroth
> Attachments: HDFS-3296.001.patch, HDFS-3296.002.patch,
> HDFS-3296.003.patch
>
>
> Running "ant -Dcompile.c++=true -Dlibhdfs=true test-c++-libhdfs" on Mac fails
> with following error:
> {noformat}
> [exec] dyld: lazy symbol binding failed: Symbol not found:
> _JNI_GetCreatedJavaVMs
> [exec] Referenced from:
> /Users/amareshwari.sr/workspace/hadoop/build/c++/Mac_OS_X-x86_64-64/lib/libhdfs.0.dylib
> [exec] Expected in: flat namespace
> [exec]
> [exec] dyld: Symbol not found: _JNI_GetCreatedJavaVMs
> [exec] Referenced from:
> /Users/amareshwari.sr/workspace/hadoop/build/c++/Mac_OS_X-x86_64-64/lib/libhdfs.0.dylib
> [exec] Expected in: flat namespace
> [exec]
> [exec]
> /Users/amareshwari.sr/workspace/hadoop/src/c++/libhdfs/tests/test-libhdfs.sh:
> line 122: 39485 Trace/BPT trap: 5 CLASSPATH=$HADOOP_CONF_DIR:$CLASSPATH
> LD_PRELOAD="$LIB_JVM_DIR/libjvm.so:$LIBHDFS_INSTALL_DIR/libhdfs.so:"
> $LIBHDFS_BUILD_DIR/$HDFS_TEST
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7508) Clean up path resolving in FSNamesystem/FSDirectory
[ https://issues.apache.org/jira/browse/HDFS-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7508: - Target Version/s: (was: 2.8.0) > Clean up path resolving in FSNamesystem/FSDirectory > --- > > Key: HDFS-7508 > URL: https://issues.apache.org/jira/browse/HDFS-7508 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > > There are several issues with our current path resolving process: > # For a single operation we may repeat resolving the same path > # Each operation needs to invoke multiple calls for path resolving, including > handling reserved path, breaking the path into components, identifying > corresponding inodes etc. This makes the path resolving process hard to > follow. > # The logic in INodesInPath is complicated and not direct. > Ideally for each operation we should only resolve the path once in the > beginning, and always use the INodesInPath instance afterwards. This requires > a lot of code cleanup. We will use this jira as the umbrella jira to track > all the efforts. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7120) When aborting NameNode or JournalNode due to metadata file problems, write the contents of the metadata directories and permissions to logs.
[ https://issues.apache.org/jira/browse/HDFS-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7120: - Target Version/s: (was: 2.8.0) > When aborting NameNode or JournalNode due to metadata file problems, write > the contents of the metadata directories and permissions to logs. > > > Key: HDFS-7120 > URL: https://issues.apache.org/jira/browse/HDFS-7120 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: journal-node, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > If the NameNode or JournalNode aborts due to an unexpected error in the > metadata directories, often the root cause is that the metadata files are in > an unexpected state, or permissions are broken on the directories. This > issue proposes that during abort, we write additional information about the > directory state and permissions to the logs. This can help speed up > diagnosis, and ultimately recovery. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10295) Balancer configures DN properties directly on-demand
[ https://issues.apache.org/jira/browse/HDFS-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-10295: -- Target Version/s: (was: 2.8.0) > Balancer configures DN properties directly on-demand > > > Key: HDFS-10295 > URL: https://issues.apache.org/jira/browse/HDFS-10295 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: 2.7.0 >Reporter: John Zhuge >Assignee: John Zhuge > > This is a further enhancement to HDFS-10294. Instead of using NN APIs, use > new DN APIs to query and set necessary properties on the DNs involved. > Details: > * Before each balancing iteration, set the properties on all DNs involved in > the current iteration. > * Need new DN APIs to query and set the balancing properties. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7211) Block invalidation work should be ordered
[
https://issues.apache.org/jira/browse/HDFS-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-7211:
-
Target Version/s: (was: 2.8.0)
> Block invalidation work should be ordered
> -
>
> Key: HDFS-7211
> URL: https://issues.apache.org/jira/browse/HDFS-7211
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.5.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> {{InvalidateBlocks#node2blocks}} uses an unordered {{LightWeightHashSet}} to
> store blocks (to be invalidated) on the same DN. This causes poor ordering
> when a single DN has a large number of blocks to invalidate. Blocks should be
> invalidated following the order of invalidation commands -- at least roughly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7121) For JournalNode operations that must succeed on all nodes, execute a pre-check to verify that the operation can succeed.
[
https://issues.apache.org/jira/browse/HDFS-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-7121:
-
Target Version/s: (was: 2.8.0)
> For JournalNode operations that must succeed on all nodes, execute a
> pre-check to verify that the operation can succeed.
>
>
> Key: HDFS-7121
> URL: https://issues.apache.org/jira/browse/HDFS-7121
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: journal-node
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>
> Several JournalNode operations are not satisfied by a quorum. They must
> succeed on every JournalNode in the cluster. If the operation succeeds on
> some nodes, but fails on others, then this may leave the nodes in an
> inconsistent state and require operations to do manual recovery steps. For
> example, if {{doPreUpgrade}} succeeds on 2 nodes and fails on 1 node, then
> the operator will need to correct the problem on the failed node and also
> manually restore the previous.tmp directory to current on the 2 successful
> nodes before reattempting the upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5284) Flatten INode hierarchy
[ https://issues.apache.org/jira/browse/HDFS-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-5284: - Target Version/s: (was: 2.8.0) > Flatten INode hierarchy > --- > > Key: HDFS-5284 > URL: https://issues.apache.org/jira/browse/HDFS-5284 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Tsz Wo Nicholas Sze >Assignee: Jing Zhao > > Currently, we have a complicated inode hierarchy for representing different > states of a file or a directory. For example, when a file is being created, > it is represented by an INodeFileUnderConstruction. When a file is being > closed, the inode is replaced by an INodeFile. If it is reopened for append, > the inode is replaced again by an INodeFileUnderConstruction. This JIRA is > to flatten the inode hierarchy. We may also improve the performance by > eliminating the inode replacement in runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10405) Test failure on trunk: TestEditLog
[ https://issues.apache.org/jira/browse/HDFS-10405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-10405: -- Target Version/s: (was: 2.8.0) > Test failure on trunk: TestEditLog > -- > > Key: HDFS-10405 > URL: https://issues.apache.org/jira/browse/HDFS-10405 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou > > It's been noticed there are some test failures: TestEditLog and > TestReconstructStripedBlocks -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7268) Can't append with SYNC_BLOCK
[ https://issues.apache.org/jira/browse/HDFS-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7268: - Target Version/s: (was: 2.8.0) > Can't append with SYNC_BLOCK > > > Key: HDFS-7268 > URL: https://issues.apache.org/jira/browse/HDFS-7268 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.4.1 >Reporter: Bogdan Raducanu > > It seems to me that currently it's not possible to start appending to a file > with CreateFlag.SYNC_BLOCK behavior (i.e. hsync after each block). So, I > think, the only way to guarantee durability when appending is if the user > calls hsync at the end of each block. > FileSystem.append doesn't accept a CreateFlag argument. > FileSystem.create, on the other hand, ignores CreateFlag.APPEND afaics. I > think the plan in HDFS-744 was to use this method if durability is needed. > It seems it might work through FileContext but in the end DFSOutputStream > never sets shouldSyncBlock when appending anyway. > Or am I missing something? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7612) TestOfflineEditsViewer.testStored() uses incorrect default value for cacheDir
[
https://issues.apache.org/jira/browse/HDFS-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-7612:
-
Target Version/s: (was: 2.8.0)
> TestOfflineEditsViewer.testStored() uses incorrect default value for cacheDir
> -
>
> Key: HDFS-7612
> URL: https://issues.apache.org/jira/browse/HDFS-7612
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>
> {code}
> final String cacheDir = System.getProperty("test.cache.data",
> "build/test/cache");
> {code}
> results in
> {{FileNotFoundException: build/test/cache/editsStoredParsed.xml (No such file
> or directory)}}
> when {{test.cache.data}} is not set.
> I can see this failing while running in Eclipse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7262) WebHDFS should use proxy doAs support in DelegationTokenAuthenticationHandler for token-management calls
[ https://issues.apache.org/jira/browse/HDFS-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7262: - Target Version/s: (was: 2.8.0) > WebHDFS should use proxy doAs support in DelegationTokenAuthenticationHandler > for token-management calls > > > Key: HDFS-7262 > URL: https://issues.apache.org/jira/browse/HDFS-7262 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Vinod Kumar Vavilapalli > > HADOOP-11207 adds support for proxy users to perform delegation-token > management operations and YARN uses them. > HDFS didn't need them because the doAs functionality is explicitly > implemented in webHDFS (JspHelper.getUGI()) - this should be changed to use > the common code to avoid duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7388) Improve the log for HA failover
[ https://issues.apache.org/jira/browse/HDFS-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7388: - Target Version/s: (was: 2.8.0) > Improve the log for HA failover > --- > > Key: HDFS-7388 > URL: https://issues.apache.org/jira/browse/HDFS-7388 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Reporter: Jing Zhao >Assignee: Jing Zhao > > Currently to debug an issue during the NN HA failover (with automatic > failover setup), we usually need to check log from all the NN and ZKFC > daemons to figure out what triggered the failover. Possible reasons include > NN health issue, connection issues between zkfc and NN, connection issues > between zkfc and ZK, and manual op started by admin. It will be helpful to > improve the log in NN and ZKFC and add more information to make the debug > easier. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8491) DN shutdown race conditions with open xceivers
[ https://issues.apache.org/jira/browse/HDFS-8491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8491: - Target Version/s: (was: 2.8.0) > DN shutdown race conditions with open xceivers > -- > > Key: HDFS-8491 > URL: https://issues.apache.org/jira/browse/HDFS-8491 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 >Reporter: Daryn Sharp > > DN shutdowns at least for restarts have many race conditions. Shutdown is > very noisy with exceptions. The DN notifies writers of the restart, waits 1s > and then interrupts the xceiver threads but does not join. The ipc server is > stopped and then the bpos services are stopped. > Xceivers then encounter NPEs in closeBlock because the block no longer exists > in the volume map when transient storage is checked. Just before that, the > DN notifies the NN the block was received. This does not appear to always be > true, but rather that the thread was interrupted. They race with bpos > shutdown, and luckily appear to lose, to send the block received. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7695) Intermittent failures in TestOpenFilesWithSnapshot
[
https://issues.apache.org/jira/browse/HDFS-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-7695:
-
Target Version/s: (was: 2.8.0)
> Intermittent failures in TestOpenFilesWithSnapshot
> --
>
> Key: HDFS-7695
> URL: https://issues.apache.org/jira/browse/HDFS-7695
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>
> This is to investigate intermittent failures of
> {{TestOpenFilesWithSnapshot}}, which is timing out on the NameNode restart as
> it is unable to leave SafeMode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7342) Lease Recovery doesn't happen some times
[ https://issues.apache.org/jira/browse/HDFS-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7342: - Target Version/s: (was: 2.8.0) > Lease Recovery doesn't happen some times > > > Key: HDFS-7342 > URL: https://issues.apache.org/jira/browse/HDFS-7342 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Attachments: HDFS-7342.04.patch, HDFS-7342.1.patch, > HDFS-7342.2.patch, HDFS-7342.3.patch > > > In some cases, LeaseManager tries to recover a lease, but is not able to. > HDFS-4882 describes a possibility of that. We should fix this -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7902) Expose truncate API for libwebhdfs
[ https://issues.apache.org/jira/browse/HDFS-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7902: - Target Version/s: 2.9.0 (was: 2.8.0) > Expose truncate API for libwebhdfs > -- > > Key: HDFS-7902 > URL: https://issues.apache.org/jira/browse/HDFS-7902 > Project: Hadoop HDFS > Issue Type: Improvement > Components: native, webhdfs >Affects Versions: 2.7.0 >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-7902.001.patch, HDFS-7902.002.patch > > > As Colin suggested in HDFS-7838, we will add truncate support for libwebhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7317) Rename StoragePolicy
[ https://issues.apache.org/jira/browse/HDFS-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7317: - Target Version/s: (was: 2.8.0) > Rename StoragePolicy > > > Key: HDFS-7317 > URL: https://issues.apache.org/jira/browse/HDFS-7317 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Andrew Wang > > As discussed on HDFS-7285, StoragePolicy might not be the best name for what > StoragePolicy currently is, which is a hardcoded mapping to replica > StorageTypes with a fallback, and being able to specify a StorageType for > creation vs. replication). Ideally the "policy" is what determines the data > temperature in the first place, with the temperature then mapping to the > actual StorageTypes to use. > There were a number of suggestions presented, e.g. StorageTag, > StoragePolicyTag. Let's figure this out here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7318) Rename some of the policies in default StoragePolicySuite
[ https://issues.apache.org/jira/browse/HDFS-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7318: - Target Version/s: (was: 2.8.0) > Rename some of the policies in default StoragePolicySuite > - > > Key: HDFS-7318 > URL: https://issues.apache.org/jira/browse/HDFS-7318 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Andrew Wang > > Right now we have default policies named based on temperature, e.g. HOT, > COLD, but also the storage type like ONESSD, ALLSSD, MEMORY. This seems > inconsistent, let's consider renaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8250) Create HDFS bindings for java.nio.file.FileSystem
[ https://issues.apache.org/jira/browse/HDFS-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8250: - Target Version/s: (was: 2.8.0) > Create HDFS bindings for java.nio.file.FileSystem > - > > Key: HDFS-8250 > URL: https://issues.apache.org/jira/browse/HDFS-8250 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Affects Versions: 2.7.0 >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky > > It's a nice to have feature as it would allow developers to have a unified > programming model while dealing with various File Systems even though this > particular issue only addresses HDFS. > It has already been done in the unrelated project, so I just need to extract > the code and provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7702) Move metadata across namenode - Effort to a real distributed namenode
[ https://issues.apache.org/jira/browse/HDFS-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7702: - Target Version/s: 2.9.0 (was: 2.8.0) > Move metadata across namenode - Effort to a real distributed namenode > - > > Key: HDFS-7702 > URL: https://issues.apache.org/jira/browse/HDFS-7702 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ray Zhang >Assignee: Ray Zhang > Attachments: Namespace Moving Tool Design Proposal.pdf > > > Implement a tool can show in memory namespace tree structure with > weight(size) and a API can move metadata across different namenode. The > purpose is moving data efficiently and faster, without moving blocks on > datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10495) Block should be marked as missing if the all the replicas are on Decommissioned nodes.
[ https://issues.apache.org/jira/browse/HDFS-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-10495: -- Target Version/s: 2.9.0, 2.7.4, 2.6.6 (was: 2.8.0, 2.7.4, 2.6.6) > Block should be marked as missing if the all the replicas are on > Decommissioned nodes. > -- > > Key: HDFS-10495 > URL: https://issues.apache.org/jira/browse/HDFS-10495 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > > As discussed on HDFS-8872, we should mark a block as missing if all the > replicas on decommissioned nodes since we can take the decommissioned nodes > out of rotation anytime. > We have seen multiple cases where all the replicas land on decommissioned > nodes. > After HDFS-7933, it doesn't mark as missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6993) Add a metric for the number of misbehaving DFSClients that try to do no-checksum reads that extend too long
[ https://issues.apache.org/jira/browse/HDFS-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6993: - Target Version/s: (was: 2.8.0) > Add a metric for the number of misbehaving DFSClients that try to do > no-checksum reads that extend too long > --- > > Key: HDFS-6993 > URL: https://issues.apache.org/jira/browse/HDFS-6993 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Minor > > Add a metric on the DataNode for the number of misbehaving DFSClients that > try to do no-checksum reads that extend too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10215) distcp throws vague exception without home directory
[
https://issues.apache.org/jira/browse/HDFS-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-10215:
--
Target Version/s: (was: 2.8.0)
> distcp throws vague exception without home directory
>
>
> Key: HDFS-10215
> URL: https://issues.apache.org/jira/browse/HDFS-10215
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: distcp
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
>
> Got this error when running {{hadoop distcp /tmp/d1 /tmp/d2}}:
> {code}
> $ hadoop distcp /tmp/d1 /tmp/d2
> 16/03/25 08:24:41 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/tmp/d1],
> targetPath=/tmp/d2, targetPathExists=true, preserveRawXattrs=false,
> filtersFile='null'}
> 16/03/25 08:24:41 INFO client.RMProxy: Connecting to ResourceManager at
> jzhuge-balancer-1.vpc.cloudera.com/172.26.21.70:8032
> 16/03/25 08:24:42 ERROR tools.DistCp: Exception encountered
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=systest, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:281)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:262)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:242)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:169)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6590)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6572)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6524)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4292)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4265)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:867)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:322)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:603)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3084)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:3049)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:957)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:953)
> at
>
[jira] [Updated] (HDFS-7030) Add more unit tests for DatanodeStorageInfo
[ https://issues.apache.org/jira/browse/HDFS-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-7030: - Target Version/s: (was: 2.8.0) > Add more unit tests for DatanodeStorageInfo > --- > > Key: HDFS-7030 > URL: https://issues.apache.org/jira/browse/HDFS-7030 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jing Zhao >Priority: Minor > > Currently we already have unit tests covering DatanodeDescriptor. As pointed > out by [~ozawa] in HDFS-6943, we should add more unit tests for > DatanodeStorageInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5656) add some configuration keys to hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-5656:
-
Target Version/s: (was: 2.8.0)
> add some configuration keys to hdfs-default.xml
> ---
>
> Key: HDFS-5656
> URL: https://issues.apache.org/jira/browse/HDFS-5656
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Colin P. McCabe
>Priority: Minor
>
> Some configuration keys like {{dfs.client.read.shortcircuit}} are not present
> in {{hdfs-default.xml}} as they should be.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9017) UI shows wrong last contact for dead nodes
[ https://issues.apache.org/jira/browse/HDFS-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9017: - Target Version/s: (was: 2.8.0) > UI shows wrong last contact for dead nodes > -- > > Key: HDFS-9017 > URL: https://issues.apache.org/jira/browse/HDFS-9017 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Assignee: J.Andreina >Priority: Minor > > It's showing the last contact as the restart of the NN host (not process, > host). Presumably it's using monotonic time 0. Ideally last contact for > nodes that never connected would be "never" instead of the epoch or boot time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5355) Hsync Metrics error
[ https://issues.apache.org/jira/browse/HDFS-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-5355: - Target Version/s: (was: 2.8.0) > Hsync Metrics error > --- > > Key: HDFS-5355 > URL: https://issues.apache.org/jira/browse/HDFS-5355 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.2.0, 3.0.0-alpha1 >Reporter: haosdent >Priority: Minor > Attachments: HDFS-5355.patch > > > The authors before comment "Expect two syncs, one from the hsync, one on > close" in TestDataNodeMetrics.java. In fact there isn't call sync when close. > One hsync call would produce two syncs, one for checksum and the other for > block. I modify relate code and make it looks like the hflush metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
