[jira] [Commented] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774433#comment-16774433 ] Wei Yan commented on HDFS-13123: [~brahmareddy] sorry for the late reply. As I have moved to other projects, feel free to move and pick up this jira. Pls let me know if anything I can help here. > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13508) RBF: Normalize paths (automatically) when adding, removing or listing mount table entries
[ https://issues.apache.org/jira/browse/HDFS-13508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13508: --- Summary: RBF: Normalize paths (automatically) when adding, removing or listing mount table entries (was: Normalize paths (automatically) when adding, removing or listing mount table entries) > RBF: Normalize paths (automatically) when adding, removing or listing mount > table entries > - > > Key: HDFS-13508 > URL: https://issues.apache.org/jira/browse/HDFS-13508 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ekanth S >Assignee: Ekanth S >Priority: Minor > > me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -ls /home/move > Mount Table Entries: > Source Destinations Owner Group Mode > /home/move hdfs-oahu->/home/move me hadoop rwxr-xr-x > me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -ls /home/move/ > Mount Table Entries: > Source Destinations Owner Group Mode > me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -rm /home/move/ > Cannot remove mount point /home/move/ > me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -add /home/move/ > hdfs-oahu /home/move/ -readonly > Cannot add mount point /home/move/ > The slash '/' at the end should be normalized before calling the API from the > CLI. > Note: add command fails with a terminating '/' . when it is an existing entry > (it checks the not-normalized value with the normalized value in the > mount-table). Adding a new mount point with '/' at the end works because the > CLI normalizes the mount before calling the API. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[ https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454468#comment-16454468 ] Wei Yan edited comment on HDFS-13326 at 4/26/18 4:16 PM: - Sorry for late review, +1 [^HDFS-13326.002.patch]. Also created HDFS-13507 for removing the update functionality from the "add" cmd. [~gangli2384] feel free to upload your previous 003 patch there :) was (Author: ywskycn): Sorry for late review, +1 [^HDFS-13326.002.patch]. Also created HDFS-13507 for removing the update functionality from the "add" cmd. [~gangli2384] feel free to upload your previous 003 patch there :) > RBF: Improve the interfaces to modify and view mount tables > --- > > Key: HDFS-13326 > URL: https://issues.apache.org/jira/browse/HDFS-13326 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch, > HDFS-13326.002.patch > > > From DFSRouterAdmin cmd, currently the update logic is implemented inside add > operation, where it has some limitation (e.g. cannot update "readonly" or > removing a destination). Given the RPC alreadys separate add and update > operations, it would be better to do the same in cmd level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[ https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454468#comment-16454468 ] Wei Yan commented on HDFS-13326: Sorry for late review, +1 [^HDFS-13326.002.patch]. Also created HDFS-13507 for removing the update functionality from the "add" cmd. [~gangli2384] feel free to upload your previous 003 patch there :) > RBF: Improve the interfaces to modify and view mount tables > --- > > Key: HDFS-13326 > URL: https://issues.apache.org/jira/browse/HDFS-13326 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch, > HDFS-13326.002.patch > > > From DFSRouterAdmin cmd, currently the update logic is implemented inside add > operation, where it has some limitation (e.g. cannot update "readonly" or > removing a destination). Given the RPC alreadys separate add and update > operations, it would be better to do the same in cmd level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
Wei Yan created HDFS-13507: -- Summary: RBF: Remove update functionality from routeradmin's add cmd Key: HDFS-13507 URL: https://issues.apache.org/jira/browse/HDFS-13507 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Wei Yan Assignee: Gang Li Follow up the discussion in HDFS-13326. We should remove the "update" functionality from routeradmin's add cmd, to make it consistent with RPC calls. Note that: this is an incompatible change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13475) RBF: Admin cannot enforce Router enter SafeMode
[
https://issues.apache.org/jira/browse/HDFS-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453468#comment-16453468
]
Wei Yan commented on HDFS-13475:
Thanks [~dibyendu_hadoop] . I think [~csun] is already working on it..
> RBF: Admin cannot enforce Router enter SafeMode
> ---
>
> Key: HDFS-13475
> URL: https://issues.apache.org/jira/browse/HDFS-13475
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Major
>
> To reproduce the issue:
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode enter
> Successfully enter safe mode.
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: true{code}
> And then,
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: false{code}
> From the code, it looks like the periodicInvoke triggers the leave.
> {code:java}
> public void periodicInvoke() {
> ..
> // Always update to indicate our cache was updated
> if (isCacheStale) {
> if (!rpcServer.isInSafeMode()) {
> enter();
> }
> } else if (rpcServer.isInSafeMode()) {
> // Cache recently updated, leave safe mode
> leave();
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13475) RBF: Admin cannot enforce Router enter SafeMode
[
https://issues.apache.org/jira/browse/HDFS-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan reassigned HDFS-13475:
--
Assignee: Chao Sun (was: Wei Yan)
> RBF: Admin cannot enforce Router enter SafeMode
> ---
>
> Key: HDFS-13475
> URL: https://issues.apache.org/jira/browse/HDFS-13475
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Major
>
> To reproduce the issue:
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode enter
> Successfully enter safe mode.
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: true{code}
> And then,
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: false{code}
> From the code, it looks like the periodicInvoke triggers the leave.
> {code:java}
> public void periodicInvoke() {
> ..
> // Always update to indicate our cache was updated
> if (isCacheStale) {
> if (!rpcServer.isInSafeMode()) {
> enter();
> }
> } else if (rpcServer.isInSafeMode()) {
> // Cache recently updated, leave safe mode
> leave();
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13475) RBF: Admin cannot enforce Router enter SafeMode
[
https://issues.apache.org/jira/browse/HDFS-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443251#comment-16443251
]
Wei Yan commented on HDFS-13475:
>From NameNode side, it has isInSafeMode() and isInStartupSafeMode(). Router
>can follow similar concept, that we have two different safeMode functions:
>isInSafeMode() and isInForcedSafeMode().
> RBF: Admin cannot enforce Router enter SafeMode
> ---
>
> Key: HDFS-13475
> URL: https://issues.apache.org/jira/browse/HDFS-13475
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Major
>
> To reproduce the issue:
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode enter
> Successfully enter safe mode.
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: true{code}
> And then,
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: false{code}
> From the code, it looks like the periodicInvoke triggers the leave.
> {code:java}
> public void periodicInvoke() {
> ..
> // Always update to indicate our cache was updated
> if (isCacheStale) {
> if (!rpcServer.isInSafeMode()) {
> enter();
> }
> } else if (rpcServer.isInSafeMode()) {
> // Cache recently updated, leave safe mode
> leave();
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13475) RBF: Admin cannot enforce Router enter SafeMode
[
https://issues.apache.org/jira/browse/HDFS-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13475:
---
Summary: RBF: Admin cannot enforce Router enter SafeMode (was: RBF: Router
always leaves SafeMode after some time even manually entering SafeMode)
> RBF: Admin cannot enforce Router enter SafeMode
> ---
>
> Key: HDFS-13475
> URL: https://issues.apache.org/jira/browse/HDFS-13475
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Major
>
> To reproduce the issue:
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode enter
> Successfully enter safe mode.
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: true{code}
> And then,
> {code:java}
> $ bin/hdfs dfsrouteradmin -safemode get
> Safe Mode: false{code}
> From the code, it looks like the periodicInvoke triggers the leave.
> {code:java}
> public void periodicInvoke() {
> ..
> // Always update to indicate our cache was updated
> if (isCacheStale) {
> if (!rpcServer.isInSafeMode()) {
> enter();
> }
> } else if (rpcServer.isInSafeMode()) {
> // Cache recently updated, leave safe mode
> leave();
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13475) RBF: Router always leaves SafeMode after some time even manually entering SafeMode
Wei Yan created HDFS-13475:
--
Summary: RBF: Router always leaves SafeMode after some time even
manually entering SafeMode
Key: HDFS-13475
URL: https://issues.apache.org/jira/browse/HDFS-13475
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Wei Yan
Assignee: Wei Yan
To reproduce the issue:
{code:java}
$ bin/hdfs dfsrouteradmin -safemode enter
Successfully enter safe mode.
$ bin/hdfs dfsrouteradmin -safemode get
Safe Mode: true{code}
And then,
{code:java}
$ bin/hdfs dfsrouteradmin -safemode get
Safe Mode: false{code}
>From the code, it looks like the periodicInvoke triggers the leave.
{code:java}
public void periodicInvoke() {
..
// Always update to indicate our cache was updated
if (isCacheStale) {
if (!rpcServer.isInSafeMode()) {
enter();
}
} else if (rpcServer.isInSafeMode()) {
// Cache recently updated, leave safe mode
leave();
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13470) RBF: Add Browse the Filesystem button to the UI
[
https://issues.apache.org/jira/browse/HDFS-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442794#comment-16442794
]
Wei Yan edited comment on HDFS-13470 at 4/18/18 4:29 PM:
-
{quote}Generating the header from javascript
{quote}
I guess this may work here.. We can let explorer.js generate NN/Router
contents, including header, links, etc. Not sure any better idea here.
One minor in [^HDFS-13470.000.patch], the tab links need to be updated:
{code:java}
Overview
Subclusters
Routers
Datanodes
Mount table{code}
was (Author: ywskycn):
{quote}Generating the header from javascript
{quote}
I guess this may work here.. We can let explorer.js generate NN/Router
contents, including header, links, etc. Not sure any better idea here.
One minor in [^HDFS-13470.000.patch], the tab links need to be updated:
{code:java}
Overview
Subclusters
Routers
Datanodes
Mount table{code}
> RBF: Add Browse the Filesystem button to the UI
> ---
>
> Key: HDFS-13470
> URL: https://issues.apache.org/jira/browse/HDFS-13470
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13470.000.patch
>
>
> After HDFS-12512 added WebHDFS, we can add the support to browse the
> filesystem to the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13470) RBF: Add Browse the Filesystem button to the UI
[
https://issues.apache.org/jira/browse/HDFS-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442794#comment-16442794
]
Wei Yan commented on HDFS-13470:
{quote}Generating the header from javascript
{quote}
I guess this may work here.. We can let explorer.js generate NN/Router
contents, including header, links, etc. Not sure any better idea here.
One minor in [^HDFS-13470.000.patch], the tab links need to be updated:
{code:java}
Overview
Subclusters
Routers
Datanodes
Mount table{code}
> RBF: Add Browse the Filesystem button to the UI
> ---
>
> Key: HDFS-13470
> URL: https://issues.apache.org/jira/browse/HDFS-13470
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13470.000.patch
>
>
> After HDFS-12512 added WebHDFS, we can add the support to browse the
> filesystem to the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13466) RBF: Add more router-related information to the UI
[
https://issues.apache.org/jira/browse/HDFS-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13466:
---
Resolution: Fixed
Fix Version/s: 3.0.4
2.9.2
3.1.1
3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Thanks for the review [~elgoiri]. Committed to trunk, branch-3.1, branch-3.0,
branch-2 and branch-2.9.
> RBF: Add more router-related information to the UI
> --
>
> Key: HDFS-13466
> URL: https://issues.apache.org/jira/browse/HDFS-13466
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4
>
> Attachments: HDFS-13466.001.patch, pic.png
>
>
> Currently in NameNode UI, the Summary part also includes information:
> {noformat}
> Security is off.
> Safemode is off.
> files and directories, * blocks = total filesystem object(s).
> Heap Memory used GB of GB Heap Memory. Max Heap Memory is GB.
> Non Heap Memory used MB of MB Commited Non Heap Memory. Max Non
> Heap Memory is .
> {noformat}
> We could add similar information for router, for better visibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13466) RBF: Add more router-related information to the UI
[
https://issues.apache.org/jira/browse/HDFS-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441260#comment-16441260
]
Wei Yan commented on HDFS-13466:
Put [^HDFS-13466.001.patch], it will look like pic.png.
BTW, not sure whether there is a bug there not, but "bin/hdfs dfsrouteradmin
-safemode -enter" looks not working...looking into it, will fix in another Jira
if there is an issue there.
> RBF: Add more router-related information to the UI
> --
>
> Key: HDFS-13466
> URL: https://issues.apache.org/jira/browse/HDFS-13466
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: HDFS-13466.001.patch, pic.png
>
>
> Currently in NameNode UI, the Summary part also includes information:
> {noformat}
> Security is off.
> Safemode is off.
> files and directories, * blocks = total filesystem object(s).
> Heap Memory used GB of GB Heap Memory. Max Heap Memory is GB.
> Non Heap Memory used MB of MB Commited Non Heap Memory. Max Non
> Heap Memory is .
> {noformat}
> We could add similar information for router, for better visibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13466) RBF: Add more router-related information to the UI
[
https://issues.apache.org/jira/browse/HDFS-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13466:
---
Status: Patch Available (was: Open)
> RBF: Add more router-related information to the UI
> --
>
> Key: HDFS-13466
> URL: https://issues.apache.org/jira/browse/HDFS-13466
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: HDFS-13466.001.patch, pic.png
>
>
> Currently in NameNode UI, the Summary part also includes information:
> {noformat}
> Security is off.
> Safemode is off.
> files and directories, * blocks = total filesystem object(s).
> Heap Memory used GB of GB Heap Memory. Max Heap Memory is GB.
> Non Heap Memory used MB of MB Commited Non Heap Memory. Max Non
> Heap Memory is .
> {noformat}
> We could add similar information for router, for better visibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13466) RBF: Add more router-related information to the UI
[
https://issues.apache.org/jira/browse/HDFS-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13466:
---
Attachment: HDFS-13466.001.patch
> RBF: Add more router-related information to the UI
> --
>
> Key: HDFS-13466
> URL: https://issues.apache.org/jira/browse/HDFS-13466
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: HDFS-13466.001.patch, pic.png
>
>
> Currently in NameNode UI, the Summary part also includes information:
> {noformat}
> Security is off.
> Safemode is off.
> files and directories, * blocks = total filesystem object(s).
> Heap Memory used GB of GB Heap Memory. Max Heap Memory is GB.
> Non Heap Memory used MB of MB Commited Non Heap Memory. Max Non
> Heap Memory is .
> {noformat}
> We could add similar information for router, for better visibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13466) RBF: Add more router-related information to the UI
[
https://issues.apache.org/jira/browse/HDFS-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13466:
---
Attachment: pic.png
> RBF: Add more router-related information to the UI
> --
>
> Key: HDFS-13466
> URL: https://issues.apache.org/jira/browse/HDFS-13466
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: HDFS-13466.001.patch, pic.png
>
>
> Currently in NameNode UI, the Summary part also includes information:
> {noformat}
> Security is off.
> Safemode is off.
> files and directories, * blocks = total filesystem object(s).
> Heap Memory used GB of GB Heap Memory. Max Heap Memory is GB.
> Non Heap Memory used MB of MB Commited Non Heap Memory. Max Non
> Heap Memory is .
> {noformat}
> We could add similar information for router, for better visibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13466) RBF: Add more router-related information to the UI
Wei Yan created HDFS-13466:
--
Summary: RBF: Add more router-related information to the UI
Key: HDFS-13466
URL: https://issues.apache.org/jira/browse/HDFS-13466
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Wei Yan
Assignee: Wei Yan
Currently in NameNode UI, the Summary part also includes information:
{noformat}
Security is off.
Safemode is off.
files and directories, * blocks = total filesystem object(s).
Heap Memory used GB of GB Heap Memory. Max Heap Memory is GB.
Non Heap Memory used MB of MB Commited Non Heap Memory. Max Non Heap
Memory is .
{noformat}
We could add similar information for router, for better visibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[ https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441093#comment-16441093 ] Wei Yan commented on HDFS-12615: [~daryn] asked about how RBF supporting file access using InodeID in another discussion. I'm not sure I understand the question correctly, I guess sth related to HDFS-4489? Daryn, pls correct me if I'm wrong here :) [~elgoiri], do you have context about supporting this? > Router-based HDFS federation phase 2 > > > Key: HDFS-12615 > URL: https://issues.apache.org/jira/browse/HDFS-12615 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Labels: RBF > > This umbrella JIRA tracks set of improvements over the Router-based HDFS > federation (HDFS-10467). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441052#comment-16441052
]
Wei Yan commented on HDFS-13326:
{quote}But one thing I'm concerning that this will be a incompatible change.
That is say, some users used to use -add option for updating mount tables won't
make sense. And they should use -update instead. We would like to add a
incompatible tag for this JIRA and make this change only for branch-2 and trunk.
{quote}
Agree. Will do that when commiting.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> From DFSRouterAdmin cmd, currently the update logic is implemented inside add
> operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439714#comment-16439714
]
Wei Yan edited comment on HDFS-13326 at 4/16/18 4:58 PM:
-
Thanks [~gangli2384] for [^HDFS-13326.001.patch]. Please fix the checkstyle
errors, most are 80-character issues.
In *RouterAdmin.java*:
The error msg
{quote}"Cannot add destination at an existing mount point " + mount);
{quote}
can be updated to
{quote}"Cannot add destination to an existing mount point. Please use -update
cmd."
{quote}
For *TestRouterAdminCLI.java*: we can merge multiple test functions together.
Don't need to check field updates one by one :)
[~elgoiri] [~linyiqun] do u guys have any more comments? We are changing the
original "-add" cmd, just to make sure it doesn't break anything from you guy
side.
was (Author: ywskycn):
Thanks [~gangli2384] for [^HDFS-13326.001.patch]. Please fix the checkstyle
errors, most are 80-character issues.
In *RouterAdmin.java*:
The error msg
{quote}"Cannot add destination at an existing mount point " + mount);
{quote}
can be updated to
{quote}"Cannot add destination to an existing mount point. Please use -update
cmd."
{quote}
For *TestRouterAdminCLI.java*: we can merge multiple test functions together.
Don't need to check field updates one by one :)
[~elgoiri] [~linyiqun] do u guys have any more comments? We are changing the
original "-add" cmd, just to make sure it doesn't break anything from you guy
side.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> From DFSRouterAdmin cmd, currently the update logic is implemented inside add
> operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439714#comment-16439714
]
Wei Yan commented on HDFS-13326:
Thanks [~gangli2384] for [^HDFS-13326.001.patch]. Please fix the checkstyle
errors, most are 80-character issues.
In *RouterAdmin.java*:
The error msg
{quote}"Cannot add destination at an existing mount point " + mount);
{quote}
can be updated to
{quote}"Cannot add destination to an existing mount point. Please use -update
cmd."
{quote}
For *TestRouterAdminCLI.java*: we can merge multiple test functions together.
Don't need to check field updates one by one :)
[~elgoiri] [~linyiqun] do u guys have any more comments? We are changing the
original "-add" cmd, just to make sure it doesn't break anything from you guy
side.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> From DFSRouterAdmin cmd, currently the update logic is implemented inside add
> operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[
https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439621#comment-16439621
]
Wei Yan commented on HDFS-12615:
{quote}Looking forward to seeing the patch, :).
{quote}
Was too busy last week... Didn't have a chance to finish it... Will try to put
sth there this week..
> Router-based HDFS federation phase 2
>
>
> Key: HDFS-12615
> URL: https://issues.apache.org/jira/browse/HDFS-12615
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: RBF
>
> This umbrella JIRA tracks set of improvements over the Router-based HDFS
> federation (HDFS-10467).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13435) RBF: Fix wrong error loggings
[
https://issues.apache.org/jira/browse/HDFS-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16435849#comment-16435849
]
Wei Yan commented on HDFS-13435:
Thanks [~linyiqun] for the patch... finally we have started to fix typoes :D
Mind to also fix these two in RouterRpcClient.java?
Line 745 which misses one {}.
{code:java}
LOG.error("Unexpected exception {} proxying {} to {}",
e.getClass(), m.getName(), ns, e);{code}
Line 1070 where method–> method.getMethodName()
{code:java}
String msg =
"Invocation to \"" + loc + "\" for \"" + method + "\" timed out";
LOG.error(msg);{code}
> RBF: Fix wrong error loggings
> -
>
> Key: HDFS-13435
> URL: https://issues.apache.org/jira/browse/HDFS-13435
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13435.001.patch
>
>
> There are many places that using {{Logger.error(String format, Object...
> arguments)}} incorrectly.
> A example:
> {code:java}
> LOG.error("Cannot remove {}", path, e);
> {code}
> The exception passed here is no meaning and won't be printed. Actually it
> should be update to
> {code:java}
> LOG.error("Cannot remove {}: {}.", path, e.getMessage());
> {code}
> or
> {code:java}
> LOG.error("Cannot remove " + path, e));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13045) RBF: Improve error message returned from subcluster
[ https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13045: --- Resolution: Fixed Fix Version/s: 3.0.4 2.9.2 3.1.1 3.2.0 2.10.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-3.1, branch-3.0, branch-2, and branch-2.9. Thanks for the contributions [~elgoiri]. > RBF: Improve error message returned from subcluster > --- > > Key: HDFS-13045 > URL: https://issues.apache.org/jira/browse/HDFS-13045 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Íñigo Goiri >Priority: Minor > Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4 > > Attachments: HDFS-13045.000.patch, HDFS-13045.001.patch, > HDFS-13045.002.patch, HDFS-13045.003.patch, HDFS-13045.004.patch > > > Currently, Router directly returns exception response from subcluster to > client, which may not have the correct error message, especially when the > error message containing a path. > One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If > user1 does a chown operation on "/a/b", and he doesn't have corresponding > privilege, currently the error msg looks like "Permission denied. user=user1 > is not the owner of inode=/c/d", which may confuse user. Would be better to > reverse the path back to original mount path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13045) RBF: Improve error message returned from subcluster
[ https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434069#comment-16434069 ] Wei Yan commented on HDFS-13045: Yetus looks happy.. Committing. > RBF: Improve error message returned from subcluster > --- > > Key: HDFS-13045 > URL: https://issues.apache.org/jira/browse/HDFS-13045 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Íñigo Goiri >Priority: Minor > Attachments: HDFS-13045.000.patch, HDFS-13045.001.patch, > HDFS-13045.002.patch, HDFS-13045.003.patch, HDFS-13045.004.patch > > > Currently, Router directly returns exception response from subcluster to > client, which may not have the correct error message, especially when the > error message containing a path. > One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If > user1 does a chown operation on "/a/b", and he doesn't have corresponding > privilege, currently the error msg looks like "Permission denied. user=user1 > is not the owner of inode=/c/d", which may confuse user. Would be better to > reverse the path back to original mount path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[
https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434064#comment-16434064
]
Wei Yan commented on HDFS-12615:
{quote} * What's the plan of RBF Rebalancer? It looks a useful tool for
users.{quote}
Regarding rebalancer, it is tracked in HDFS-13123. Let me put the poc patch
this week, and clear the implementation details, and then we can split out some
sub-tasks there.
> Router-based HDFS federation phase 2
>
>
> Key: HDFS-12615
> URL: https://issues.apache.org/jira/browse/HDFS-12615
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: RBF
>
> This umbrella JIRA tracks set of improvements over the Router-based HDFS
> federation (HDFS-10467).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13045) RBF: Improve error message returned from subcluster
[ https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432781#comment-16432781 ] Wei Yan commented on HDFS-13045: Sure +1 on [^HDFS-13045.003.patch] > RBF: Improve error message returned from subcluster > --- > > Key: HDFS-13045 > URL: https://issues.apache.org/jira/browse/HDFS-13045 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Íñigo Goiri >Priority: Minor > Attachments: HDFS-13045.000.patch, HDFS-13045.001.patch, > HDFS-13045.002.patch, HDFS-13045.003.patch > > > Currently, Router directly returns exception response from subcluster to > client, which may not have the correct error message, especially when the > error message containing a path. > One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If > user1 does a chown operation on "/a/b", and he doesn't have corresponding > privilege, currently the error msg looks like "Permission denied. user=user1 > is not the owner of inode=/c/d", which may confuse user. Would be better to > reverse the path back to original mount path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13045) RBF: Improve error message returned from subcluster
[ https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan reassigned HDFS-13045: -- Assignee: Íñigo Goiri (was: Wei Yan) > RBF: Improve error message returned from subcluster > --- > > Key: HDFS-13045 > URL: https://issues.apache.org/jira/browse/HDFS-13045 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Íñigo Goiri >Priority: Minor > Attachments: HDFS-13045.000.patch, HDFS-13045.001.patch, > HDFS-13045.002.patch, HDFS-13045.003.patch > > > Currently, Router directly returns exception response from subcluster to > client, which may not have the correct error message, especially when the > error message containing a path. > One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If > user1 does a chown operation on "/a/b", and he doesn't have corresponding > privilege, currently the error msg looks like "Permission denied. user=user1 > is not the owner of inode=/c/d", which may confuse user. Would be better to > reverse the path back to original mount path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13045) RBF: Improve error message returned from subcluster
[ https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432506#comment-16432506 ] Wei Yan commented on HDFS-13045: Thanks [~elgoiri] for [^HDFS-13045.003.patch]. Another corner case would be, the dst appears more than once in the error message, while replaceFirst would only replace the first one. But here the implementation would be more tricky, and I don't have a case where the NN would generate such like msg. I'm ok with current approach, and we can come back if you find other types of unhandled errors in the future. > RBF: Improve error message returned from subcluster > --- > > Key: HDFS-13045 > URL: https://issues.apache.org/jira/browse/HDFS-13045 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: HDFS-13045.000.patch, HDFS-13045.001.patch, > HDFS-13045.002.patch, HDFS-13045.003.patch > > > Currently, Router directly returns exception response from subcluster to > client, which may not have the correct error message, especially when the > error message containing a path. > One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If > user1 does a chown operation on "/a/b", and he doesn't have corresponding > privilege, currently the error msg looks like "Permission denied. user=user1 > is not the owner of inode=/c/d", which may confuse user. Would be better to > reverse the path back to original mount path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429077#comment-16429077
]
Wei Yan commented on HDFS-13326:
[~gangli2384], some comments on [^HDFS-13326.000.patch].
(1) For add cmd, if mount entry exists, we can directly print out error message
and leave the "update" to the update cmd (following existing RPC
implementation).
(2) For update cmd, currently RPC side supports both add and update, so we can
just follow. To be simple, just parsing the input parameters (like what we did
in add cmd) and building a MountTable object, then sending to RPC end.
Some minors:
(1) Change the "namespace" to "nameservice"
(2) Set readonly to false if no input parameter
(3) Remove the line change in TestRouterAdmin.java.
(4) Also need to update
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
(5) Need to fix the checkstyle errors
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch
>
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13045) RBF: Improve error message returned from subcluster
[
https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428768#comment-16428768
]
Wei Yan commented on HDFS-13045:
I cannot produce the error pattern I mentioned above through some fs
operations. Not sure whether it is still valid. The following code is just a
quick test against this error pattern.
{code:java}
String msg = "Parent directory doesn't exist: /a/a/b";
String src = "/ns1/a";
String dst = "/a";
String newMsg = msg.replaceAll(dst, src);
int minLen = Math.min(dst.length(), src.length());
for (int i = 0; newMsg.equals(msg) && i < minLen; i++) {
// Check if we can replace sub folders
String dst1 = dst.substring(0, dst.length() - 1 - i);
String src1 = src.substring(0, src.length() - 1 - i);
newMsg = msg.replaceAll(dst1, src1);
}
System.out.println(newMsg);{code}
I think currently the patch cannot handle it, as it replaceAll at the first
place.
One more nit thing is, we may also need to set the stackTrace back when
generating the new exception.
> RBF: Improve error message returned from subcluster
> ---
>
> Key: HDFS-13045
> URL: https://issues.apache.org/jira/browse/HDFS-13045
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: HDFS-13045.000.patch, HDFS-13045.001.patch,
> HDFS-13045.002.patch
>
>
> Currently, Router directly returns exception response from subcluster to
> client, which may not have the correct error message, especially when the
> error message containing a path.
> One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If
> user1 does a chown operation on "/a/b", and he doesn't have corresponding
> privilege, currently the error msg looks like "Permission denied. user=user1
> is not the owner of inode=/c/d", which may confuse user. Would be better to
> reverse the path back to original mount path.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13237) [Documentation] RBF: Mount points across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427723#comment-16427723 ] Wei Yan commented on HDFS-13237: [~elgoiri] thanks for the fix. +1 on[^HDFS-13237.003.patch] > [Documentation] RBF: Mount points across multiple subclusters > - > > Key: HDFS-13237 > URL: https://issues.apache.org/jira/browse/HDFS-13237 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Attachments: HDFS-13237.000.patch, HDFS-13237.001.patch, > HDFS-13237.002.patch, HDFS-13237.003.patch > > > Document the feature to spread mount points across multiple subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13353) RBF: TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13353:
---
Resolution: Fixed
Fix Version/s: 3.0.4
2.9.2
3.1.1
3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-3.1, branch-3.0, branch-2, and branch-2.9. Thanks
for the contribution [~tasanuma0829], and [~elgoiri] for the review.
> RBF: TestRouterWebHDFSContractCreate failed
> ---
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4
>
> Attachments: HDFS-13353.1.patch, HDFS-13353.2.patch,
> HDFS-13353.3.patch
>
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:676)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1074)
> at
> org.apache.hadoop.hdfs.
[jira] [Commented] (HDFS-13237) [Documentation] RBF: Mount points across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427238#comment-16427238
]
Wei Yan commented on HDFS-13237:
Sorry for the late. Some minors.
{quote}For deciding where to create a new file it uses the order parameter.
It currently supports the following methods:
{quote}
The first '.' should be ','? Also would be better to "a new file" --> "a new
file/folder".
{quote}* RANDOM: Random subcluster. This is usually useful for reading
workloads and balance the load across. Folders are created in all subclusters.
{quote}
RANDOM can balance both READ and WRITE workload across subcusters, not just
reading workload, right?
{quote}the difference is that HASH would make all the files/folder within a
folder
{quote}
Here "files/folder" should be "files/folders".
{quote}][-order HASH|LOCAL|RANDOM|HASH_ALL|SPACE|]]
{quote}
There is an extra '|' at the end.
> [Documentation] RBF: Mount points across multiple subclusters
> -
>
> Key: HDFS-13237
> URL: https://issues.apache.org/jira/browse/HDFS-13237
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: HDFS-13237.000.patch, HDFS-13237.001.patch,
> HDFS-13237.002.patch
>
>
> Document the feature to spread mount points across multiple subclusters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427203#comment-16427203
]
Wei Yan commented on HDFS-13326:
Thanks [~gangli2384] for the patch. Will put comments soon.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch
>
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13353) RBF: TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427200#comment-16427200
]
Wei Yan commented on HDFS-13353:
+1 on [^HDFS-13353.3.patch]. Thanks [~tasanuma0829]
Will commit once Yetus returns.
> RBF: TestRouterWebHDFSContractCreate failed
> ---
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-13353.1.patch, HDFS-13353.2.patch,
> HDFS-13353.3.patch
>
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:676)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1074)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1085)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:930)
> ... 15 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundEx
[jira] [Commented] (HDFS-13353) RBF: TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427132#comment-16427132
]
Wei Yan commented on HDFS-13353:
Thanks [~tasanuma0829] for the patch [^HDFS-13353.2.patch]. It looks good to
me. Do you mind to fix the checkstyle issue? just move '{' back to previous
line.
For the TestRouterWebHDFSContractAppend, I don't have a clue, but feel free to
open another Jira if you want to continue investigating.
> RBF: TestRouterWebHDFSContractCreate failed
> ---
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-13353.1.patch, HDFS-13353.2.patch
>
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:676)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1074)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1085)
> at
[jira] [Commented] (HDFS-13045) RBF: Improve error message returned from subcluster
[
https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426165#comment-16426165
]
Wei Yan commented on HDFS-13045:
{quote}bq.I went with your first proposal there; I had to remember the original
path but that's fairly simple (didn't check super carefully if there was
another place with the original path though).
{quote}
Recording the orginal path is a very good idea. One tracky here is, if the
destination contains repeated destination paths, the replacement will replace
all destination paths. For example, mount point "/ns1/a" maps "/a" in ns1, and
the error message like "./a/a/b...", the replacement will replace both "/a".
BTW, could u explain more about the following code... get confused here.
{code:java}
while (newMsg.equals(msg) && i < dst.length() && i < src.length()) {
// Check if we can replace sub folders
char dstChar = dst.charAt(dst.length() - 1 - i);
char srcChar = src.charAt(src.length() - 1 - i);
if (dstChar == srcChar && dstChar != '/') {
i++;
} else {
String dst1 = dst.substring(0, dst.length() - 1 - i);
String src1 = src.substring(0, src.length() - 1 - i);
newMsg = msg.replaceAll(dst1, src1);
}
}{code}
> RBF: Improve error message returned from subcluster
> ---
>
> Key: HDFS-13045
> URL: https://issues.apache.org/jira/browse/HDFS-13045
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: HDFS-13045.000.patch, HDFS-13045.001.patch
>
>
> Currently, Router directly returns exception response from subcluster to
> client, which may not have the correct error message, especially when the
> error message containing a path.
> One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If
> user1 does a chown operation on "/a/b", and he doesn't have corresponding
> privilege, currently the error msg looks like "Permission denied. user=user1
> is not the owner of inode=/c/d", which may confuse user. Would be better to
> reverse the path back to original mount path.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13353) RBF: TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426012#comment-16426012
]
Wei Yan commented on HDFS-13353:
[~tasanuma0829] just for your reference, previously I added the following two
testcases to TestWebHdfsFileSystemContract, which also failed in NN's WebHDFS.
{code:java}
@Test
public void testCreatedFileIsImmediatelyVisible() throws Throwable {
Path path = path("testCreatedFileIsImmediatelyVisible");
try(FSDataOutputStream out = fs.create(path,
false,
4096,
(short) 1,
1024)) {
// Thread.sleep(5000);
assertTrue(fs.exists(path));
}
}
@Test
public void testCreatedFileIsVisibleOnFlush() throws Throwable {
Path path = path("testCreatedFileIsVisibleOnFlush");
try(FSDataOutputStream out = fs.create(path,
false,
4096,
(short) 1,
1024)) {
out.write('a');
out.flush();
// Thread.sleep(5000);
assertTrue(fs.exists(path));
}
}{code}
> RBF: TestRouterWebHDFSContractCreate failed
> ---
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-13353.1.patch
>
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at
[jira] [Commented] (HDFS-13045) RBF: Improve error message returned from subcluster
[
https://issues.apache.org/jira/browse/HDFS-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425989#comment-16425989
]
Wei Yan commented on HDFS-13045:
We need to change the exception message by replacing resolved path back to
mount point one. This part code is in RouterRpcClient.java, invokeSequential()
function:
{code:java}
} catch (IOException ioe) {
// Record it and move on
lastThrownException = (IOException) ioe;
if (firstThrownException == null) {
firstThrownException = lastThrownException;
}
{code}
where we have the target nameservice information.
One option here is to re-translate (ns, target location) pair back to mount
path one, and then reset the error message. It looks little trick. Another
option is just adding destination NS to the error message, but that breaks the
compatibility.
[~elgoiri] any other thoughts?
> RBF: Improve error message returned from subcluster
> ---
>
> Key: HDFS-13045
> URL: https://issues.apache.org/jira/browse/HDFS-13045
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
> Attachments: HDFS-13045.000.patch
>
>
> Currently, Router directly returns exception response from subcluster to
> client, which may not have the correct error message, especially when the
> error message containing a path.
> One example, we have a mount path "/a/b" mapped to subclusterA's "/c/d". If
> user1 does a chown operation on "/a/b", and he doesn't have corresponding
> privilege, currently the error msg looks like "Permission denied. user=user1
> is not the owner of inode=/c/d", which may confuse user. Would be better to
> reverse the path back to original mount path.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13353) RBF: TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424173#comment-16424173
]
Wei Yan commented on HDFS-13353:
[~tasanuma0829] From I tried last time, I think
testCreatedFileIsImmediatelyVisible doesn't work with existing NN WebHDFS.
> RBF: TestRouterWebHDFSContractCreate failed
> ---
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-13353.1.patch
>
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:676)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1074)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1085)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:930)
> ... 15 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundExcept
[jira] [Commented] (HDFS-13353) RBF: TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417535#comment-16417535
]
Wei Yan commented on HDFS-13353:
Thanks for the patch [~tasanuma0829]. Given that webhdfs doesn't support
hflush/hsync, maybe we can disable this set of testcases to avoid flaky
failures, including both testCreatedFileIsImmediatelyVisible and
testCreatedFileIsVisibleOnFlush.
I tried sth yesterday, and I think currently NN WebHDFS
(TestWebHdfsFileSystemContract.java) also doesn't cover this kind of testcases.
> RBF: TestRouterWebHDFSContractCreate failed
> ---
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-13353.1.patch
>
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:676)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1074)
> at
> org.apache.hadoop.hdfs.web.WebHdfsF
[jira] [Commented] (HDFS-13353) RBF: TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415971#comment-16415971
]
Wei Yan commented on HDFS-13353:
actually it also failed in my local env (using a bad code base previously).
> RBF: TestRouterWebHDFSContractCreate failed
> ---
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:676)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1074)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1085)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:930)
> ... 15 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /test/testCreatedFileIsVisibleOnFlush
> at
> org.apache
[jira] [Commented] (HDFS-13353) TestRouterWebHDFSContractCreate failed
[
https://issues.apache.org/jira/browse/HDFS-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415927#comment-16415927
]
Wei Yan commented on HDFS-13353:
Just tried in my local env. Looks liek the test passed. Not sure why it failed
in the jenkins. looking into it.
> TestRouterWebHDFSContractCreate failed
> --
>
> Key: HDFS-13353
> URL: https://issues.apache.org/jira/browse/HDFS-13353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> {noformat}
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 21.685 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
> [ERROR]
> testCreatedFileIsVisibleOnFlush(org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate)
> Time elapsed: 0.147 s <<< ERROR!
> java.io.FileNotFoundException: expected path to be visible before file
> closed: not found
> webhdfs://0.0.0.0:43796/test/testCreatedFileIsVisibleOnFlush in
> webhdfs://0.0.0.0:43796/test
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:936)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:914)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsVisibleOnFlush(AbstractContractCreateTest.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /test/testCreatedFileIsVisibleOnFlush
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$800(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:642)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:676)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1074)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1085)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:930)
> ... 15 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /test/testCreatedFileIsVisibleOnFlu
[jira] [Commented] (HDFS-13291) RBF: Implement available space based OrderResolver
[ https://issues.apache.org/jira/browse/HDFS-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413371#comment-16413371 ] Wei Yan commented on HDFS-13291: +1 [^HDFS-13291.009.patch] > RBF: Implement available space based OrderResolver > -- > > Key: HDFS-13291 > URL: https://issues.apache.org/jira/browse/HDFS-13291 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-13291.001.patch, HDFS-13291.002.patch, > HDFS-13291.003.patch, HDFS-13291.004.patch, HDFS-13291.005.patch, > HDFS-13291.006.patch, HDFS-13291.007.patch, HDFS-13291.008.patch, > HDFS-13291.009.patch > > > Implement available space based OrderResolver, this type resolver will > benefit for balancing the data across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413350#comment-16413350 ] Wei Yan commented on HDFS-13204: LGTM, +1 > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Assignee: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, HDFS-13204.002.patch, > HDFS-13204.003.patch, HDFS-13204.004.patch, HDFS-13204.005.patch, > HDFS-13204.006.patch, HDFS-13204.007.patch, HDFS-13204.008.patch, > Routers.png, Subclusters.png, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png, > image-2018-03-23-18-06-54-354.png, image-2018-03-26-10-10-10-930.png, > image-2018-03-26-10-21-24-171.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12512:
---
Resolution: Fixed
Fix Version/s: 3.1.1
3.2.0
3.0.2
2.9.1
2.10.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-3.1, branch-3.0, branch-2, and branch-2.9. Thanks
[~elgoiri] for the review and the initial patch.
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Fix For: 2.10.0, 2.9.1, 3.0.2, 3.2.0, 3.1.1
>
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch, HDFS-12512.012.patch, HDFS-12512.013.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16411720#comment-16411720
]
Wei Yan commented on HDFS-11156:
[~andrew.wang] [~cheersyang] this patch should be reverted from branch-2,
right? Still found some code there
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156-branch-2.01.patch, HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch, HDFS-11156.04.patch, HDFS-11156.05.patch,
> HDFS-11156.06.patch, HDFS-11156.07.patch, HDFS-11156.08.patch,
> HDFS-11156.09.patch, HDFS-11156.10.patch, HDFS-11156.11.patch,
> HDFS-11156.12.patch, HDFS-11156.13.patch, HDFS-11156.14.patch,
> HDFS-11156.15.patch, HDFS-11156.16.patch, Output_JSON_format_v10.jpg,
> SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16411551#comment-16411551
]
Wei Yan commented on HDFS-12512:
The test failures are not related. Commiting...
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch, HDFS-12512.012.patch, HDFS-12512.013.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12512:
---
Attachment: HDFS-12512.013.patch
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch, HDFS-12512.012.patch, HDFS-12512.013.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410336#comment-16410336
]
Wei Yan commented on HDFS-12512:
Good catch.. uploaded a new patch.
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch, HDFS-12512.012.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12512:
---
Attachment: HDFS-12512.012.patch
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch, HDFS-12512.012.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12512:
---
Attachment: HDFS-12512.011.patch
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410313#comment-16410313
]
Wei Yan commented on HDFS-12512:
Rebase the patch. [^HDFS-12512.011.patch]
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410302#comment-16410302
]
Wei Yan commented on HDFS-12512:
Sure, let me do it now.
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410043#comment-16410043
]
Wei Yan commented on HDFS-13326:
thanks, [~elgoiri]
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410029#comment-16410029
]
Wei Yan commented on HDFS-13326:
[~elgoiri], could u help add [~gangli2384] as a contributor? we have one more
people help work on RBF :)
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Priority: Minor
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan reassigned HDFS-13326:
--
Assignee: (was: Wei Yan)
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Priority: Minor
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13326:
---
Description:
1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
add operation, where it has some limitation (e.g. cannot update "readonly" or
removing a destination). Given the RPC alreadys separate add and update
operations, it would be better to do the same in cmd level.
2. Currently in the MountTable tab, the "readonly" field always show empty, no
matter whether the mount entry is readonly or not. From the code perspective,
it tries to show:
{code:java}
{code}
The federationhealth.html will load hadoop.css, however the hadoop.css doesn't
have classes with a prefix "dfshealth-mount-read-only". This could be fixed in
HDFS-13204.
was:
So currently in the MountTable tab, the "readonly" field always show empty, no
matter whether the mount entry is readonly or not. From the code perspective,
it tries to show:
{code:java}
{code}
The federationhealth.html will load hadoop.css, however the hadoop.css doesn't
have classes with a prefix "dfshealth-mount-read-only".
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13326:
---
Summary: RBF: Improve the interfaces to modify and view mount tables (was:
RBF: router webUI's MountTable tab doesn't show "readonly" info)
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
>
> So currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13291) RBF: Implement available space based OrderResolver
[ https://issues.apache.org/jira/browse/HDFS-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410017#comment-16410017 ] Wei Yan commented on HDFS-13291: Thanks [~linyiqun] for the patch. Some minors: * Typo "sapce" in "available-sapce-resolver.update-period" and "available-sapce-resolver.balanced-space-preference-fraction". * We could add a warning msg if balancedPreference < 0.5 where users mostly like make a wrong config. > RBF: Implement available space based OrderResolver > -- > > Key: HDFS-13291 > URL: https://issues.apache.org/jira/browse/HDFS-13291 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-13291.001.patch, HDFS-13291.002.patch, > HDFS-13291.003.patch, HDFS-13291.004.patch, HDFS-13291.005.patch > > > Implement available space based OrderResolver, this type resolver will > benefit for balancing the data across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: router webUI's MountTable tab doesn't show "readonly" info
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409932#comment-16409932
]
Wei Yan commented on HDFS-13326:
{quote}The APIs already allow that so it should be a matter of tweaking the CLI.
{quote}
Agree. Let's add a "update" cmd there. BTW, it's ok to change context of this
Jira for the new task (adding "update"), right? Don't want to create another
Jira to overwhelm HDFS--12615.
> RBF: router webUI's MountTable tab doesn't show "readonly" info
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
>
> So currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12792) RBF: Test Router-based federation using HDFSContract
[
https://issues.apache.org/jira/browse/HDFS-12792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409927#comment-16409927
]
Wei Yan commented on HDFS-12792:
+1 for [^HDFS-12792.011.patch].
One minor, as we disable SAFEMODE, so we don't need to config the following
field, right?
{code:java}
conf.setTimeDuration(DFS_ROUTER_SAFEMODE_EXTENSION,100,
TimeUnit.MILLISECONDS);{code}
> RBF: Test Router-based federation using HDFSContract
>
>
> Key: HDFS-12792
> URL: https://issues.apache.org/jira/browse/HDFS-12792
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12615.000.patch, HDFS-12792.001.patch,
> HDFS-12792.002.patch, HDFS-12792.003.patch, HDFS-12792.004.patch,
> HDFS-12792.005.patch, HDFS-12792.006.patch, HDFS-12792.007.patch,
> HDFS-12792.008.patch, HDFS-12792.009.patch, HDFS-12792.010.patch,
> HDFS-12792.011.patch
>
>
> Router-based federation should support HDFSContract.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409849#comment-16409849
]
Wei Yan commented on HDFS-12512:
Finally Yetus runs rbf testcases. I add the hdfs.xml in [^HDFS-12512.010.patch].
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12512:
---
Attachment: HDFS-12512.010.patch
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: router webUI's MountTable tab doesn't show "readonly" info
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408857#comment-16408857
]
Wei Yan commented on HDFS-13326:
{quote}-add option updates also.
{quote}
yes, I didn't scroll down to the see detailed implementation...
But one interesting thing here, we cannot "switching a mount entry from
readonly to non-readonly", or "removing a target path from a mount entry".
For readonly, currently it only supports updating from falst to true.
{code:java}
if (readonly) {
existingEntry.setReadOnly(true);
}{code}
For target path, it will always tries to add new target locations.
{code:java}
for (String nsId : nss) {
if (!existingEntry.addDestination(nsId, dest)) {
System.err.println("Cannot add destination at " + nsId + " " + dest);
}
}{code}
> RBF: router webUI's MountTable tab doesn't show "readonly" info
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
>
> So currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: router webUI's MountTable tab doesn't show "readonly" info
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408781#comment-16408781
]
Wei Yan commented on HDFS-13326:
Sorry, I didn't type it clearly. I mean the "update" command in dfsrouteradmin.
{code:java}
Federation Admin Tools:
[-add[-readonly] [-order
HASH|LOCAL|RANDOM|HASH_ALL] -owner -group -mode ]
[-rm ]
[-ls ]{code}
Currently it only supports -add and -rm, and no direct way to "update"...
> RBF: router webUI's MountTable tab doesn't show "readonly" info
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
>
> So currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408779#comment-16408779 ] Wei Yan commented on HDFS-13204: [~liuhongtong], thanks for the patch. [^HDFS-13204.004.patch] LGTM, and I tried in my local env. One more item, we need to add two more css classes as mentioned in HDFS-13326. > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Assignee: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, HDFS-13204.002.patch, > HDFS-13204.003.patch, HDFS-13204.004.patch, Routers.png, Subclusters.png, > image-2018-02-28-18-33-09-972.png, image-2018-02-28-18-33-47-661.png, > image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: router webUI's MountTable tab doesn't show "readonly" info
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408754#comment-16408754
]
Wei Yan commented on HDFS-13326:
[~elgoiri], sure, let me take a pass HDFS-13204.
{quote}Regarding the information, in the WebUI I think is easy to see these
things but the cmd might be a little overwhelming.
{quote}
Agree. let's leave the cmd as it is now.
One more question, do u remember any story about "update mount entry" in the
RouterAdmin cmd? Didn't find it now, we forgot to add?
> RBF: router webUI's MountTable tab doesn't show "readonly" info
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
>
> So currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: router webUI's MountTable tab doesn't show "readonly" info
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408743#comment-16408743
]
Wei Yan commented on HDFS-13326:
Also, currently cmd "bin/hdfs dfsrouteradmin -ls" has less information than the
WebUI, like "readonly", "order", "date created"... Not sure whether we should
all more such information for better debuggability. Also, we don't want to
overwhelm the outputs of the cmd, for better visibility. Opinions, [~elgoiri]
[~linyiqun]?
> RBF: router webUI's MountTable tab doesn't show "readonly" info
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Minor
>
> So currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13326) RBF: router webUI's MountTable tab doesn't show "readonly" info
Wei Yan created HDFS-13326:
--
Summary: RBF: router webUI's MountTable tab doesn't show
"readonly" info
Key: HDFS-13326
URL: https://issues.apache.org/jira/browse/HDFS-13326
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Wei Yan
Assignee: Wei Yan
So currently in the MountTable tab, the "readonly" field always show empty, no
matter whether the mount entry is readonly or not. From the code perspective,
it tries to show:
{code:java}
{code}
The federationhealth.html will load hadoop.css, however the hadoop.css doesn't
have classes with a prefix "dfshealth-mount-read-only".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408468#comment-16408468
]
Wei Yan commented on HDFS-12512:
Rebase a new patch [^HDFS-12512.009.patch].
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12512:
---
Attachment: HDFS-12512.009.patch
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13318) RBF: Fix FindBugs in hadoop-hdfs-rbf
[ https://issues.apache.org/jira/browse/HDFS-13318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406714#comment-16406714 ] Wei Yan commented on HDFS-13318: [~ekanth], [~csun] you guys want a try? > RBF: Fix FindBugs in hadoop-hdfs-rbf > > > Key: HDFS-13318 > URL: https://issues.apache.org/jira/browse/HDFS-13318 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Minor > > hadoop-hdfs-rbf has 3 FindBug warnings: > * NamenodePriorityComparator should be serializable > * RemoteMethod.getTypes() may expose internal representation > * RemoteMethod may store mutable objects -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 3.0.2 2.9.1 2.10.0 3.1.0 Status: Resolved (was: Patch Available) Thanks [~elgoiri] [~linyiqun] for the review. Committed to trunk, branch-3.1, branch-3.0, branch-2, and branch-2.9. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.2, 3.2.0 > > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-2.9.003.patch, > HDFS-13215_branch-2.9.004.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.0.002.patch, HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405842#comment-16405842 ] Wei Yan commented on HDFS-13215: [~linyiqun] any other comments? I just pushed the trunk patch, and the integration looks ok. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-2.9.003.patch, > HDFS-13215_branch-2.9.004.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.0.002.patch, HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215_branch-2.9.004.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-2.9.003.patch, > HDFS-13215_branch-2.9.004.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.0.002.patch, HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405836#comment-16405836 ] Wei Yan commented on HDFS-13215: Thanks [~linyiqun]. There are some files generated by intellij when switching branch... let me remove them. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-2.9.003.patch, > HDFS-13215_branch-3.0.001.patch, HDFS-13215_branch-3.0.002.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405776#comment-16405776 ] Wei Yan commented on HDFS-13215: Thanks [~linyiqun]. Uploaded [^HDFS-13215_branch-2.9.003.patch] [^HDFS-13215_branch-3.0.002.patch] to remove quota-related fields. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-2.9.003.patch, > HDFS-13215_branch-3.0.001.patch, HDFS-13215_branch-3.0.002.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215_branch-3.0.002.patch HDFS-13215_branch-2.9.003.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-2.9.003.patch, > HDFS-13215_branch-3.0.001.patch, HDFS-13215_branch-3.0.002.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405493#comment-16405493 ] Wei Yan commented on HDFS-13215: Thanks so much for the reviews.. I'll wait until tonight to check if [~linyiqun] has more comments before commiting. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405490#comment-16405490 ] Wei Yan commented on HDFS-13215: Uploaded [^HDFS-13215_branch-2.002.patch], [^HDFS-13215_branch-2.9.002.patch] to fix the dist issue for branch-2 > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215_branch-2.9.002.patch HDFS-13215_branch-2.002.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.002.patch, HDFS-13215_branch-2.9.001.patch, > HDFS-13215_branch-2.9.002.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405474#comment-16405474 ] Wei Yan commented on HDFS-13215: I mixed a change in hadoop-dist/pom.xml, where it has the list of "cp" cmds. Adding it > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.9.001.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405366#comment-16405366 ] Wei Yan commented on HDFS-13215: Uploaded 4 patches for the other 4 branches. Verified build and testcases locally. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.9.001.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215_branch-3.1.001.patch HDFS-13215_branch-3.0.001.patch HDFS-13215_branch-2.9.001.patch HDFS-13215_branch-2.001.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch, HDFS-13215_branch-2.001.patch, > HDFS-13215_branch-2.9.001.patch, HDFS-13215_branch-3.0.001.patch, > HDFS-13215_branch-3.1.001.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405146#comment-16405146 ] Wei Yan commented on HDFS-13215: Upload [^HDFS-13215.009.patch] for fixing the missing findbugs. Patches for other branches will be uploaded soon. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215.009.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch, HDFS-13215.009.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405086#comment-16405086 ] Wei Yan commented on HDFS-13215: Good catch, [~linyiqun]. Looks like findbugs didn't report any issue even we missed this. I'll add it back to hadoop-hdfs-rbf in the next patch. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404378#comment-16404378 ] Wei Yan commented on HDFS-13215: Thanks [~elgoiri] for the detailed review. Meanwhile, I'll prepare patches for other branches. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404338#comment-16404338 ] Wei Yan commented on HDFS-13215: Thanks, [~elgoiri]. Upload [^HDFS-13215.008.patch] to address the comments. All rbf site and .xml files moved to hadoop-hdfs-rbf module. Regarding the test failure, I think it is because of log4j configuration. By default, the log4j debug is disable, so in my env doesn't have the failure; but in your running, the debug mode is enabled, which triggers "FSEditLogAsync.java:128". I add the log4j.properties there, which may fix the issue. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215.008.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch, > HDFS-13215.008.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[
https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403272#comment-16403272
]
Wei Yan commented on HDFS-13215:
{quote}I think the error for {{dfs.namenode.fs-limits.min-block-size}} has to
do with the fact that {{hdfs-site.xml}} has that value set to 0 which allows
any value.
{quote}
I add a separate hdfs-site.xml to hadoop-hdfs-rbf, like what hadoop-hdfs-httfs
does.
Upload a new patch [^HDFS-13215.007.patch]. I move webapps to hadoop-hdfs-rbf,
and verified it in a dev env, looks ok.
> RBF: Move Router to its own module
> --
>
> Key: HDFS-13215
> URL: https://issues.apache.org/jira/browse/HDFS-13215
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch,
> HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch,
> HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch
>
>
> We are splitting the HDFS client code base and potentially Router-based
> Federation is also independent enough to be in its own package.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13215: --- Attachment: HDFS-13215.007.patch > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch, HDFS-13215.007.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403153#comment-16403153 ] Wei Yan commented on HDFS-13215: The three testcases passed in my local env. You miss some patches? I'll move the webapps out. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403141#comment-16403141 ] Wei Yan commented on HDFS-13215: Good catch, [~elgoiri]. There is another dir: hadoop-hdfs/src/main/webapps/router, not sure whether we should move it out or just leave it there. I'm thinking, in the future, if we plan to separate hdfs components just like yarn, we could leave it there and seperate to another module (like hadoop-yarn-ui) which includes all HDFS UI related. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403111#comment-16403111 ] Wei Yan commented on HDFS-13215: Strange Yetus didn't run any HDFS testcases.. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16402584#comment-16402584 ] Wei Yan commented on HDFS-13215: Thanks, [~elgoiri]. I run all RBF tests locally and they're fine. Not sure whether Yetus is ok. Working on branch-2 patch, close to go. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Wei Yan >Priority: Major > Attachments: HDFS-13215.000.patch, HDFS-13215.001.patch, > HDFS-13215.002.patch, HDFS-13215.003.patch, HDFS-13215.004.patch, > HDFS-13215.005.patch, HDFS-13215.006.patch > > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13303) branch-2 build is broken due to BlockManager
Wei Yan created HDFS-13303: -- Summary: branch-2 build is broken due to BlockManager Key: HDFS-13303 URL: https://issues.apache.org/jira/browse/HDFS-13303 Project: Hadoop HDFS Issue Type: Bug Reporter: Wei Yan I think this is because of HDFS-12886. cc [~elgoiri] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
