[jira] [Updated] (HDFS-14937) [SBN read] ObserverReadProxyProvider should throw InterruptException
[ https://issues.apache.org/jira/browse/HDFS-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14937: Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > [SBN read] ObserverReadProxyProvider should throw InterruptException > > > Key: HDFS-14937 > URL: https://issues.apache.org/jira/browse/HDFS-14937 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xuzq >Assignee: xuzq >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14937-trunk-001.patch, HDFS-14937-trunk-002.patch > > > ObserverReadProxyProvider should throw InterruptException immediately if one > Observer catch InterruptException in invoking. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14937) [SBN read] ObserverReadProxyProvider should throw InterruptException
[ https://issues.apache.org/jira/browse/HDFS-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004687#comment-17004687 ] Ayush Saxena commented on HDFS-14937: - Committed to trunk. Thanx [~xuzq_zander] for the contribution and [~vagarychen] for the review!!! > [SBN read] ObserverReadProxyProvider should throw InterruptException > > > Key: HDFS-14937 > URL: https://issues.apache.org/jira/browse/HDFS-14937 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xuzq >Assignee: xuzq >Priority: Major > Attachments: HDFS-14937-trunk-001.patch, HDFS-14937-trunk-002.patch > > > ObserverReadProxyProvider should throw InterruptException immediately if one > Observer catch InterruptException in invoking. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[ https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004685#comment-17004685 ] Ayush Saxena commented on HDFS-15051: - Is only {{WRITE}} permission required for the parent? Give a check what is the scenario with mkdirs(), what all permissions are checked, we should keep same as mkdir IMO. > RBF: Propose to revoke WRITE MountTableEntry privilege to super user only > - > > Key: HDFS-15051 > URL: https://issues.apache.org/jira/browse/HDFS-15051 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch, > HDFS-15051.003.patch, HDFS-15051.004.patch, HDFS-15051.005.patch, > HDFS-15051.006.patch > > > The current permission checker of #MountTableStoreImpl is not very restrict. > In some case, any user could add/update/remove MountTableEntry without the > expected permission checking. > The following code segment try to check permission when operate > MountTableEntry, however mountTable object is from Client/RouterAdmin > {{MountTable mountTable = request.getEntry();}}, and user could pass any mode > which could bypass the permission checker. > {code:java} > public void checkPermission(MountTable mountTable, FsAction access) > throws AccessControlException { > if (isSuperUser()) { > return; > } > FsPermission mode = mountTable.getMode(); > if (getUser().equals(mountTable.getOwnerName()) > && mode.getUserAction().implies(access)) { > return; > } > if (isMemberOfGroup(mountTable.getGroupName()) > && mode.getGroupAction().implies(access)) { > return; > } > if (!getUser().equals(mountTable.getOwnerName()) > && !isMemberOfGroup(mountTable.getGroupName()) > && mode.getOtherAction().implies(access)) { > return; > } > throw new AccessControlException( > "Permission denied while accessing mount table " > + mountTable.getSourcePath() > + ": user " + getUser() + " does not have " + access.toString() > + " permissions."); > } > {code} > I just propose revoke WRITE MountTableEntry privilege to super user only. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15082) RBF: Check each component length of destination path when add/update mount entry
[ https://issues.apache.org/jira/browse/HDFS-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004682#comment-17004682 ] Ayush Saxena commented on HDFS-15082: - is it similar to HDFS-13576? Regarding the patch : * Are we having a separate configuration at the Router to specify the path length independent of the namespace? If so, I don't think it makes sense. * The default should be 0, to disable this by default to ensure compatibility? * What is the use case for having this, or say advantage of having this? > RBF: Check each component length of destination path when add/update mount > entry > > > Key: HDFS-15082 > URL: https://issues.apache.org/jira/browse/HDFS-15082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15082.001.patch > > > When add/update mount entry, each component length of destination path could > exceed filesystem path component length limit, reference to > `dfs.namenode.fs-limits.max-component-length` of NameNode. So we should check > each component length of destination path when add/update mount entry at > Router side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15082) RBF: Check each component length of destination path when add/update mount entry
[ https://issues.apache.org/jira/browse/HDFS-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15082: --- Attachment: HDFS-15082.001.patch Status: Patch Available (was: Open) submit patch v001 and pending Jenkins. > RBF: Check each component length of destination path when add/update mount > entry > > > Key: HDFS-15082 > URL: https://issues.apache.org/jira/browse/HDFS-15082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15082.001.patch > > > When add/update mount entry, each component length of destination path could > exceed filesystem path component length limit, reference to > `dfs.namenode.fs-limits.max-component-length` of NameNode. So we should check > each component length of destination path when add/update mount entry at > Router side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15082) RBF: Check each component length of destination path when add/update mount entry
Xiaoqiao He created HDFS-15082: -- Summary: RBF: Check each component length of destination path when add/update mount entry Key: HDFS-15082 URL: https://issues.apache.org/jira/browse/HDFS-15082 Project: Hadoop HDFS Issue Type: Sub-task Components: rbf Reporter: Xiaoqiao He Assignee: Xiaoqiao He When add/update mount entry, each component length of destination path could exceed filesystem path component length limit, reference to `dfs.namenode.fs-limits.max-component-length` of NameNode. So we should check each component length of destination path when add/update mount entry at Router side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[ https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004668#comment-17004668 ] Xiaoqiao He commented on HDFS-15051: v006 rebase trunk and fix checkstyle. > RBF: Propose to revoke WRITE MountTableEntry privilege to super user only > - > > Key: HDFS-15051 > URL: https://issues.apache.org/jira/browse/HDFS-15051 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch, > HDFS-15051.003.patch, HDFS-15051.004.patch, HDFS-15051.005.patch, > HDFS-15051.006.patch > > > The current permission checker of #MountTableStoreImpl is not very restrict. > In some case, any user could add/update/remove MountTableEntry without the > expected permission checking. > The following code segment try to check permission when operate > MountTableEntry, however mountTable object is from Client/RouterAdmin > {{MountTable mountTable = request.getEntry();}}, and user could pass any mode > which could bypass the permission checker. > {code:java} > public void checkPermission(MountTable mountTable, FsAction access) > throws AccessControlException { > if (isSuperUser()) { > return; > } > FsPermission mode = mountTable.getMode(); > if (getUser().equals(mountTable.getOwnerName()) > && mode.getUserAction().implies(access)) { > return; > } > if (isMemberOfGroup(mountTable.getGroupName()) > && mode.getGroupAction().implies(access)) { > return; > } > if (!getUser().equals(mountTable.getOwnerName()) > && !isMemberOfGroup(mountTable.getGroupName()) > && mode.getOtherAction().implies(access)) { > return; > } > throw new AccessControlException( > "Permission denied while accessing mount table " > + mountTable.getSourcePath() > + ": user " + getUser() + " does not have " + access.toString() > + " permissions."); > } > {code} > I just propose revoke WRITE MountTableEntry privilege to super user only. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[ https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15051: --- Attachment: HDFS-15051.006.patch > RBF: Propose to revoke WRITE MountTableEntry privilege to super user only > - > > Key: HDFS-15051 > URL: https://issues.apache.org/jira/browse/HDFS-15051 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch, > HDFS-15051.003.patch, HDFS-15051.004.patch, HDFS-15051.005.patch, > HDFS-15051.006.patch > > > The current permission checker of #MountTableStoreImpl is not very restrict. > In some case, any user could add/update/remove MountTableEntry without the > expected permission checking. > The following code segment try to check permission when operate > MountTableEntry, however mountTable object is from Client/RouterAdmin > {{MountTable mountTable = request.getEntry();}}, and user could pass any mode > which could bypass the permission checker. > {code:java} > public void checkPermission(MountTable mountTable, FsAction access) > throws AccessControlException { > if (isSuperUser()) { > return; > } > FsPermission mode = mountTable.getMode(); > if (getUser().equals(mountTable.getOwnerName()) > && mode.getUserAction().implies(access)) { > return; > } > if (isMemberOfGroup(mountTable.getGroupName()) > && mode.getGroupAction().implies(access)) { > return; > } > if (!getUser().equals(mountTable.getOwnerName()) > && !isMemberOfGroup(mountTable.getGroupName()) > && mode.getOtherAction().implies(access)) { > return; > } > throw new AccessControlException( > "Permission denied while accessing mount table " > + mountTable.getSourcePath() > + ": user " + getUser() + " does not have " + access.toString() > + " permissions."); > } > {code} > I just propose revoke WRITE MountTableEntry privilege to super user only. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15075) Remove process command timing from BPServiceActor
[ https://issues.apache.org/jira/browse/HDFS-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004667#comment-17004667 ] Xiaoqiao He commented on HDFS-15075: update v003, fix failed unit tests and checkstyle. > Remove process command timing from BPServiceActor > - > > Key: HDFS-15075 > URL: https://issues.apache.org/jira/browse/HDFS-15075 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15075.001.patch, HDFS-15075.002.patch, > HDFS-15075.003.patch > > > HDFS-14997 moved the command processing into async. > Right now, we are checking the time to add to a queue. > We should remove this one and maybe move the timing within the thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15075) Remove process command timing from BPServiceActor
[ https://issues.apache.org/jira/browse/HDFS-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15075: --- Attachment: HDFS-15075.003.patch > Remove process command timing from BPServiceActor > - > > Key: HDFS-15075 > URL: https://issues.apache.org/jira/browse/HDFS-15075 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15075.001.patch, HDFS-15075.002.patch, > HDFS-15075.003.patch > > > HDFS-14997 moved the command processing into async. > Right now, we are checking the time to add to a queue. > We should remove this one and maybe move the timing within the thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14957) INodeReference Space Consumed was not same in QuotaUsage and ContentSummary
[ https://issues.apache.org/jira/browse/HDFS-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004659#comment-17004659 ] Surendra Singh Lilhore commented on HDFS-14957: --- [~hemanthboyina], sorry for late reply. You need to re-base your test class. > INodeReference Space Consumed was not same in QuotaUsage and ContentSummary > --- > > Key: HDFS-14957 > URL: https://issues.apache.org/jira/browse/HDFS-14957 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.1.4 >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14957.001.patch, HDFS-14957.002.patch, > HDFS-14957.JPG > > > for INodeReferences , space consumed was different in QuotaUsage and Content > Summary -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15074) DataNode.DataTransfer thread should catch all the expception and log it.
[ https://issues.apache.org/jira/browse/HDFS-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004656#comment-17004656 ] Hudson commented on HDFS-15074: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17800 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17800/]) HDFS-15074. DataNode.DataTransfer thread should catch all the expception (surendralilhore: rev ee51eadda01e02ac5759ca19756f6f961c8eb0cd) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java > DataNode.DataTransfer thread should catch all the expception and log it. > > > Key: HDFS-15074 > URL: https://issues.apache.org/jira/browse/HDFS-15074 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.1.1 >Reporter: Surendra Singh Lilhore >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-15074.001.patch, HDFS-15074.002.patch > > > Some time If this thread is throwing exception other than IOException, will > not be able to trash it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15074) DataNode.DataTransfer thread should catch all the expception and log it.
[ https://issues.apache.org/jira/browse/HDFS-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-15074: -- Fix Version/s: 3.2.2 3.1.4 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk, branch-3.2, branch-3.1 > DataNode.DataTransfer thread should catch all the expception and log it. > > > Key: HDFS-15074 > URL: https://issues.apache.org/jira/browse/HDFS-15074 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.1.1 >Reporter: Surendra Singh Lilhore >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-15074.001.patch, HDFS-15074.002.patch > > > Some time If this thread is throwing exception other than IOException, will > not be able to trash it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004640#comment-17004640 ] Takanobu Asanuma edited comment on HDFS-15063 at 12/29/19 5:12 AM: --- Thanks for your explanation and updating the patch, [~hemanthboyina]. After applying [^HDFS-15063.002.patch], users can execute GETFILESTATUS of WEBHDFS REST API against HttpFS server. But it still fails when users use {{WebHdfsFileSystem#getFileStatus()}} against HttpFS server. I confirmed it with adding the following codes to the bottom of testECPolicy(). {code:java} WebHdfsFileSystem httpfsWebHdfs = (WebHdfsFileSystem) FileSystem.get(new URI("webhdfs://" + TestJettyHelper.getJettyURL().toURI().getAuthority()), TestHdfsHelper.getHdfsConf()); HdfsFileStatus httpfsFileStatus = (HdfsFileStatus)httpfsWebHdfs.getFileStatus(ecFile); assertNotNull(httpfsFileStatus.getErasureCodingPolicy()); // This should succeed. {code} This is because the return value of {{FSOperations#toJsonInner}} doesn't have {{ecPolicyObj}} and {{JsonUtilClient#toFileStatus}} can't get the ecPolicy information. Could you check it? was (Author: tasanuma0829): Thanks for your explanation and updating the patch, [~hemanthboyina]. After applying [^HDFS-15063.002.patch], users can execute GETFILESTATUS of WEBHDFS REST AI against HttpFS server. But it still fails when users use {{WebHdfsFileSystem#getFileStatus()}} against HttpFS server. I confirmed it with adding the following codes to the bottom of testECPolicy(). {code:java} WebHdfsFileSystem httpfsWebHdfs = (WebHdfsFileSystem) FileSystem.get(new URI("webhdfs://" + TestJettyHelper.getJettyURL().toURI().getAuthority()), TestHdfsHelper.getHdfsConf()); HdfsFileStatus httpfsFileStatus = (HdfsFileStatus)httpfsWebHdfs.getFileStatus(ecFile); assertNotNull(httpfsFileStatus.getErasureCodingPolicy()); // This should succeed. {code} This is because the return value of {{FSOperations#toJsonInner}} doesn't have {{ecPolicyObj}} and {{JsonUtilClient#toFileStatus}} can't get the ecPolicy information. Could you check it? > HttpFS : getFileStatus doesn't return ecPolicy > -- > > Key: HDFS-15063 > URL: https://issues.apache.org/jira/browse/HDFS-15063 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15063.001.patch, HDFS-15063.002.patch > > > Currently LISTSTATUS call to HttpFS returns a json. These jsonArray elements > have the ecPolicy name. > But when HttpFsFileSystem converts it back into a FileStatus object, then > ecPolicy is not added -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15063) HttpFS : getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004640#comment-17004640 ] Takanobu Asanuma commented on HDFS-15063: - Thanks for your explanation and updating the patch, [~hemanthboyina]. After applying [^HDFS-15063.002.patch], users can execute GETFILESTATUS of WEBHDFS REST AI against HttpFS server. But it still fails when users use {{WebHdfsFileSystem#getFileStatus()}} against HttpFS server. I confirmed it with adding the following codes to the bottom of testECPolicy(). {code:java} WebHdfsFileSystem httpfsWebHdfs = (WebHdfsFileSystem) FileSystem.get(new URI("webhdfs://" + TestJettyHelper.getJettyURL().toURI().getAuthority()), TestHdfsHelper.getHdfsConf()); HdfsFileStatus httpfsFileStatus = (HdfsFileStatus)httpfsWebHdfs.getFileStatus(ecFile); assertNotNull(httpfsFileStatus.getErasureCodingPolicy()); // This should succeed. {code} This is because the return value of {{FSOperations#toJsonInner}} doesn't have {{ecPolicyObj}} and {{JsonUtilClient#toFileStatus}} can't get the ecPolicy information. Could you check it? > HttpFS : getFileStatus doesn't return ecPolicy > -- > > Key: HDFS-15063 > URL: https://issues.apache.org/jira/browse/HDFS-15063 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15063.001.patch, HDFS-15063.002.patch > > > Currently LISTSTATUS call to HttpFS returns a json. These jsonArray elements > have the ecPolicy name. > But when HttpFsFileSystem converts it back into a FileStatus object, then > ecPolicy is not added -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14546) Document block placement policies
[ https://issues.apache.org/jira/browse/HDFS-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004637#comment-17004637 ] Ayush Saxena commented on HDFS-14546: - There is no PR now, I remember the PR had some comments from [~weichiu], Are they addressed? [~weichiu] If you have time, can you give a check once. > Document block placement policies > - > > Key: HDFS-14546 > URL: https://issues.apache.org/jira/browse/HDFS-14546 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Amithsha >Priority: Major > Labels: documentation > Attachments: HDFS-14546-01.patch, HDFS-14546-02.patch, > HDFS-14546-03.patch, HDFS-14546-04.patch, HDFS-14546-05.patch, > HDFS-14546-06.patch, HDFS-14546-07.patch, HDFS-14546-08.patch, > HdfsDesign.patch > > > Currently, all the documentation refers to the default block placement policy. > However, over time there have been new policies: > * BlockPlacementPolicyRackFaultTolerant (HDFS-7891) > * BlockPlacementPolicyWithNodeGroup (HDFS-3601) > * BlockPlacementPolicyWithUpgradeDomain (HDFS-9006) > We should update the documentation to refer to them explaining their > particularities and probably how to setup each one of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[ https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004495#comment-17004495 ] Ayush Saxena commented on HDFS-14934: - Thanx [~tasanuma] for the review and commit > [SBN Read] Standby NN throws many InterruptedExceptions when > dfs.ha.tail-edits.period is 0 > -- > > Key: HDFS-14934 > URL: https://issues.apache.org/jira/browse/HDFS-14934 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Takanobu Asanuma >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1 > > Attachments: HDFS-14934-01.patch > > > When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many > warn logs in standby NN. > {noformat} > 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper > (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread > (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted: > java.lang.InterruptedException > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509) > at > com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82) > at > org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48) > at > org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[ https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14934: Fix Version/s: 2.10.1 3.2.2 3.1.4 3.3.0 Assignee: Ayush Saxena Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk, branch-3.2 and branch-3.1. And committed to branch-2.10 with fixing a minor conflict. Thanks for your contribution, [~ayushtkn]! > [SBN Read] Standby NN throws many InterruptedExceptions when > dfs.ha.tail-edits.period is 0 > -- > > Key: HDFS-14934 > URL: https://issues.apache.org/jira/browse/HDFS-14934 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Takanobu Asanuma >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1 > > Attachments: HDFS-14934-01.patch > > > When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many > warn logs in standby NN. > {noformat} > 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper > (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread > (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted: > java.lang.InterruptedException > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509) > at > com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82) > at > org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48) > at > org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[ https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004477#comment-17004477 ] Hudson commented on HDFS-14934: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17799 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17799/]) HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when (tasanuma: rev dc32f583afffc372f78fb45211c3e7ce13f6a4be) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java > [SBN Read] Standby NN throws many InterruptedExceptions when > dfs.ha.tail-edits.period is 0 > -- > > Key: HDFS-14934 > URL: https://issues.apache.org/jira/browse/HDFS-14934 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Takanobu Asanuma >Priority: Major > Attachments: HDFS-14934-01.patch > > > When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many > warn logs in standby NN. > {noformat} > 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper > (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread > (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted: > java.lang.InterruptedException > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509) > at > com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82) > at > org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48) > at > org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[ https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004470#comment-17004470 ] Takanobu Asanuma commented on HDFS-14934: - +1 on [^HDFS-14934-01.patch]. Will commit it soon. > [SBN Read] Standby NN throws many InterruptedExceptions when > dfs.ha.tail-edits.period is 0 > -- > > Key: HDFS-14934 > URL: https://issues.apache.org/jira/browse/HDFS-14934 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Takanobu Asanuma >Priority: Major > Attachments: HDFS-14934-01.patch > > > When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many > warn logs in standby NN. > {noformat} > 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper > (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread > (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted: > java.lang.InterruptedException > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509) > at > com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82) > at > org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48) > at > org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org