[jira] [Created] (HDFS-15245) Improve JournalNode web UI
Jianfei Jiang created HDFS-15245: Summary: Improve JournalNode web UI Key: HDFS-15245 URL: https://issues.apache.org/jira/browse/HDFS-15245 Project: Hadoop HDFS Issue Type: Improvement Components: journal-node, ui Affects Versions: 3.2.1 Reporter: Jianfei Jiang Assignee: Jianfei Jiang At present, journalnode web UI is a almost blank page. It almost has no useful infomation. # Add some infomation of journalnode at the main page. # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, Metrics, Configuration & Process Thread Dump -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15244) Make Decommission related configurations runtime refreshable
[
https://issues.apache.org/jira/browse/HDFS-15244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HDFS-15244.
-
Resolution: Duplicate
> Make Decommission related configurations runtime refreshable
>
>
> Key: HDFS-15244
> URL: https://issues.apache.org/jira/browse/HDFS-15244
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>
> There are certain configurations that can be tuned to increase the speed of
> Decommissioning, But restarting namenode for it seems to be a heavy operation.
> Propose to make these configurations, reconfigurable during runtime in order
> to speed-up decommissioning as per need.
> Proposed Configs :
> * {{dfs.namenode.replication.work.multiplier.per.iteration}}
> * {{dfs.namenode.replication.max-streams-hard-limit}}
> * {{dfs.namenode.replication.max-streams}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Attachment: jn web 2.PNG jn web 1.PNG HDFS-15245.001.patch > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.001.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Status: Patch Available (was: In Progress) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.001.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15245 started by Jianfei Jiang. > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.001.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Status: Open (was: Patch Available) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Attachment: (was: HDFS-15245.001.patch) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Attachment: HDFS-15245.002.patch > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Status: Patch Available (was: Open) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[
https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068361#comment-17068361
]
Ayush Saxena commented on HDFS-15243:
-
Thanx [~rain_lyy] for putting this up.
Protected directories is to prevent accidental delete of the whole directory,
not to prevent data deletions inside the directory. Please check the
description in core-default.xml :
{noformat}
A comma-separated list of directories which cannot
be deleted or renamed even by the superuser unless they are empty. This
setting can be used to guard important system directories
against accidental deletion due to administrator error.
{noformat}
> Child directory should not be deleted or renamed if parent directory is a
> protected directory
> -
>
> Key: HDFS-15243
> URL: https://issues.apache.org/jira/browse/HDFS-15243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: 3.1.1
>Affects Versions: 3.1.1
>Reporter: liuyanyu
>Priority: Major
>
> HDFS-8983 add fs.protected.directories to support protected directories on
> NameNode. But as I test, when set a parent directory（eg /testA) to
> protected directory, the child directory (eg /testA/testB) still can be
> deleted or renamed. When we protect a directory mainly for protecting the
> data under this directory , So I think the child directory should not be
> delete or renamed if the parent directory is a protected directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068375#comment-17068375 ] Wei-Chiu Chuang commented on HDFS-15245: Big +1 to the proposal! This is long overdue. I wish it also shows additional metrics like RpcDetailedActivity, but I am happy with the current one. > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068387#comment-17068387 ] liuyanyu commented on HDFS-15243: - Thanks [~ayushtkn] for reviewing this. In our production cluster, When we protect a directory we mainly to protect the data under this directory not just protect this directory . Because the import business data directories are hundreds or thousands in our cluster, we can not set so many directories to fs.protected.directories. So we want to mark a parent directory of several business directories to protected directory, then the child directories can not be delete or renamed. > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068388#comment-17068388
]
Ayush Saxena commented on HDFS-15245:
-
Tried this, works fine.
Thanx [~jiangjianfei] for taking this up.
Would be good to have more details like {{RpcDetailedActivity}} as [~weichiu]
said, but can be done separately too as follow ups too if you want.
+1 from my side too.
> Improve JournalNode web UI
> --
>
> Key: HDFS-15245
> URL: https://issues.apache.org/jira/browse/HDFS-15245
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: journal-node, ui
>Affects Versions: 3.2.1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15245.002.patch, jn web 1.PNG, jn web 2.PNG
>
>
> At present, journalnode web UI is a almost blank page. It almost has no
> useful infomation.
> # Add some infomation of journalnode at the main page.
> # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level,
> Metrics, Configuration & Process Thread Dump
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15238) RBF：NamenodeHeartbeatService caused memory to grow rapidly
[ https://issues.apache.org/jira/browse/HDFS-15238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068400#comment-17068400 ] Ayush Saxena commented on HDFS-15238: - Thanx [~xuzq_zander] for the patch. v002 LGTM +1 Will fix the typo while committing. > RBF：NamenodeHeartbeatService caused memory to grow rapidly > -- > > Key: HDFS-15238 > URL: https://issues.apache.org/jira/browse/HDFS-15238 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xuzq >Assignee: xuzq >Priority: Major > Attachments: HDFS-15238-002.patch, HDFS-15238-trunk-001.patch > > > NamenodeHeartbeatService will get NameNode's HA status every 5s, and created > HAServiceProtocol every time. > When creating HAServiceProtocol, it also will new Configuration. > Over time, there will be more and more entries for REGISTER in Configuration > until fullGc happen. > The entry will piles up again, after reaching a certain threshold, the > fullGc is triggered again. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068406#comment-17068406
]
Jianfei Jiang commented on HDFS-15245:
--
Thanks [~weichiu] and [~ayushtkn] for code review.
The RpcDetailedActivity statistics in jmx are some detail infomation. I wonder
whether it is proper to add to the main page. As a comparison, nn and dn seem
not have such detailed infomation at the home page. Hope you can give some
advice.
{
"name" : "Hadoop:service=JournalNode,name=RpcDetailedActivityForPort8485",
"modelerType" : "RpcDetailedActivityForPort8485",
"tag.port" : "8485",
"tag.Context" : "rpcdetailed",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"StartLogSegmentNumOps" : 67,
"StartLogSegmentAvgTime" : 6.6716417910447765,
"FinalizeLogSegmentNumOps" : 66,
"FinalizeLogSegmentAvgTime" : 1.5606060606060606,
"JournalNumOps" : 133,
"JournalAvgTime" : 0.15789473684210525,
"GetEditLogManifestNumOps" : 139,
"GetEditLogManifestAvgTime" : 3.1510791366906474,
"GetEditLogManifestFromJournalNumOps" : 68,
"GetEditLogManifestFromJournalAvgTime" : 0.8823529411764706,
"JournalOutOfSyncExceptionNumOps" : 1,
"JournalOutOfSyncExceptionAvgTime" : 13.0
}, {
"name" : "Hadoop:service=JournalNode,name=RpcActivityForPort8485",
"modelerType" : "RpcActivityForPort8485",
"tag.port" : "8485",
"tag.serverName" : "QJournalProtocolService",
"tag.Context" : "rpc",
"tag.NumOpenConnectionsPerUser" : "{}",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"ReceivedBytes" : 106302,
"SentBytes" : 76209,
"RpcQueueTimeNumOps" : 474,
"RpcQueueTimeAvgTime" : 0.0,
"RpcLockWaitTimeNumOps" : 474,
"RpcLockWaitTimeAvgTime" : 0.0,
"RpcProcessingTimeNumOps" : 474,
"RpcProcessingTimeAvgTime" : 2.282700421940928,
"DeferredRpcProcessingTimeNumOps" : 0,
"DeferredRpcProcessingTimeAvgTime" : 0.0,
"RpcAuthenticationFailures" : 0,
"RpcAuthenticationSuccesses" : 0,
"RpcAuthorizationFailures" : 0,
"RpcAuthorizationSuccesses" : 270,
"RpcClientBackoff" : 0,
"RpcSlowCalls" : 0,
"NumOpenConnections" : 0,
"CallQueueLength" : 0,
"NumDroppedConnections" : 0
}
> Improve JournalNode web UI
> --
>
> Key: HDFS-15245
> URL: https://issues.apache.org/jira/browse/HDFS-15245
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: journal-node, ui
>Affects Versions: 3.2.1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15245.002.patch, jn web 1.PNG, jn web 2.PNG
>
>
> At present, journalnode web UI is a almost blank page. It almost has no
> useful infomation.
> # Add some infomation of journalnode at the main page.
> # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level,
> Metrics, Configuration & Process Thread Dump
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068406#comment-17068406
]
Jianfei Jiang edited comment on HDFS-15245 at 3/27/20, 8:16 AM:
Thanks [~weichiu] and [~ayushtkn] for code review.
The RpcDetailedActivity statistics in jmx are some detail infomation. I wonder
whether it is proper to add to the main page. As a comparison, nn and dn seem
not have such detailed infomation at the home page. Hope you can give some
advice.
{
"name" : "Hadoop:service=JournalNode,name=RpcDetailedActivityForPort8485",
"modelerType" : "RpcDetailedActivityForPort8485",
"tag.port" : "8485",
"tag.Context" : "rpcdetailed",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"StartLogSegmentNumOps" : 67,
"StartLogSegmentAvgTime" : 6.6716417910447765,
"FinalizeLogSegmentNumOps" : 66,
"FinalizeLogSegmentAvgTime" : 1.5606060606060606,
"JournalNumOps" : 133,
"JournalAvgTime" : 0.15789473684210525,
"GetEditLogManifestNumOps" : 139,
"GetEditLogManifestAvgTime" : 3.1510791366906474,
"GetEditLogManifestFromJournalNumOps" : 68,
"GetEditLogManifestFromJournalAvgTime" : 0.8823529411764706,
"JournalOutOfSyncExceptionNumOps" : 1,
"JournalOutOfSyncExceptionAvgTime" : 13.0
}, {
"name" : "Hadoop:service=JournalNode,name=RpcActivityForPort8485",
"modelerType" : "RpcActivityForPort8485",
"tag.port" : "8485",
"tag.serverName" : "QJournalProtocolService",
"tag.Context" : "rpc",
"tag.NumOpenConnectionsPerUser" : "{}",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"ReceivedBytes" : 106302,
"SentBytes" : 76209,
"RpcQueueTimeNumOps" : 474,
"RpcQueueTimeAvgTime" : 0.0,
"RpcLockWaitTimeNumOps" : 474,
"RpcLockWaitTimeAvgTime" : 0.0,
"RpcProcessingTimeNumOps" : 474,
"RpcProcessingTimeAvgTime" : 2.282700421940928,
"DeferredRpcProcessingTimeNumOps" : 0,
"DeferredRpcProcessingTimeAvgTime" : 0.0,
"RpcAuthenticationFailures" : 0,
"RpcAuthenticationSuccesses" : 0,
"RpcAuthorizationFailures" : 0,
"RpcAuthorizationSuccesses" : 270,
"RpcClientBackoff" : 0,
"RpcSlowCalls" : 0,
"NumOpenConnections" : 0,
"CallQueueLength" : 0,
"NumDroppedConnections" : 0
}
was (Author: jiangjianfei):
Thanks [~weichiu] and [~ayushtkn] for code review.
The RpcDetailedActivity statistics in jmx are some detail infomation. I wonder
whether it is proper to add to the main page. As a comparison, nn and dn seem
not have such detailed infomation at the home page. Hope you can give some
advice.
{
"name" : "Hadoop:service=JournalNode,name=RpcDetailedActivityForPort8485",
"modelerType" : "RpcDetailedActivityForPort8485",
"tag.port" : "8485",
"tag.Context" : "rpcdetailed",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"StartLogSegmentNumOps" : 67,
"StartLogSegmentAvgTime" : 6.6716417910447765,
"FinalizeLogSegmentNumOps" : 66,
"FinalizeLogSegmentAvgTime" : 1.5606060606060606,
"JournalNumOps" : 133,
"JournalAvgTime" : 0.15789473684210525,
"GetEditLogManifestNumOps" : 139,
"GetEditLogManifestAvgTime" : 3.1510791366906474,
"GetEditLogManifestFromJournalNumOps" : 68,
"GetEditLogManifestFromJournalAvgTime" : 0.8823529411764706,
"JournalOutOfSyncExceptionNumOps" : 1,
"JournalOutOfSyncExceptionAvgTime" : 13.0
}, {
"name" : "Hadoop:service=JournalNode,name=RpcActivityForPort8485",
"modelerType" : "RpcActivityForPort8485",
"tag.port" : "8485",
"tag.serverName" : "QJournalProtocolService",
"tag.Context" : "rpc",
"tag.NumOpenConnectionsPerUser" : "{}",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"ReceivedBytes" : 106302,
"SentBytes" : 76209,
"RpcQueueTimeNumOps" : 474,
"RpcQueueTimeAvgTime" : 0.0,
"RpcLockWaitTimeNumOps" : 474,
"RpcLockWaitTimeAvgTime" : 0.0,
"RpcProcessingTimeNumOps" : 474,
"RpcProcessingTimeAvgTime" : 2.282700421940928,
"DeferredRpcProcessingTimeNumOps" : 0,
"DeferredRpcProcessingTimeAvgTime" : 0.0,
"RpcAuthenticationFailures" : 0,
"RpcAuthenticationSuccesses" : 0,
"RpcAuthorizationFailures" : 0,
"RpcAuthorizationSuccesses" : 270,
"RpcClientBackoff" : 0,
"RpcSlowCalls" : 0,
"NumOpenConnections" : 0,
"CallQueueLength" : 0,
"NumDroppedConnections" : 0
}
> Improve JournalNode web UI
> --
>
> Key: HDFS-15245
> URL: https://issues.apache.org/jira/browse/HDFS-15245
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: journal-node, ui
>Affects Versions: 3.2.1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15245.002.patch, jn web 1.PNG, jn web 2.PNG
>
>
> At present, journalnode web UI is a almost blank page. It almost has no
> useful infomation.
> # Add some infomation of journalnode at the main page.
> # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level,
> Metrics, Configuration & Process Thread Dump
>
--
This
[jira] [Comment Edited] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068406#comment-17068406
]
Jianfei Jiang edited comment on HDFS-15245 at 3/27/20, 8:17 AM:
Thanks [~weichiu] and [~ayushtkn] for code review.
The RpcDetailedActivity statistics in jmx are some detail infomation. I wonder
whether it is proper to add to the main page. As a comparison, nn and dn seem
not have such detailed infomation at the home page. Hope you can give some
advice.
{
"name" : "Hadoop:service=JournalNode,name=RpcDetailedActivityForPort8485",
"modelerType" : "RpcDetailedActivityForPort8485",
"tag.port" : "8485",
"tag.Context" : "rpcdetailed",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"StartLogSegmentNumOps" : 67,
"StartLogSegmentAvgTime" : 6.6716417910447765,
"FinalizeLogSegmentNumOps" : 66,
"FinalizeLogSegmentAvgTime" : 1.5606060606060606,
"JournalNumOps" : 133,
"JournalAvgTime" : 0.15789473684210525,
"GetEditLogManifestNumOps" : 139,
"GetEditLogManifestAvgTime" : 3.1510791366906474,
"GetEditLogManifestFromJournalNumOps" : 68,
"GetEditLogManifestFromJournalAvgTime" : 0.8823529411764706,
"JournalOutOfSyncExceptionNumOps" : 1,
"JournalOutOfSyncExceptionAvgTime" : 13.0
}
{
"name" : "Hadoop:service=JournalNode,name=RpcActivityForPort8485",
"modelerType" : "RpcActivityForPort8485",
"tag.port" : "8485",
"tag.serverName" : "QJournalProtocolService",
"tag.Context" : "rpc",
"tag.NumOpenConnectionsPerUser" : "{}",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"ReceivedBytes" : 106302,
"SentBytes" : 76209,
"RpcQueueTimeNumOps" : 474,
"RpcQueueTimeAvgTime" : 0.0,
"RpcLockWaitTimeNumOps" : 474,
"RpcLockWaitTimeAvgTime" : 0.0,
"RpcProcessingTimeNumOps" : 474,
"RpcProcessingTimeAvgTime" : 2.282700421940928,
"DeferredRpcProcessingTimeNumOps" : 0,
"DeferredRpcProcessingTimeAvgTime" : 0.0,
"RpcAuthenticationFailures" : 0,
"RpcAuthenticationSuccesses" : 0,
"RpcAuthorizationFailures" : 0,
"RpcAuthorizationSuccesses" : 270,
"RpcClientBackoff" : 0,
"RpcSlowCalls" : 0,
"NumOpenConnections" : 0,
"CallQueueLength" : 0,
"NumDroppedConnections" : 0
}
was (Author: jiangjianfei):
Thanks [~weichiu] and [~ayushtkn] for code review.
The RpcDetailedActivity statistics in jmx are some detail infomation. I wonder
whether it is proper to add to the main page. As a comparison, nn and dn seem
not have such detailed infomation at the home page. Hope you can give some
advice.
{
"name" : "Hadoop:service=JournalNode,name=RpcDetailedActivityForPort8485",
"modelerType" : "RpcDetailedActivityForPort8485",
"tag.port" : "8485",
"tag.Context" : "rpcdetailed",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"StartLogSegmentNumOps" : 67,
"StartLogSegmentAvgTime" : 6.6716417910447765,
"FinalizeLogSegmentNumOps" : 66,
"FinalizeLogSegmentAvgTime" : 1.5606060606060606,
"JournalNumOps" : 133,
"JournalAvgTime" : 0.15789473684210525,
"GetEditLogManifestNumOps" : 139,
"GetEditLogManifestAvgTime" : 3.1510791366906474,
"GetEditLogManifestFromJournalNumOps" : 68,
"GetEditLogManifestFromJournalAvgTime" : 0.8823529411764706,
"JournalOutOfSyncExceptionNumOps" : 1,
"JournalOutOfSyncExceptionAvgTime" : 13.0
}, {
"name" : "Hadoop:service=JournalNode,name=RpcActivityForPort8485",
"modelerType" : "RpcActivityForPort8485",
"tag.port" : "8485",
"tag.serverName" : "QJournalProtocolService",
"tag.Context" : "rpc",
"tag.NumOpenConnectionsPerUser" : "{}",
"tag.Hostname" : "zdh-deployer-jiangjianfei01",
"ReceivedBytes" : 106302,
"SentBytes" : 76209,
"RpcQueueTimeNumOps" : 474,
"RpcQueueTimeAvgTime" : 0.0,
"RpcLockWaitTimeNumOps" : 474,
"RpcLockWaitTimeAvgTime" : 0.0,
"RpcProcessingTimeNumOps" : 474,
"RpcProcessingTimeAvgTime" : 2.282700421940928,
"DeferredRpcProcessingTimeNumOps" : 0,
"DeferredRpcProcessingTimeAvgTime" : 0.0,
"RpcAuthenticationFailures" : 0,
"RpcAuthenticationSuccesses" : 0,
"RpcAuthorizationFailures" : 0,
"RpcAuthorizationSuccesses" : 270,
"RpcClientBackoff" : 0,
"RpcSlowCalls" : 0,
"NumOpenConnections" : 0,
"CallQueueLength" : 0,
"NumDroppedConnections" : 0
}
> Improve JournalNode web UI
> --
>
> Key: HDFS-15245
> URL: https://issues.apache.org/jira/browse/HDFS-15245
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: journal-node, ui
>Affects Versions: 3.2.1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15245.002.patch, jn web 1.PNG, jn web 2.PNG
>
>
> At present, journalnode web UI is a almost blank page. It almost has no
> useful infomation.
> # Add some infomation of journalnode at the main page.
> # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level,
> Metrics, Configuration & Process Thread Dump
>
--
Th
[jira] [Updated] (HDFS-15238) RBF：NamenodeHeartbeatService caused memory to grow rapidly
[ https://issues.apache.org/jira/browse/HDFS-15238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15238: Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > RBF：NamenodeHeartbeatService caused memory to grow rapidly > -- > > Key: HDFS-15238 > URL: https://issues.apache.org/jira/browse/HDFS-15238 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xuzq >Assignee: xuzq >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15238-002.patch, HDFS-15238-trunk-001.patch > > > NamenodeHeartbeatService will get NameNode's HA status every 5s, and created > HAServiceProtocol every time. > When creating HAServiceProtocol, it also will new Configuration. > Over time, there will be more and more entries for REGISTER in Configuration > until fullGc happen. > The entry will piles up again, after reaching a certain threshold, the > fullGc is triggered again. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15238) RBF：NamenodeHeartbeatService caused memory to grow rapidly
[ https://issues.apache.org/jira/browse/HDFS-15238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15238: Attachment: HDFS-15238-003.patch > RBF：NamenodeHeartbeatService caused memory to grow rapidly > -- > > Key: HDFS-15238 > URL: https://issues.apache.org/jira/browse/HDFS-15238 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xuzq >Assignee: xuzq >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15238-002.patch, HDFS-15238-003.patch, > HDFS-15238-trunk-001.patch > > > NamenodeHeartbeatService will get NameNode's HA status every 5s, and created > HAServiceProtocol every time. > When creating HAServiceProtocol, it also will new Configuration. > Over time, there will be more and more entries for REGISTER in Configuration > until fullGc happen. > The entry will piles up again, after reaching a certain threshold, the > fullGc is triggered again. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15238) RBF：NamenodeHeartbeatService caused memory to grow rapidly
[ https://issues.apache.org/jira/browse/HDFS-15238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068419#comment-17068419 ] Ayush Saxena commented on HDFS-15238: - Committed to trunk. Thanx [~xuzq_zander] for the contribution, [~elgoiri] and [~hexiaoqiao] for the reviews!!! Attached the committed to patch with fixed typo. > RBF：NamenodeHeartbeatService caused memory to grow rapidly > -- > > Key: HDFS-15238 > URL: https://issues.apache.org/jira/browse/HDFS-15238 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xuzq >Assignee: xuzq >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15238-002.patch, HDFS-15238-003.patch, > HDFS-15238-trunk-001.patch > > > NamenodeHeartbeatService will get NameNode's HA status every 5s, and created > HAServiceProtocol every time. > When creating HAServiceProtocol, it also will new Configuration. > Over time, there will be more and more entries for REGISTER in Configuration > until fullGc happen. > The entry will piles up again, after reaching a certain threshold, the > fullGc is triggered again. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15238) RBF：NamenodeHeartbeatService caused memory to grow rapidly
[ https://issues.apache.org/jira/browse/HDFS-15238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068427#comment-17068427 ] Hudson commented on HDFS-15238: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18098 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18098/]) HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. (ayushsaxena: rev cbe71eaddf21c0ac5967e67cdd4623bd770aa9e5) * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java > RBF：NamenodeHeartbeatService caused memory to grow rapidly > -- > > Key: HDFS-15238 > URL: https://issues.apache.org/jira/browse/HDFS-15238 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xuzq >Assignee: xuzq >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-15238-002.patch, HDFS-15238-003.patch, > HDFS-15238-trunk-001.patch > > > NamenodeHeartbeatService will get NameNode's HA status every 5s, and created > HAServiceProtocol every time. > When creating HAServiceProtocol, it also will new Configuration. > Over time, there will be more and more entries for REGISTER in Configuration > until fullGc happen. > The entry will piles up again, after reaching a certain threshold, the > fullGc is triggered again. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068513#comment-17068513 ] Ayush Saxena commented on HDFS-15243: - You tend to allow read/write but just prevent delete? prevent deletion of sub-directories or even files under the protected directories? Can you elaborate a little more. In any case, we can't change the existing behavior, may be we can introduce a new configuration, to make the protected-Directories recursive or something like that. Let me know if you would like to work on this, Will assign the Jira to you.. > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068528#comment-17068528
]
Hadoop QA commented on HDFS-15245:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 10 unchanged - 1 fixed = 17 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 38s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
52s{color} | {color:red} The patch generated 13 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.qjournal.server.TestJournalNodeMXBean |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.tools.TestStoragePolicyCommands |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.qjournal.client.TestQJMWithFaults |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-15245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12997967/HDFS-15245.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ba5bb0ffed8f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / eaaaba1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/29030/artifact/out/diff-checkstyle-hadoop-
[jira] [Commented] (HDFS-15244) Make Decommission related configurations runtime refreshable
[
https://issues.apache.org/jira/browse/HDFS-15244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068561#comment-17068561
]
Brahma Reddy Battula commented on HDFS-15244:
-
duplicated to which jira..?
> Make Decommission related configurations runtime refreshable
>
>
> Key: HDFS-15244
> URL: https://issues.apache.org/jira/browse/HDFS-15244
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>
> There are certain configurations that can be tuned to increase the speed of
> Decommissioning, But restarting namenode for it seems to be a heavy operation.
> Propose to make these configurations, reconfigurable during runtime in order
> to speed-up decommissioning as per need.
> Proposed Configs :
> * {{dfs.namenode.replication.work.multiplier.per.iteration}}
> * {{dfs.namenode.replication.max-streams-hard-limit}}
> * {{dfs.namenode.replication.max-streams}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15244) Make Decommission related configurations runtime refreshable
[
https://issues.apache.org/jira/browse/HDFS-15244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068562#comment-17068562
]
Ayush Saxena commented on HDFS-15244:
-
HDFS-14560
> Make Decommission related configurations runtime refreshable
>
>
> Key: HDFS-15244
> URL: https://issues.apache.org/jira/browse/HDFS-15244
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>
> There are certain configurations that can be tuned to increase the speed of
> Decommissioning, But restarting namenode for it seems to be a heavy operation.
> Propose to make these configurations, reconfigurable during runtime in order
> to speed-up decommissioning as per need.
> Proposed Configs :
> * {{dfs.namenode.replication.work.multiplier.per.iteration}}
> * {{dfs.namenode.replication.max-streams-hard-limit}}
> * {{dfs.namenode.replication.max-streams}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15241) Distcp print wrong log info when use -log
[ https://issues.apache.org/jira/browse/HDFS-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068582#comment-17068582 ] Brahma Reddy Battula commented on HDFS-15241: - [~rain_lyy] nice finding, we can give "bytescopied" here. > Distcp print wrong log info when use -log > - > > Key: HDFS-15241 > URL: https://issues.apache.org/jira/browse/HDFS-15241 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: image-2020-03-25-17-28-33-394.png > > > when run distcp with -log /logpath -v, distcp will print copy status and file > info to /logpath, but print log with wrong file zise. The logs print as > follows: > FILE_COPIED: source=hdfs://ns1/test/stax2-api-3.1.4.jar, size=161867 --> > target=hdfs://ns1/tmp/target/stax2-api-3.1.4.jar, size=0 > As I analysis ,the root cause is as follows: > targrtFileStatus got before copying. So targrtFileStatus is null. Here should > get targrtFileStatus again after file copying. > !image-2020-03-25-17-28-33-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Attachment: HDFS-15245.003.patch > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, jn web > 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Status: In Progress (was: Patch Available) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, jn web > 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Release Note: fix the failed new testcase, asflicense seems not related. Status: Patch Available (was: In Progress) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, jn web > 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068788#comment-17068788 ] Íñigo Goiri commented on HDFS-15245: Thanks [~jiangjianfei], I have to say I've had to go through the raw JMX output a bunch of times to check the state of the edits. Having it in the UI makes it much cleaner when debugging. Let's fix the checkstyles raised by Yetus. Regarding detailed RPC metrics, let's open a new JIRA and do a Web UI component to be used by all the HDFS components (NN, DN, Router, and JN). This view could go to the "Utilities" tab of all components. > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, jn web > 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068791#comment-17068791
]
Íñigo Goiri commented on HDFS-15245:
BTW, if I'm not wrong, a JN could host edits for multiple cluster ids, not
ideal but technically possible.
Actually, in JournalNode#setClusterId(), there is some incomplete logic because
of this.
Technically, it should be {{List getClusterIds();}}.
> Improve JournalNode web UI
> --
>
> Key: HDFS-15245
> URL: https://issues.apache.org/jira/browse/HDFS-15245
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: journal-node, ui
>Affects Versions: 3.2.1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, jn web
> 1.PNG, jn web 2.PNG
>
>
> At present, journalnode web UI is a almost blank page. It almost has no
> useful infomation.
> # Add some infomation of journalnode at the main page.
> # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level,
> Metrics, Configuration & Process Thread Dump
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-13183:
---
Attachment: HDFS-13183.004.patch
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Affects Versions: 2.7.5, 3.1.0, 2.9.1, 2.8.4, 3.0.2
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch, HDFS-13183.004.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068809#comment-17068809
]
Xiaoqiao He commented on HDFS-13183:
Considering this feature is deployed and used by many users, I would like to
pick up this again and submit new patch based on branch trunk.
I would like to state,
A. v004 offer configuration to enable/disable this feature for users. It is
disable by default.
B. This feature is just one choice for end users to send high load request to
Active NN, Observer NN or Standby NN.
C. based on my internal cluster practice over 2 years, it is helpful to reduce
load of Active NN.
Hi [~weichiu][~elgoiri] and other guys, anyone would like to give review about
v004? Thanks.
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Affects Versions: 2.7.5, 3.1.0, 2.9.1, 2.8.4, 3.0.2
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch, HDFS-13183.004.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15169) RBF: Router FSCK should consider the mount table
[ https://issues.apache.org/jira/browse/HDFS-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15169: --- Attachment: HDFS-15169.003.patch > RBF: Router FSCK should consider the mount table > > > Key: HDFS-15169 > URL: https://issues.apache.org/jira/browse/HDFS-15169 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Akira Ajisaka >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15169.001.patch, HDFS-15169.002.patch, > HDFS-15169.003.patch > > > HDFS-13989 implemented FSCK to DFSRouter, however, it just redirects the > requests to all the active downstream NameNodes for now. The DFSRouter should > consider the mount table when redirecting the requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15169) RBF: Router FSCK should consider the mount table
[ https://issues.apache.org/jira/browse/HDFS-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068823#comment-17068823 ] Xiaoqiao He commented on HDFS-15169: Thanks [~elgoiri] for your reviews. v003 try to refactor the main flow codes and fix the checkstyle. The failed unit test seems unrelated with the changes. I try to run times at local it does not reproduce. Yetus execution logs also shows unrelated, please help to give another checks, Thanks again. > RBF: Router FSCK should consider the mount table > > > Key: HDFS-15169 > URL: https://issues.apache.org/jira/browse/HDFS-15169 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Akira Ajisaka >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15169.001.patch, HDFS-15169.002.patch, > HDFS-15169.003.patch > > > HDFS-13989 implemented FSCK to DFSRouter, however, it just redirects the > requests to all the active downstream NameNodes for now. The DFSRouter should > consider the mount table when redirecting the requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15051) RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
[
https://issues.apache.org/jira/browse/HDFS-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068824#comment-17068824
]
Xiaoqiao He commented on HDFS-15051:
[~ayushtkn],[~elgoiri] any thought or suggestions here?
> RBF: Propose to revoke WRITE MountTableEntry privilege to super user only
> -
>
> Key: HDFS-15051
> URL: https://issues.apache.org/jira/browse/HDFS-15051
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-15051.001.patch, HDFS-15051.002.patch,
> HDFS-15051.003.patch, HDFS-15051.004.patch, HDFS-15051.005.patch,
> HDFS-15051.006.patch, HDFS-15051.007.patch, HDFS-15051.008.patch
>
>
> The current permission checker of #MountTableStoreImpl is not very restrict.
> In some case, any user could add/update/remove MountTableEntry without the
> expected permission checking.
> The following code segment try to check permission when operate
> MountTableEntry, however mountTable object is from Client/RouterAdmin
> {{MountTable mountTable = request.getEntry();}}, and user could pass any mode
> which could bypass the permission checker.
> {code:java}
> public void checkPermission(MountTable mountTable, FsAction access)
> throws AccessControlException {
> if (isSuperUser()) {
> return;
> }
> FsPermission mode = mountTable.getMode();
> if (getUser().equals(mountTable.getOwnerName())
> && mode.getUserAction().implies(access)) {
> return;
> }
> if (isMemberOfGroup(mountTable.getGroupName())
> && mode.getGroupAction().implies(access)) {
> return;
> }
> if (!getUser().equals(mountTable.getOwnerName())
> && !isMemberOfGroup(mountTable.getGroupName())
> && mode.getOtherAction().implies(access)) {
> return;
> }
> throw new AccessControlException(
> "Permission denied while accessing mount table "
> + mountTable.getSourcePath()
> + ": user " + getUser() + " does not have " + access.toString()
> + " permissions.");
> }
> {code}
> I just propose revoke WRITE MountTableEntry privilege to super user only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-13183:
---
Affects Version/s: (was: 3.0.2)
(was: 2.8.4)
(was: 2.9.1)
(was: 3.1.0)
(was: 2.7.5)
Status: Patch Available (was: Open)
Try to trigger Jenkins.
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch, HDFS-13183.004.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068906#comment-17068906
]
Hadoop QA commented on HDFS-15245:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 10 unchanged - 1 fixed = 13 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 56s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-15245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12997999/HDFS-15245.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 362ddbdc4f80 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cbe71ea |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/29031/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/29031/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/29031/testReport/ |
| Max. process+thread count | 2639 (vs. ulimit of 5500) |
|
[jira] [Commented] (HDFS-15169) RBF: Router FSCK should consider the mount table
[
https://issues.apache.org/jira/browse/HDFS-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068910#comment-17068910
]
Hadoop QA commented on HDFS-15169:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
44s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-15169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998009/HDFS-15169.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f03ac546778e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cbe71ea |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/29032/testReport/ |
| Max. process+thread count | 2967 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29032/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Router FSCK should consider the mount table
>
>
> Key: HDFS-15169
> U
[jira] [Created] (HDFS-15246) ArrayIndexOfboundsException in BlockManager CreateLocatedBlock
hemanthboyina created HDFS-15246: Summary: ArrayIndexOfboundsException in BlockManager CreateLocatedBlock Key: HDFS-15246 URL: https://issues.apache.org/jira/browse/HDFS-15246 Project: Hadoop HDFS Issue Type: Bug Reporter: hemanthboyina Assignee: hemanthboyina -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15246) ArrayIndexOfboundsException in BlockManager CreateLocatedBlock
[ https://issues.apache.org/jira/browse/HDFS-15246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-15246: - Description: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1362) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1501) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2047) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:770) > ArrayIndexOfboundsException in BlockManager CreateLocatedBlock > -- > > Key: HDFS-15246 > URL: https://issues.apache.org/jira/browse/HDFS-15246 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > > java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 > > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1362) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1501) > at > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2047) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:770) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068994#comment-17068994
]
Hadoop QA commented on HDFS-13183:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 657 unchanged - 0 fixed = 658 total (was 657) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
18s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 5s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Nullcheck of namenodes at line 116 of value previously dereferenced in
org.apache.hadoop.hdfs.server.balancer.NameNodeConnector.newNameNodeConnectors(Collection,
Collection, String, Path, Configuration, int) At NameNodeConnector.java:116
of value previously dereferenced in
org.apache.hadoop.hdfs.server.balancer.NameNodeConnector.newNameNodeConnectors(Collection,
Collection, String, Path, Configuration, int) At NameNodeConnector.java:[line
114] |
| Failed junit tests |
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA I
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069006#comment-17069006
]
Hadoop QA commented on HDFS-13183:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 657 unchanged - 0 fixed = 658 total (was 657) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
28s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 10s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Nullcheck of namenodes at line 116 of value previously dereferenced in
org.apache.hadoop.hdfs.server.balancer.NameNodeConnector.newNameNodeConnectors(Collection,
Collection, String, Path, Configuration, int) At NameNodeConnector.java:116
of value previously dereferenced in
org.apache.hadoop.hdfs.server.balancer.NameNodeConnector.newNameNodeConnectors(Collection,
Collection, String, Path, Configuration, int) At NameNodeConnector.java:[line
114] |
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-13183 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998008/HDFS-13183.004.patch |
| Optional Tests | dupnam
[jira] [Commented] (HDFS-15191) EOF when reading legacy buffer in BlockTokenIdentifier
[
https://issues.apache.org/jira/browse/HDFS-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069016#comment-17069016
]
Chen Liang commented on HDFS-15191:
---
[~Steven Rand] the change makes sense to me, nice catch on the issue! +1 to
v004 patch, will commit soon.
> EOF when reading legacy buffer in BlockTokenIdentifier
> --
>
> Key: HDFS-15191
> URL: https://issues.apache.org/jira/browse/HDFS-15191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Attachments: HDFS-15191-001.patch, HDFS-15191-002.patch,
> HDFS-15191.003.patch, HDFS-15191.004.patch
>
>
> We have an HDFS client application which recently upgraded from 3.2.0 to
> 3.2.1. After this upgrade (but not before), we sometimes see these errors
> when this application is used with clusters still running Hadoop 2.x (more
> specifically CDH 5.12.1):
> {code}
> WARN [2020-02-24T00:54:32.856Z]
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing
> remote block reader. (_sampled: true)
> java.io.EOFException:
> at java.io.DataInputStream.readByte(DataInputStream.java:272)
> at
> org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
> at
> org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
> at
> org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
> at
> org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
> {code}
> We get this warning for all DataNodes with a copy of the block, so the read
> fails.
> I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to
> cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging
> [~vagarychen] in case you have any ideas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15191) EOF when reading legacy buffer in BlockTokenIdentifier
[
https://issues.apache.org/jira/browse/HDFS-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-15191:
--
Fix Version/s: 3.3.1
3.2.2
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk and branch-3.2, thanks [~Steven Rand] for the contribution.
> EOF when reading legacy buffer in BlockTokenIdentifier
> --
>
> Key: HDFS-15191
> URL: https://issues.apache.org/jira/browse/HDFS-15191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Fix For: 3.2.2, 3.3.1
>
> Attachments: HDFS-15191-001.patch, HDFS-15191-002.patch,
> HDFS-15191.003.patch, HDFS-15191.004.patch
>
>
> We have an HDFS client application which recently upgraded from 3.2.0 to
> 3.2.1. After this upgrade (but not before), we sometimes see these errors
> when this application is used with clusters still running Hadoop 2.x (more
> specifically CDH 5.12.1):
> {code}
> WARN [2020-02-24T00:54:32.856Z]
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing
> remote block reader. (_sampled: true)
> java.io.EOFException:
> at java.io.DataInputStream.readByte(DataInputStream.java:272)
> at
> org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
> at
> org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
> at
> org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
> at
> org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
> {code}
> We get this warning for all DataNodes with a copy of the block, so the read
> fails.
> I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to
> cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging
> [~vagarychen] in case you have any ideas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15191) EOF when reading legacy buffer in BlockTokenIdentifier
[
https://issues.apache.org/jira/browse/HDFS-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069022#comment-17069022
]
Hudson commented on HDFS-15191:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18099 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18099/])
HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. (cliang:
rev f531a4a487c9133bce20d08e09da4d4a35bff13d)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/token/block/TestBlockToken.java
> EOF when reading legacy buffer in BlockTokenIdentifier
> --
>
> Key: HDFS-15191
> URL: https://issues.apache.org/jira/browse/HDFS-15191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Fix For: 3.2.2, 3.3.1
>
> Attachments: HDFS-15191-001.patch, HDFS-15191-002.patch,
> HDFS-15191.003.patch, HDFS-15191.004.patch
>
>
> We have an HDFS client application which recently upgraded from 3.2.0 to
> 3.2.1. After this upgrade (but not before), we sometimes see these errors
> when this application is used with clusters still running Hadoop 2.x (more
> specifically CDH 5.12.1):
> {code}
> WARN [2020-02-24T00:54:32.856Z]
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing
> remote block reader. (_sampled: true)
> java.io.EOFException:
> at java.io.DataInputStream.readByte(DataInputStream.java:272)
> at
> org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
> at
> org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
> at
> org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
> at
> org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
> {code}
> We get this warning for all DataNodes with a copy of the block, so the read
> fails.
> I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to
> cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging
> [~vagarychen] in case you have any ideas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additiona
[jira] [Commented] (HDFS-15191) EOF when reading legacy buffer in BlockTokenIdentifier
[
https://issues.apache.org/jira/browse/HDFS-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069037#comment-17069037
]
Steven Rand commented on HDFS-15191:
Thanks, [~vagarychen]!
Do you think the backcompat break is significant enough that it's worth trying
to get this into 3.3.0 release?
> EOF when reading legacy buffer in BlockTokenIdentifier
> --
>
> Key: HDFS-15191
> URL: https://issues.apache.org/jira/browse/HDFS-15191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Fix For: 3.2.2, 3.3.1
>
> Attachments: HDFS-15191-001.patch, HDFS-15191-002.patch,
> HDFS-15191.003.patch, HDFS-15191.004.patch
>
>
> We have an HDFS client application which recently upgraded from 3.2.0 to
> 3.2.1. After this upgrade (but not before), we sometimes see these errors
> when this application is used with clusters still running Hadoop 2.x (more
> specifically CDH 5.12.1):
> {code}
> WARN [2020-02-24T00:54:32.856Z]
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing
> remote block reader. (_sampled: true)
> java.io.EOFException:
> at java.io.DataInputStream.readByte(DataInputStream.java:272)
> at
> org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
> at
> org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
> at
> org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
> at
> org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
> {code}
> We get this warning for all DataNodes with a copy of the block, so the read
> fails.
> I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to
> cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging
> [~vagarychen] in case you have any ideas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15169) RBF: Router FSCK should consider the mount table
[ https://issues.apache.org/jira/browse/HDFS-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069134#comment-17069134 ] Íñigo Goiri commented on HDFS-15169: +1 on [^HDFS-15169.003.patch]. > RBF: Router FSCK should consider the mount table > > > Key: HDFS-15169 > URL: https://issues.apache.org/jira/browse/HDFS-15169 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Akira Ajisaka >Assignee: Xiaoqiao He >Priority: Major > Attachments: HDFS-15169.001.patch, HDFS-15169.002.patch, > HDFS-15169.003.patch > > > HDFS-13989 implemented FSCK to DFSRouter, however, it just redirects the > requests to all the active downstream NameNodes for now. The DFSRouter should > consider the mount table when redirecting the requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: image-2020-03-28-09-23-31-335.png > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > Attachments: image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069165#comment-17069165 ] liuyanyu commented on HDFS-15243: - Yes, we tend to allow read/write but just prevent delete sub-directories under the protected directories. If performance is ok, we even want to protect files from being deleted. My idea is to check if parent directory is in the protected directories list when we try to directory , like this. But this seems to cause performance degradation. waiting for your advice [~ayushtkn] !image-2020-03-28-09-23-31-335.png! > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > Attachments: image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15241) Distcp print wrong log info when use -log
[ https://issues.apache.org/jira/browse/HDFS-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069170#comment-17069170 ] liuyanyu commented on HDFS-15241: - Thanks [~brahmareddy]， will upload patch. > Distcp print wrong log info when use -log > - > > Key: HDFS-15241 > URL: https://issues.apache.org/jira/browse/HDFS-15241 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: image-2020-03-25-17-28-33-394.png > > > when run distcp with -log /logpath -v, distcp will print copy status and file > info to /logpath, but print log with wrong file zise. The logs print as > follows: > FILE_COPIED: source=hdfs://ns1/test/stax2-api-3.1.4.jar, size=161867 --> > target=hdfs://ns1/tmp/target/stax2-api-3.1.4.jar, size=0 > As I analysis ,the root cause is as follows: > targrtFileStatus got before copying. So targrtFileStatus is null. Here should > get targrtFileStatus again after file copying. > !image-2020-03-25-17-28-33-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069201#comment-17069201 ] Jianfei Jiang commented on HDFS-15245: -- Thank you [~elgoiri]. Yes the journalnode maybe used by more than more cluster. I forget it as I never use as this. Update a new patch to change String getClusterId() to List getClusterIds() and also solve some checkstyle problem. > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, jn web > 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Status: In Progress (was: Patch Available) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, > HDFS-15245.004.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Attachment: HDFS-15245.004.patch > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, > HDFS-15245.004.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Status: Patch Available (was: In Progress) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, > HDFS-15245.004.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-13183:
---
Attachment: HDFS-13183.005.patch
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch, HDFS-13183.004.patch, HDFS-13183.005.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069207#comment-17069207
]
Xiaoqiao He commented on HDFS-13183:
v005 try to fix the findbugs and checkstyle.
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch,
> HDFS-13183-trunk.003.patch, HDFS-13183.004.patch, HDFS-13183.005.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15241) Distcp print wrong log info when use -log
[ https://issues.apache.org/jira/browse/HDFS-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15241: Attachment: HDFS-15241.001.patch Status: Patch Available (was: Open) > Distcp print wrong log info when use -log > - > > Key: HDFS-15241 > URL: https://issues.apache.org/jira/browse/HDFS-15241 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: HDFS-15241.001.patch, image-2020-03-25-17-28-33-394.png > > > when run distcp with -log /logpath -v, distcp will print copy status and file > info to /logpath, but print log with wrong file zise. The logs print as > follows: > FILE_COPIED: source=hdfs://ns1/test/stax2-api-3.1.4.jar, size=161867 --> > target=hdfs://ns1/tmp/target/stax2-api-3.1.4.jar, size=0 > As I analysis ,the root cause is as follows: > targrtFileStatus got before copying. So targrtFileStatus is null. Here should > get targrtFileStatus again after file copying. > !image-2020-03-25-17-28-33-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15241) Distcp print wrong log info when use -log
[
https://issues.apache.org/jira/browse/HDFS-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069239#comment-17069239
]
Hadoop QA commented on HDFS-15241:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
32s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-15241 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998056/HDFS-15241.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6f006a70d1ac 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f531a4a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/29038/testReport/ |
| Max. process+thread count | 444 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29038/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Distcp print wrong log info when use -log
> ---
[jira] [Updated] (HDFS-15245) Improve JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-15245: - Release Note: (was: fix the failed new testcase, asflicense seems not related.) > Improve JournalNode web UI > -- > > Key: HDFS-15245 > URL: https://issues.apache.org/jira/browse/HDFS-15245 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.2.1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: HDFS-15245.002.patch, HDFS-15245.003.patch, > HDFS-15245.004.patch, jn web 1.PNG, jn web 2.PNG > > > At present, journalnode web UI is a almost blank page. It almost has no > useful infomation. > # Add some infomation of journalnode at the main page. > # Add a dropdown menu Utilities like DN and JN, contains Logs, Log Level, > Metrics, Configuration & Process Thread Dump > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15245) Improve JournalNode web UI
[
https://issues.apache.org/jira/browse/HDFS-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069261#comment-17069261
]
Hadoop QA commented on HDFS-15245:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 42s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 585 unchanged -
0 fixed = 586 total (was 585) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 9 unchanged - 2 fixed = 11 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 47s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.TestINodeAttributeProvider |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-15245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998054/HDFS-15245.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux df54fe7da89c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU
