[jira] [Commented] (HDFS-14783) expired SampleStat need to be removed from SlowPeersReport
[
https://issues.apache.org/jira/browse/HDFS-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060623#comment-17060623
]
Haibin Huang commented on HDFS-14783:
-
[~ayushtkn] [~elgoiri] , can you take a look for this, thanks.
> expired SampleStat need to be removed from SlowPeersReport
> --
>
> Key: HDFS-14783
> URL: https://issues.apache.org/jira/browse/HDFS-14783
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HDFS-14783, HDFS-14783-001.patch, HDFS-14783-002.patch
>
>
> SlowPeersReport is calculated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> the SampleStat is stored in a LinkedBlockingDeque, it won't be
> removed until the queue is full and a newest one is generated. Therefore, if
> dn1 don't send any packet to dn2 for a long time, the old SampleStat will
> keep staying in the queue, and will be used to calculated slowpeer.I think
> these old SampleStats should be considered as expired message and ignore them
> when generating a new SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14783) expired SampleStat need to be removed from SlowPeersReport
[
https://issues.apache.org/jira/browse/HDFS-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibin Huang updated HDFS-14783:
Description:
SlowPeersReport is calculated by the SampleStat between tow dn, so it can
present on nn's jmx like this:
{code:java}
"SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
{code}
the SampleStat is stored in a LinkedBlockingDeque, it won't be
removed until the queue is full and a newest one is generated. Therefore, if
dn1 don't send any packet to dn2 for a long time, the old SampleStat will keep
staying in the queue, and will be used to calculated slowpeer.I think these old
SampleStats should be considered as expired message and ignore them when
generating a new SlowPeersReport.
was:
SlowPeersReport in namenode's jmx can tell us which datanode is slow node, and
it is calculated by the average duration between two datanode sending packet.
Here is an example, if dn1 send packet to dn2 tasks too long in average (over
the *upperLimitLatency*), you will see SlowPeersReport in namenode's jmx like
this :
{code:java}
"SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
{code}
However, if dn1 just sending some packet to dn2 with a slow speed in the
beginning , then didn't send any packet to dn2 for a long time, which will keep
the abovementioned SlowPeersReport staying on namenode's jmx . I think this
SlowPeersReport might be an expired message, because the network between dn1
and dn2 may have returned to normal, but the SlowPeersReport is still on
nameonode's jmx until next time dn1 sending packet to dn2. So I use a timestamp
to record when an *org.apache.hadoop.metrics2.util.SampleStat* is created, and
calculate the average duration with the valid *SampleStat ,* which is judged by
it timestamp.
> expired SampleStat need to be removed from SlowPeersReport
> --
>
> Key: HDFS-14783
> URL: https://issues.apache.org/jira/browse/HDFS-14783
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HDFS-14783, HDFS-14783-001.patch, HDFS-14783-002.patch
>
>
> SlowPeersReport is calculated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> the SampleStat is stored in a LinkedBlockingDeque, it won't be
> removed until the queue is full and a newest one is generated. Therefore, if
> dn1 don't send any packet to dn2 for a long time, the old SampleStat will
> keep staying in the queue, and will be used to calculated slowpeer.I think
> these old SampleStats should be considered as expired message and ignore them
> when generating a new SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15154) Allow only hdfs superusers the ability to assign HDFS storage policies
[ https://issues.apache.org/jira/browse/HDFS-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060600#comment-17060600 ] Siddharth Wagle edited comment on HDFS-15154 at 3/17/20, 3:50 AM: -- We could fallback to not have the deprecation and adding a new boolean-valued config which indicates whether superuser only, any comment [~arp], since the change to enum was a suggestion from you? was (Author: swagle): We could fallback to not have the deprecation and adding a new boolean-valued config which indicates whether superuser only, It any comment [~arp], since the change to enum was a suggestion from you? > Allow only hdfs superusers the ability to assign HDFS storage policies > -- > > Key: HDFS-15154 > URL: https://issues.apache.org/jira/browse/HDFS-15154 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Bob Cauthen >Assignee: Siddharth Wagle >Priority: Major > Attachments: HDFS-15154.01.patch, HDFS-15154.02.patch, > HDFS-15154.03.patch, HDFS-15154.04.patch, HDFS-15154.05.patch, > HDFS-15154.06.patch, HDFS-15154.07.patch, HDFS-15154.08.patch, > HDFS-15154.09.patch, HDFS-15154.10.patch, HDFS-15154.11.patch, > HDFS-15154.12.patch > > > Please provide a way to limit only HDFS superusers the ability to assign HDFS > Storage Policies to HDFS directories. > Currently, and based on Jira HDFS-7093, all storage policies can be disabled > cluster wide by setting the following: > dfs.storage.policy.enabled to false > But we need a way to allow only HDFS superusers the ability to assign an HDFS > Storage Policy to an HDFS directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14783) expired SampleStat need to be removed from SlowPeersReport
[
https://issues.apache.org/jira/browse/HDFS-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibin Huang updated HDFS-14783:
Attachment: HDFS-14783-002.patch
> expired SampleStat need to be removed from SlowPeersReport
> --
>
> Key: HDFS-14783
> URL: https://issues.apache.org/jira/browse/HDFS-14783
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HDFS-14783, HDFS-14783-001.patch, HDFS-14783-002.patch
>
>
> SlowPeersReport in namenode's jmx can tell us which datanode is slow node,
> and it is calculated by the average duration between two datanode sending
> packet. Here is an example, if dn1 send packet to dn2 tasks too long in
> average (over the *upperLimitLatency*), you will see SlowPeersReport in
> namenode's jmx like this :
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> However, if dn1 just sending some packet to dn2 with a slow speed in the
> beginning , then didn't send any packet to dn2 for a long time, which will
> keep the abovementioned SlowPeersReport staying on namenode's jmx . I think
> this SlowPeersReport might be an expired message, because the network between
> dn1 and dn2 may have returned to normal, but the SlowPeersReport is still on
> nameonode's jmx until next time dn1 sending packet to dn2. So I use a
> timestamp to record when an *org.apache.hadoop.metrics2.util.SampleStat* is
> created, and calculate the average duration with the valid *SampleStat ,*
> which is judged by it timestamp.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14783) expired SampleStat need to be removed from SlowPeersReport
[
https://issues.apache.org/jira/browse/HDFS-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibin Huang updated HDFS-14783:
Summary: expired SampleStat need to be removed from SlowPeersReport (was:
expired SlowPeersReport will keep staying on namenode's jmx)
> expired SampleStat need to be removed from SlowPeersReport
> --
>
> Key: HDFS-14783
> URL: https://issues.apache.org/jira/browse/HDFS-14783
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HDFS-14783, HDFS-14783-001.patch
>
>
> SlowPeersReport in namenode's jmx can tell us which datanode is slow node,
> and it is calculated by the average duration between two datanode sending
> packet. Here is an example, if dn1 send packet to dn2 tasks too long in
> average (over the *upperLimitLatency*), you will see SlowPeersReport in
> namenode's jmx like this :
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> However, if dn1 just sending some packet to dn2 with a slow speed in the
> beginning , then didn't send any packet to dn2 for a long time, which will
> keep the abovementioned SlowPeersReport staying on namenode's jmx . I think
> this SlowPeersReport might be an expired message, because the network between
> dn1 and dn2 may have returned to normal, but the SlowPeersReport is still on
> nameonode's jmx until next time dn1 sending packet to dn2. So I use a
> timestamp to record when an *org.apache.hadoop.metrics2.util.SampleStat* is
> created, and calculate the average duration with the valid *SampleStat ,*
> which is judged by it timestamp.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15154) Allow only hdfs superusers the ability to assign HDFS storage policies
[ https://issues.apache.org/jira/browse/HDFS-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060600#comment-17060600 ] Siddharth Wagle commented on HDFS-15154: We could fallback to not have the deprecation and adding a new boolean-valued config which indicates whether superuser only, It any comment [~arp], since the change to enum was a suggestion from you? > Allow only hdfs superusers the ability to assign HDFS storage policies > -- > > Key: HDFS-15154 > URL: https://issues.apache.org/jira/browse/HDFS-15154 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Bob Cauthen >Assignee: Siddharth Wagle >Priority: Major > Attachments: HDFS-15154.01.patch, HDFS-15154.02.patch, > HDFS-15154.03.patch, HDFS-15154.04.patch, HDFS-15154.05.patch, > HDFS-15154.06.patch, HDFS-15154.07.patch, HDFS-15154.08.patch, > HDFS-15154.09.patch, HDFS-15154.10.patch, HDFS-15154.11.patch, > HDFS-15154.12.patch > > > Please provide a way to limit only HDFS superusers the ability to assign HDFS > Storage Policies to HDFS directories. > Currently, and based on Jira HDFS-7093, all storage policies can be disabled > cluster wide by setting the following: > dfs.storage.policy.enabled to false > But we need a way to allow only HDFS superusers the ability to assign an HDFS > Storage Policy to an HDFS directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15221) Add checking of effective filesystem during initializing storage locations
[ https://issues.apache.org/jira/browse/HDFS-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060582#comment-17060582 ] Yang Yun edited comment on HDFS-15221 at 3/17/20, 2:22 AM: --- Thanks [~elgoiri] for the review. Updated new path HDFS-15221-006.patch with below changes, * move the javadoc to the header of function * change the fuction to return false instead of using exceptions. * remove the bracklines in the javadocs. I'm sorry that I don't know where to document this feature? was (Author: hadoop_yangyun): Thanks [~elgoiri] for the review. Updated new path HDFS-15221-006.patch with below changes, * move the javadoc to the header of function * change the fuction to return false instead of using exceptions. * remove the bracklines in the javadocs. I'm sorry that I don't know where to cocument this feature? > Add checking of effective filesystem during initializing storage locations > -- > > Key: HDFS-15221 > URL: https://issues.apache.org/jira/browse/HDFS-15221 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15221-002.patch, HDFS-15221-003.patch, > HDFS-15221-004.patch, HDFS-15221-005.patch, HDFS-15221-006.patch, > HDFS-15221.patch > > > We sometimes mount different disks for different storage types as the storage > location. It's important to check the volume is mounted rightly before > initializing storage locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15221) Add checking of effective filesystem during initializing storage locations
[ https://issues.apache.org/jira/browse/HDFS-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060582#comment-17060582 ] Yang Yun commented on HDFS-15221: - Thanks [~elgoiri] for the review. Updated new path HDFS-15221-006.patch with below changes, * move the javadoc to the header of function * change the fuction to return false instead of using exceptions. * remove the bracklines in the javadocs. I'm sorry that I don't know where to cocument this feature? > Add checking of effective filesystem during initializing storage locations > -- > > Key: HDFS-15221 > URL: https://issues.apache.org/jira/browse/HDFS-15221 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15221-002.patch, HDFS-15221-003.patch, > HDFS-15221-004.patch, HDFS-15221-005.patch, HDFS-15221-006.patch, > HDFS-15221.patch > > > We sometimes mount different disks for different storage types as the storage > location. It's important to check the volume is mounted rightly before > initializing storage locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15221) Add checking of effective filesystem during initializing storage locations
[ https://issues.apache.org/jira/browse/HDFS-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15221: Attachment: HDFS-15221-006.patch Status: Patch Available (was: Open) > Add checking of effective filesystem during initializing storage locations > -- > > Key: HDFS-15221 > URL: https://issues.apache.org/jira/browse/HDFS-15221 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15221-002.patch, HDFS-15221-003.patch, > HDFS-15221-004.patch, HDFS-15221-005.patch, HDFS-15221-006.patch, > HDFS-15221.patch > > > We sometimes mount different disks for different storage types as the storage > location. It's important to check the volume is mounted rightly before > initializing storage locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15221) Add checking of effective filesystem during initializing storage locations
[ https://issues.apache.org/jira/browse/HDFS-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15221: Status: Open (was: Patch Available) > Add checking of effective filesystem during initializing storage locations > -- > > Key: HDFS-15221 > URL: https://issues.apache.org/jira/browse/HDFS-15221 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15221-002.patch, HDFS-15221-003.patch, > HDFS-15221-004.patch, HDFS-15221-005.patch, HDFS-15221.patch > > > We sometimes mount different disks for different storage types as the storage > location. It's important to check the volume is mounted rightly before > initializing storage locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15214) WebHDFS: Add snapshot counts to Content Summary
[
https://issues.apache.org/jira/browse/HDFS-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060522#comment-17060522
]
Hadoop QA commented on HDFS-15214:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 67
unchanged - 1 fixed = 67 total (was 68) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 36s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
32s{color} | {color:red} The patch generated 18 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs |
| | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestFsShellPermission |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForStoragePolicy |
| | hadoop.hdfs.tools.TestECAdmin |
| | hadoop.hdfs.tools.TestViewFSStoragePolicyCommands |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary |
| | hadoop.hdfs.TestFileCorruption |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr |
| |
[jira] [Commented] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
[
https://issues.apache.org/jira/browse/HDFS-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060436#comment-17060436
]
Hadoop QA commented on HDFS-15227:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 24s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| | hadoop.hdfs.qjournal.client.TestQJMWithFaults |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-15227 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12996846/HDFS-15227-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux db902f163b16 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ea68863 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28961/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28961/testReport/ |
| Max. process+thread count | 2772 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-15221) Add checking of effective filesystem during initializing storage locations
[ https://issues.apache.org/jira/browse/HDFS-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060421#comment-17060421 ] Íñigo Goiri commented on HDFS-15221: Thanks [~hadoop_yangyun] for the update. Minor comments: * Move the javadoc from 2814-2819 to the function. * Instead of using exceptions (more expensive and should be exceptions not regular code path), let's return false and check for that. * Let's not add so many breaklines in the javadocs between param, return, etc. * Anyway we can document this feature? Other than that, this looks pretty much ready to go. > Add checking of effective filesystem during initializing storage locations > -- > > Key: HDFS-15221 > URL: https://issues.apache.org/jira/browse/HDFS-15221 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15221-002.patch, HDFS-15221-003.patch, > HDFS-15221-004.patch, HDFS-15221-005.patch, HDFS-15221.patch > > > We sometimes mount different disks for different storage types as the storage > location. It's important to check the volume is mounted rightly before > initializing storage locations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060415#comment-17060415 ] Íñigo Goiri commented on HDFS-14919: +1 on [^HDFS-14919.003.patch]. > Provide Non DFS Used per DataNode in DataNode UI > > > Key: HDFS-14919 > URL: https://issues.apache.org/jira/browse/HDFS-14919 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch, > HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png, > screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15214) WebHDFS: Add snapshot counts to Content Summary
[ https://issues.apache.org/jira/browse/HDFS-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060404#comment-17060404 ] hemanthboyina commented on HDFS-15214: -- thanks for the review [~elgoiri] i have updated the patch with your comment fixed , please review > WebHDFS: Add snapshot counts to Content Summary > --- > > Key: HDFS-15214 > URL: https://issues.apache.org/jira/browse/HDFS-15214 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15214.001.patch, HDFS-15214.002.patch, > HDFS-15214.003.patch, HDFS-15214.004.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15214) WebHDFS: Add snapshot counts to Content Summary
[ https://issues.apache.org/jira/browse/HDFS-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-15214: - Attachment: HDFS-15214.004.patch > WebHDFS: Add snapshot counts to Content Summary > --- > > Key: HDFS-15214 > URL: https://issues.apache.org/jira/browse/HDFS-15214 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15214.001.patch, HDFS-15214.002.patch, > HDFS-15214.003.patch, HDFS-15214.004.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060383#comment-17060383 ] Ayush Saxena commented on HDFS-14919: - Nothing as such, may be just my eyes have got used to the previous layout, Just felt the capacity column got little expanded. So... Anyway the code seems fine. +1 > Provide Non DFS Used per DataNode in DataNode UI > > > Key: HDFS-14919 > URL: https://issues.apache.org/jira/browse/HDFS-14919 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch, > HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png, > screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[
https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060382#comment-17060382
]
Hadoop QA commented on HDFS-14919:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HDFS-14919 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14919 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28963/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Provide Non DFS Used per DataNode in DataNode UI
>
>
> Key: HDFS-14919
> URL: https://issues.apache.org/jira/browse/HDFS-14919
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch,
> HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png,
> screenshot-1.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060377#comment-17060377 ] Íñigo Goiri commented on HDFS-14919: It looks good to me, what's the issue with the capacity column? > Provide Non DFS Used per DataNode in DataNode UI > > > Key: HDFS-14919 > URL: https://issues.apache.org/jira/browse/HDFS-14919 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch, > HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png, > screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060367#comment-17060367 ] Ayush Saxena commented on HDFS-14919: - Thanx [~leosun08] for the update. Seems Working Now !screenshot-1.png! Functionality wise seems Ok but the capacity column doesn't look very good to see. [~elgoiri] any opinions on this? > Provide Non DFS Used per DataNode in DataNode UI > > > Key: HDFS-14919 > URL: https://issues.apache.org/jira/browse/HDFS-14919 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch, > HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png, > screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14919: Attachment: screenshot-1.png > Provide Non DFS Used per DataNode in DataNode UI > > > Key: HDFS-14919 > URL: https://issues.apache.org/jira/browse/HDFS-14919 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch, > HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png, > screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15223) FSCK fails if one namenode is not available
[ https://issues.apache.org/jira/browse/HDFS-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060358#comment-17060358 ] Ayush Saxena commented on HDFS-15223: - Thanx [~ferhui] and [~elgoiri] for the reviews. Have made the changes in v2 > FSCK fails if one namenode is not available > --- > > Key: HDFS-15223 > URL: https://issues.apache.org/jira/browse/HDFS-15223 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15223-01.patch, HDFS-15223-02.patch > > > If one namenode is not available FSCK should try on other namenode, ignoring > the namenode not available -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15223) FSCK fails if one namenode is not available
[ https://issues.apache.org/jira/browse/HDFS-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15223: Attachment: HDFS-15223-02.patch > FSCK fails if one namenode is not available > --- > > Key: HDFS-15223 > URL: https://issues.apache.org/jira/browse/HDFS-15223 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15223-01.patch, HDFS-15223-02.patch > > > If one namenode is not available FSCK should try on other namenode, ignoring > the namenode not available -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6874) Add GETFILEBLOCKLOCATIONS operation to HttpFS
[
https://issues.apache.org/jira/browse/HDFS-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060352#comment-17060352
]
Íñigo Goiri commented on HDFS-6874:
---
* We can fix the NoWhitespaceAfter checkstyles.
* I know we are testing implicitly the methods in JsonUtilClient but I think
having unit test for things like toStringArray() is needed (actually, aren't
there existing methods to do this?).
* Avoid the white line change in WebHdfsFileSystem#1814.
* It would make sense to add a method for the whole if in 1828-1831; something
like {{private boolean isGetFileBlockLocationsException(Exception e)}}.
* Make HttpFSFileSystem#toBlockLocations() static and add a test for it.
* BaseTestHttpFSWith#testGetFileBlockLocations shouldn't catch the exception
but just surface it.
In general, it's not the best pattern to try to use the GETFILEBLOCKLOCATIONS
method and then capture the exception and try GET_BLOCK_LOCATIONS; is there a
better way to know if we can use one or the other?
> Add GETFILEBLOCKLOCATIONS operation to HttpFS
> -
>
> Key: HDFS-6874
> URL: https://issues.apache.org/jira/browse/HDFS-6874
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Affects Versions: 2.4.1, 2.7.3
>Reporter: Gao Zhong Liang
>Assignee: Weiwei Yang
>Priority: Major
> Labels: BB2015-05-TBR
> Attachments: HDFS-6874-1.patch, HDFS-6874-branch-2.6.0.patch,
> HDFS-6874.011.patch, HDFS-6874.02.patch, HDFS-6874.03.patch,
> HDFS-6874.04.patch, HDFS-6874.05.patch, HDFS-6874.06.patch,
> HDFS-6874.07.patch, HDFS-6874.08.patch, HDFS-6874.09.patch,
> HDFS-6874.10.patch, HDFS-6874.patch
>
>
> GETFILEBLOCKLOCATIONS operation is missing in HttpFS, which is already
> supported in WebHDFS. For the request of GETFILEBLOCKLOCATIONS in
> org.apache.hadoop.fs.http.server.HttpFSServer, BAD_REQUEST is returned so far:
> ...
> case GETFILEBLOCKLOCATIONS: {
> response = Response.status(Response.Status.BAD_REQUEST).build();
> break;
> }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15223) FSCK fails if one namenode is not available
[ https://issues.apache.org/jira/browse/HDFS-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060332#comment-17060332 ] Íñigo Goiri commented on HDFS-15223: Thanks for bringing this up. A couple comments: * I think it would be good to log the exception instead of just ignoring. * Add a comment before TestHAFsck#81 saying what we are doing high level. > FSCK fails if one namenode is not available > --- > > Key: HDFS-15223 > URL: https://issues.apache.org/jira/browse/HDFS-15223 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15223-01.patch > > > If one namenode is not available FSCK should try on other namenode, ignoring > the namenode not available -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
[ https://issues.apache.org/jira/browse/HDFS-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15227: Status: Patch Available (was: Open) > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > --- > > Key: HDFS-15227 > URL: https://issues.apache.org/jira/browse/HDFS-15227 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15227-01.patch, TestToRepro.patch > > > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > "hdfs fsck / -files -blocks -upgradedomains" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
[ https://issues.apache.org/jira/browse/HDFS-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15227: Attachment: HDFS-15227-01.patch > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > --- > > Key: HDFS-15227 > URL: https://issues.apache.org/jira/browse/HDFS-15227 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15227-01.patch, TestToRepro.patch > > > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > "hdfs fsck / -files -blocks -upgradedomains" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
[
https://issues.apache.org/jira/browse/HDFS-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060328#comment-17060328
]
Ayush Saxena commented on HDFS-15227:
-
I have put a patch that can (by means of breakpoints) can reproduce the issue.
Put breakpoint at :
{code:java}
//display under construction block info.
if (!blocks.isLastBlockComplete() && lastBlock != null) {
ExtendedBlock block = lastBlock.getBlock(); // First Breakpoint...
{code}
Second here :
{code:java}
@Override
public void run() {
DataNodeTestUtils.resumeIBR(cluster.getDataNodes().get(0)); //Second
here...
{code}
Once the first breakpoint is hit(in the NamenodeFSCK), before progressing
ahead, make the second breakpoint progress to shoot the IBR's.
Then the first breakpoint thread.. This shall throw NPE.
> FSCK -upgradedomains is failing for upgradedomains when more than 2 million
> blocks present in hdfs and write in progress of some blocks
> ---
>
> Key: HDFS-15227
> URL: https://issues.apache.org/jira/browse/HDFS-15227
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: krishna reddy
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: TestToRepro.patch
>
>
> FSCK -upgradedomains is failing for upgradedomains when more than 2 million
> blocks present in hdfs and write in progress of some blocks
> "hdfs fsck / -files -blocks -upgradedomains"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
[ https://issues.apache.org/jira/browse/HDFS-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15227: Attachment: TestToRepro.patch > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > --- > > Key: HDFS-15227 > URL: https://issues.apache.org/jira/browse/HDFS-15227 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ayush Saxena >Priority: Major > Attachments: TestToRepro.patch > > > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > "hdfs fsck / -files -blocks -upgradedomains" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10499) TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails Intermittently
[
https://issues.apache.org/jira/browse/HDFS-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060305#comment-17060305
]
Ahmed Hussein commented on HDFS-10499:
--
the failed test case is not related to the patches.
> TestNameNodeMetadataConsistency#testGenerationStampInFuture Fails
> Intermittently
> -
>
> Key: HDFS-10499
> URL: https://issues.apache.org/jira/browse/HDFS-10499
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, test
>Affects Versions: 3.0.0-alpha1
>Reporter: Hanisha Koneru
>Assignee: Yiqun Lin
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10499-branch-2.10.001.patch, HDFS-10499.001.patch,
> HDFS-10499.002.patch
>
>
> Per https://builds.apache.org/job/PreCommit-HDFS-Build/15646/testReport/, we
> had the following failure. Local rerun is successful.
> Stack Trace:
> {panel}
> java.lang.AssertionError: expected:<17> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency.testGenerationStampInFuture(TestNameNodeMetadataConsistency.java:113)
> {panel}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
[
https://issues.apache.org/jira/browse/HDFS-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060252#comment-17060252
]
Ayush Saxena commented on HDFS-15227:
-
Thanx [~mkris.re...@gmail.com] for the report, Was able to reproduce this.
The reason being if dfs.namenode.file.close.num-committed-allowed is 1, so it
allows to close the file even when the block is in committed state(not complete)
The NPE happens in a very rare condition :
If IBR for the last block comes post the if check here :
{code:java}
//display under construction block info.
if (!blocks.isLastBlockComplete() && lastBlock != null) {
ExtendedBlock block = lastBlock.getBlock();
// IBR Comes here and marks the block complete from committed state. Hence
making uc null
{code}
Will put up a fix adding the null check
> FSCK -upgradedomains is failing for upgradedomains when more than 2 million
> blocks present in hdfs and write in progress of some blocks
> ---
>
> Key: HDFS-15227
> URL: https://issues.apache.org/jira/browse/HDFS-15227
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: krishna reddy
>Assignee: Ayush Saxena
>Priority: Major
>
> FSCK -upgradedomains is failing for upgradedomains when more than 2 million
> blocks present in hdfs and write in progress of some blocks
> "hdfs fsck / -files -blocks -upgradedomains"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
[ https://issues.apache.org/jira/browse/HDFS-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reassigned HDFS-15227: --- Assignee: Ayush Saxena > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > --- > > Key: HDFS-15227 > URL: https://issues.apache.org/jira/browse/HDFS-15227 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ayush Saxena >Priority: Major > > FSCK -upgradedomains is failing for upgradedomains when more than 2 million > blocks present in hdfs and write in progress of some blocks > "hdfs fsck / -files -blocks -upgradedomains" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
krishna reddy created HDFS-15227: Summary: FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks Key: HDFS-15227 URL: https://issues.apache.org/jira/browse/HDFS-15227 Project: Hadoop HDFS Issue Type: Bug Reporter: krishna reddy FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks "hdfs fsck / -files -blocks -upgradedomains" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula resolved HDFS-15226.
-
Fix Version/s: (was: 3.2.1)
(was: 3.2.0)
Target Version/s: (was: 3.2.1)
Resolution: Duplicate
Linking the duplicate defect for future reference.
> Ranger integrates HDFS and discovers NPE
>
>
> Key: HDFS-15226
> URL: https://issues.apache.org/jira/browse/HDFS-15226
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.6
> Environment: Apache Ranger1.2 && Hadoop2.7.6
>Reporter: bianqi
>Priority: Critical
>
> When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
> occurred when executing hdfs dfs -ls /.
> However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
> -ls / without any errors, and the directory list can be displayed normally.
> {quote}java.lang.NullPointerException
> at java.lang.String.checkBounds(String.java:384)
> at java.lang.String.(String.java:425)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
> DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
> to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
> xx:8502 Call#0 Retry#0
> {quote}
> When I checked the HDFS source code and debug hdfs source . I found
> pathByNameArr[i] is null.
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
>
> {quote}
> I found that the code of the trunk branch has been fixed and currently has
> not been merged in the latest 3.2.1 version.
> I hope that this patch can be merged into other branches as soon as
> possible，thank you very much!
>
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null)
> Unknown macro: \{ String[] elements = new String[pathIdx + 1]; /** * {@link
> INode#getPathComponents(String)}
> returns a null component
> * for the root only path "/". Assign an empty string if so.
> */
> if (pathByNameArr.length == 1 && pathByNameArr[0] == null)
> Unknown macro: \{ elements[0] = ""; }
> else
> Unknown macro: { for (int i = 0; i < elements.length; i++)
> Unknown macro: \{ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); }}
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Reopened] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula reopened HDFS-15226:
-
> Ranger integrates HDFS and discovers NPE
>
>
> Key: HDFS-15226
> URL: https://issues.apache.org/jira/browse/HDFS-15226
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.6
> Environment: Apache Ranger1.2 && Hadoop2.7.6
>Reporter: bianqi
>Priority: Critical
> Fix For: 3.2.0, 3.2.1
>
>
> When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
> occurred when executing hdfs dfs -ls /.
> However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
> -ls / without any errors, and the directory list can be displayed normally.
> {quote}java.lang.NullPointerException
> at java.lang.String.checkBounds(String.java:384)
> at java.lang.String.(String.java:425)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
> DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
> to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
> xx:8502 Call#0 Retry#0
> {quote}
> When I checked the HDFS source code and debug hdfs source . I found
> pathByNameArr[i] is null.
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
>
> {quote}
> I found that the code of the trunk branch has been fixed and currently has
> not been merged in the latest 3.2.1 version.
> I hope that this patch can be merged into other branches as soon as
> possible，thank you very much!
>
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null)
> Unknown macro: \{ String[] elements = new String[pathIdx + 1]; /** * {@link
> INode#getPathComponents(String)}
> returns a null component
> * for the root only path "/". Assign an empty string if so.
> */
> if (pathByNameArr.length == 1 && pathByNameArr[0] == null)
> Unknown macro: \{ elements[0] = ""; }
> else
> Unknown macro: { for (int i = 0; i < elements.length; i++)
> Unknown macro: \{ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); }}
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060216#comment-17060216
]
bianqi commented on HDFS-15226:
---
Thanks for your reply, I understand.
> Ranger integrates HDFS and discovers NPE
>
>
> Key: HDFS-15226
> URL: https://issues.apache.org/jira/browse/HDFS-15226
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.6
> Environment: Apache Ranger1.2 && Hadoop2.7.6
>Reporter: bianqi
>Priority: Critical
> Fix For: 3.2.0, 3.2.1
>
>
> When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
> occurred when executing hdfs dfs -ls /.
> However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
> -ls / without any errors, and the directory list can be displayed normally.
> {quote}java.lang.NullPointerException
> at java.lang.String.checkBounds(String.java:384)
> at java.lang.String.(String.java:425)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
> DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
> to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
> xx:8502 Call#0 Retry#0
> {quote}
> When I checked the HDFS source code and debug hdfs source . I found
> pathByNameArr[i] is null.
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
>
> {quote}
> I found that the code of the trunk branch has been fixed and currently has
> not been merged in the latest 3.2.1 version.
> I hope that this patch can be merged into other branches as soon as
> possible，thank you very much!
>
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null)
> Unknown macro: \{ String[] elements = new String[pathIdx + 1]; /** * {@link
> INode#getPathComponents(String)}
> returns a null component
> * for the root only path "/". Assign an empty string if so.
> */
> if (pathByNameArr.length == 1 && pathByNameArr[0] == null)
> Unknown macro: \{ elements[0] = ""; }
> else
> Unknown macro: { for (int i = 0; i < elements.length; i++)
> Unknown macro: \{ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); }}
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bianqi resolved HDFS-15226.
---
Resolution: Fixed
> Ranger integrates HDFS and discovers NPE
>
>
> Key: HDFS-15226
> URL: https://issues.apache.org/jira/browse/HDFS-15226
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.6
> Environment: Apache Ranger1.2 && Hadoop2.7.6
>Reporter: bianqi
>Priority: Critical
> Fix For: 3.2.1, 3.2.0
>
>
> When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
> occurred when executing hdfs dfs -ls /.
> However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
> -ls / without any errors, and the directory list can be displayed normally.
> {quote}java.lang.NullPointerException
> at java.lang.String.checkBounds(String.java:384)
> at java.lang.String.(String.java:425)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
> DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
> to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
> xx:8502 Call#0 Retry#0
> {quote}
> When I checked the HDFS source code and debug hdfs source . I found
> pathByNameArr[i] is null.
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
>
> {quote}
> I found that the code of the trunk branch has been fixed and currently has
> not been merged in the latest 3.2.1 version.
> I hope that this patch can be merged into other branches as soon as
> possible，thank you very much!
>
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null)
> Unknown macro: \{ String[] elements = new String[pathIdx + 1]; /** * {@link
> INode#getPathComponents(String)}
> returns a null component
> * for the root only path "/". Assign an empty string if so.
> */
> if (pathByNameArr.length == 1 && pathByNameArr[0] == null)
> Unknown macro: \{ elements[0] = ""; }
> else
> Unknown macro: { for (int i = 0; i < elements.length; i++)
> Unknown macro: \{ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); }}
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060206#comment-17060206
]
Brahma Reddy Battula commented on HDFS-15226:
-
Hope you are asking "HDFS-12614", this is already merged 3.2 also.
FYI..
[https://github.com/apache/hadoop/blame/branch-3.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java]
> Ranger integrates HDFS and discovers NPE
>
>
> Key: HDFS-15226
> URL: https://issues.apache.org/jira/browse/HDFS-15226
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.6
> Environment: Apache Ranger1.2 && Hadoop2.7.6
>Reporter: bianqi
>Priority: Critical
> Fix For: 3.2.0, 3.2.1
>
>
> When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
> occurred when executing hdfs dfs -ls /.
> However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
> -ls / without any errors, and the directory list can be displayed normally.
> {quote}java.lang.NullPointerException
> at java.lang.String.checkBounds(String.java:384)
> at java.lang.String.(String.java:425)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
> DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
> to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
> xx:8502 Call#0 Retry#0
> {quote}
> When I checked the HDFS source code and debug hdfs source . I found
> pathByNameArr[i] is null.
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
>
> {quote}
> I found that the code of the trunk branch has been fixed and currently has
> not been merged in the latest 3.2.1 version.
> I hope that this patch can be merged into other branches as soon as
> possible，thank you very much!
>
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null)
> Unknown macro: \{ String[] elements = new String[pathIdx + 1]; /** * {@link
> INode#getPathComponents(String)}
> returns a null component
> * for the root only path "/". Assign an empty string if so.
> */
> if (pathByNameArr.length == 1 && pathByNameArr[0] == null)
> Unknown macro: \{ elements[0] = ""; }
> else
> Unknown macro: { for (int i = 0; i < elements.length; i++)
> Unknown macro: \{ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); }}
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bianqi updated HDFS-15226:
--
Description:
When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error occurred
when executing hdfs dfs -ls /.
However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs -ls
/ without any errors, and the directory list can be displayed normally.
{quote}java.lang.NullPointerException
at java.lang.String.checkBounds(String.java:384)
at java.lang.String.(String.java:425)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from xx:8502
Call#0 Retry#0
{quote}
When I checked the HDFS source code and debug hdfs source . I found
pathByNameArr[i] is null.
{quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
pathIdx,
INode inode, int snapshotId) {
INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
if (getAttributesProvider() != null) {
String[] elements = new String[pathIdx + 1];
for (int i = 0; i < elements.length; i++) {
elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
}
inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
}
return inodeAttrs;
}
{quote}
I found that the code of the trunk branch has been fixed and currently has not
been merged in the latest 3.2.1 version.
I hope that this patch can be merged into other branches as soon as
possible，thank you very much!
{quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
pathIdx,
INode inode, int snapshotId) {
INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
if (getAttributesProvider() != null)
Unknown macro: \{ String[] elements = new String[pathIdx + 1]; /** * {@link
INode#getPathComponents(String)}
returns a null component
* for the root only path "/". Assign an empty string if so.
*/
if (pathByNameArr.length == 1 && pathByNameArr[0] == null)
Unknown macro: \{ elements[0] = ""; }
else
Unknown macro: { for (int i = 0; i < elements.length; i++)
Unknown macro: \{ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); }}
inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
}
return inodeAttrs;
}
{quote}
was:
When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error occurred
when executing hdfs dfs -ls /.
However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs -ls
/ without any errors, and the directory list can be displayed normally.
{quote}java.lang.NullPointerException
at java.lang.String.checkBounds(String.java:384)
at java.lang.String.(String.java:425)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
[jira] [Updated] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bianqi updated HDFS-15226:
--
Attachment: (was: image-2020-03-16-14-01-03-078.png)
> Ranger integrates HDFS and discovers NPE
>
>
> Key: HDFS-15226
> URL: https://issues.apache.org/jira/browse/HDFS-15226
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.6
> Environment: Apache Ranger1.2 && Hadoop2.7.6
>Reporter: bianqi
>Priority: Critical
> Fix For: 3.2.0, 3.2.1
>
>
> When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
> occurred when executing hdfs dfs -ls /.
> However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
> -ls / without any errors, and the directory list can be displayed normally.
> {quote}java.lang.NullPointerException
> at java.lang.String.checkBounds(String.java:384)
> at java.lang.String.(String.java:425)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
> DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
> to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from
> xx:8502 Call#0 Retry#0
> {quote}
> When I checked the HDFS source code, I compared hadoop2.7.1 and hadoop2.7.6
> and found that 2.7.6 added the following methods
> [^image-2020-03-16-14-01-03-078.png]
> I found that the code of the latest master branch has been fixed and
> currently has not been merged in the latest 3.2.1 version.
>
> {quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
> pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> /**
> * \{@link INode#getPathComponents(String)} returns a null component
> * for the root only path "/". Assign an empty string if so.
> */
> if (pathByNameArr.length == 1 && pathByNameArr[0] == null) {
> elements[0] = "";
> } else {
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
> }
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
> }
> return inodeAttrs;
> }
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15226) Ranger integrates HDFS and discovers NPE
[
https://issues.apache.org/jira/browse/HDFS-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bianqi updated HDFS-15226:
--
Description:
When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error occurred
when executing hdfs dfs -ls /.
However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs -ls
/ without any errors, and the directory list can be displayed normally.
{quote}java.lang.NullPointerException
at java.lang.String.checkBounds(String.java:384)
at java.lang.String.(String.java:425)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding
to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from xx:8502
Call#0 Retry#0
{quote}
When I checked the HDFS source code, I compared hadoop2.7.1 and hadoop2.7.6 and
found that 2.7.6 added the following methods
[^image-2020-03-16-14-01-03-078.png]
I found that the code of the latest master branch has been fixed and currently
has not been merged in the latest 3.2.1 version.
{quote}private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int
pathIdx,
INode inode, int snapshotId) {
INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
if (getAttributesProvider() != null) {
String[] elements = new String[pathIdx + 1];
/**
* \{@link INode#getPathComponents(String)} returns a null component
* for the root only path "/". Assign an empty string if so.
*/
if (pathByNameArr.length == 1 && pathByNameArr[0] == null) {
elements[0] = "";
} else {
for (int i = 0; i < elements.length; i++) {
elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
}
}
inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
}
return inodeAttrs;
}
{quote}
was:
When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
occurred when executing hdfs dfs -ls /.
However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
-ls / without any errors, and the directory list can be displayed normally.
{quote}java.lang.NullPointerException
at java.lang.String.checkBounds(String.java:384)
at java.lang.String.(String.java:425)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
[jira] [Commented] (HDFS-15202) HDFS-client: boost ShortCircuit Cache
[
https://issues.apache.org/jira/browse/HDFS-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059985#comment-17059985
]
Danil Lipovoy commented on HDFS-15202:
--
[~quanli]
I did 2 tests:
1. Via HBase. It was massive reading huge tables:
8 tables by 64 regions by 1.88 Gb data in each = 900 Gb total
Random read in 100, 200,... 500 threads via YCSB (good utility special for DB
performance testing)
It show +25% increase perforormance. Don't so much because the bottle neck is
here HBase .
2. Direct reading by function (I provided more details above):
FSDataInputStream in = fileSystem.open(path);
int res = in.read(position, byteBuffer, 0, 65536);
This function working in separate thread. Each thread is reading separate file
~1Gb. I run 10, 20, ... 200 threads and it means were reading 10, 20, ... 200
files. Importdant notice - this performance possible only with fast SSD or when
data cached by Linux buff/cache (my case). In this scenario increase
performance about +500%.
Environment:
4 nodes E5-2698 v4 @ 2.20GHz (40 cores each), 700 Gb Mem.
> HDFS-client: boost ShortCircuit Cache
> -
>
> Key: HDFS-15202
> URL: https://issues.apache.org/jira/browse/HDFS-15202
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient
> Environment: 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 8 RegionServers (2 by host)
> 8 tables by 64 regions by 1.88 Gb data in each = 900 Gb total
> Random read in 800 threads via YCSB and a little bit updates (10% of reads)
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: HDFS_CPU_full_cycle.png, cpu_SSC.png, cpu_SSC2.png,
> hdfs_cpu.png, hdfs_reads.png, hdfs_scc_3_test.png,
> hdfs_scc_test_full-cycle.png, locks.png, requests_SSC.png
>
>
> ТотI want to propose how to improve reading performance HDFS-client. The
> idea: create few instances ShortCircuit caches instead of one.
> The key points:
> 1. Create array of caches (set by
> clientShortCircuitNum=*dfs.client.short.circuit.num*, see in the pull
> requests below):
> {code:java}
> private ClientContext(String name, DfsClientConf conf, Configuration config) {
> ...
> shortCircuitCache = new ShortCircuitCache[this.clientShortCircuitNum];
> for (int i = 0; i < this.clientShortCircuitNum; i++) {
> this.shortCircuitCache[i] = ShortCircuitCache.fromConf(scConf);
> }
> {code}
> 2 Then divide blocks by caches:
> {code:java}
> public ShortCircuitCache getShortCircuitCache(long idx) {
> return shortCircuitCache[(int) (idx % clientShortCircuitNum)];
> }
> {code}
> 3. And how to call it:
> {code:java}
> ShortCircuitCache cache =
> clientContext.getShortCircuitCache(block.getBlockId());
> {code}
> The last number of offset evenly distributed from 0 to 9 - that's why all
> caches will full approximately the same.
> It is good for performance. Below the attachment, it is load test reading
> HDFS via HBase where clientShortCircuitNum = 1 vs 3. We can see that
> performance grows ~30%, CPU usage about +15%.
> Hope it is interesting for someone.
> Ready to explain some unobvious things.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059983#comment-17059983 ] Lisheng Sun commented on HDFS-14919: Thank [~ayushtkn] for your reminder. I changed dfshealth.js too and upload the v003 patch. > Provide Non DFS Used per DataNode in DataNode UI > > > Key: HDFS-14919 > URL: https://issues.apache.org/jira/browse/HDFS-14919 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch, > HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14919) Provide Non DFS Used per DataNode in DataNode UI
[
https://issues.apache.org/jira/browse/HDFS-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059981#comment-17059981
]
Hadoop QA commented on HDFS-14919:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
38m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HDFS-14919 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12996801/HDFS-14919.003.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux db4497264847 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ea68863 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 470 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/28960/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Provide Non DFS Used per DataNode in DataNode UI
>
>
> Key: HDFS-14919
> URL: https://issues.apache.org/jira/browse/HDFS-14919
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14919.001.patch, HDFS-14919.002.patch,
> HDFS-14919.003.patch, hadoop2.6_datanode_ui.png, hadoop3.1_datanode_ui.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15226) Ranger integrates HDFS and discovers NPE
bianqi created HDFS-15226:
-
Summary: Ranger integrates HDFS and discovers NPE
Key: HDFS-15226
URL: https://issues.apache.org/jira/browse/HDFS-15226
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 2.7.6
Environment: Apache Ranger1.2 && Hadoop2.7.6
Reporter: bianqi
Fix For: 3.2.1, 3.2.0
Attachments: image-2020-03-16-14-01-03-078.png
When I integrated ranger1.2 with Hadoop2.7.6, the following NPE error
occurred when executing hdfs dfs -ls /.
However, when I integrated ranger1.2 with Hadoop2.7.1, executing hdfs dfs
-ls / without any errors, and the directory list can be displayed normally.
{quote}java.lang.NullPointerException
at java.lang.String.checkBounds(String.java:384)
at java.lang.String.(String.java:425)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:337)
at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.getINodeAttrs(FSPermissionChecker.java:238)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:183)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3832)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)
DEBUG org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020: responding to
org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from xx:8502
Call#0 Retry#0{quote}
When I checked the HDFS source code, I compared hadoop2.7.1 and
hadoop2.7.6 and found that 2.7.6 added the following methods
!image-2020-03-16-14-01-03-078.png|thumbnail!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
