[jira] [Commented] (HDFS-15566) NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0
[
https://issues.apache.org/jira/browse/HDFS-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195882#comment-17195882
]
Hadoop QA commented on HDFS-15566:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 38m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
16s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 332 unchanged - 0 fixed = 337 total (was 332) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 30s{color}
|
[jira] [Updated] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Boyina updated HDFS-15574:
--
Fix Version/s: 3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Remove unnecessary sort of block list in DirectoryScanner
> -
>
> Key: HDFS-15574
> URL: https://issues.apache.org/jira/browse/HDFS-15574
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HDFS-15574.001.patch, HDFS-15574.002.patch,
> HDFS-15574.003.patch
>
>
> These lines of code in DirectoryScanner#scan(), obtain a snapshot of the
> finalized blocks from memory, and then sort them, under the DN lock. However
> the blocks are stored in a sorted structure (FoldedTreeSet) and hence the
> sort should be unnecessary.
> {code}
> final List bl = dataset.getFinalizedBlocks(bpid);
> Collections.sort(bl); // Sort based on blockId
> {code}
> This Jira removes the sort, and renames the getFinalizedBlocks to
> getSortedFinalizedBlocks to make the intent of the method more clear.
> Also added a test, just in case the underlying block structure is ever
> changed to something unsorted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195871#comment-17195871
]
Hemanth Boyina commented on HDFS-15574:
---
committed to trunk
thanks for contribution [~sodonnell] , thanks for the review [~liuml07]
> Remove unnecessary sort of block list in DirectoryScanner
> -
>
> Key: HDFS-15574
> URL: https://issues.apache.org/jira/browse/HDFS-15574
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15574.001.patch, HDFS-15574.002.patch,
> HDFS-15574.003.patch
>
>
> These lines of code in DirectoryScanner#scan(), obtain a snapshot of the
> finalized blocks from memory, and then sort them, under the DN lock. However
> the blocks are stored in a sorted structure (FoldedTreeSet) and hence the
> sort should be unnecessary.
> {code}
> final List bl = dataset.getFinalizedBlocks(bpid);
> Collections.sort(bl); // Sort based on blockId
> {code}
> This Jira removes the sort, and renames the getFinalizedBlocks to
> getSortedFinalizedBlocks to make the intent of the method more clear.
> Also added a test, just in case the underlying block structure is ever
> changed to something unsorted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195868#comment-17195868
]
Hemanth Boyina commented on HDFS-15574:
---
+1 , will commit shortly
> Remove unnecessary sort of block list in DirectoryScanner
> -
>
> Key: HDFS-15574
> URL: https://issues.apache.org/jira/browse/HDFS-15574
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15574.001.patch, HDFS-15574.002.patch,
> HDFS-15574.003.patch
>
>
> These lines of code in DirectoryScanner#scan(), obtain a snapshot of the
> finalized blocks from memory, and then sort them, under the DN lock. However
> the blocks are stored in a sorted structure (FoldedTreeSet) and hence the
> sort should be unnecessary.
> {code}
> final List bl = dataset.getFinalizedBlocks(bpid);
> Collections.sort(bl); // Sort based on blockId
> {code}
> This Jira removes the sort, and renames the getFinalizedBlocks to
> getSortedFinalizedBlocks to make the intent of the method more clear.
> Also added a test, just in case the underlying block structure is ever
> changed to something unsorted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15559:
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks [~leosun08] for your report and contributions!
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: HDFS-15559.001.patch, HDFS-15559.002.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195812#comment-17195812
]
Lisheng Sun commented on HDFS-15559:
Hi [~hexiaoqiao]
I confirm the failed UTs are ok at local. And they are not related to this
patch. Thank you.
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch, HDFS-15559.002.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15570) Hadoop Erasure Coding ISA-L Check Request..
[ https://issues.apache.org/jira/browse/HDFS-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] isurika updated HDFS-15570: --- Issue Type: Bug (was: Task) > Hadoop Erasure Coding ISA-L Check Request.. > --- > > Key: HDFS-15570 > URL: https://issues.apache.org/jira/browse/HDFS-15570 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, hadoop-client >Affects Versions: 3.2.1 > Environment: *-OS Version* : CentOS Linux release 8.0.1905 (Core) > *-Hadoop Version* : hadoop-3.2.1 > >Reporter: isurika >Priority: Major > Fix For: 3.2.1 > > > I am testing the performance of erasure coding in Hadoop 3.2.1 version > environment. > Apply ISA-L by referring to the manual > ([https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]) > *▶ JOB LIST (Proceed in manual order)* > # ISA-L library installed on all servers > ld ISA-L library : [https://github.com/01org/isa-l] > # hadoop build source (hadoop-3.2.1-src.tar.gz / mvn package -Pdist,native > -Dtar -Drequire.isal -Drequire.snappy) > # Deploy the built hadoop-3.2.1 folder to all servers > But In Test (File upload Time/Select Orc table in Hive) > There is no improvement in performance when tested. > Ask for advice on work > *[Question 1]* > Is there anything wrong with the installation? Are there any missing tasks? > *[Question 2]* > Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) > *[Question 3]* > Whether to use ISA-L in Hadoop, How to check? > ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. > But no > ※ The warn log that occurs before applying ISA-L is It no longer occurs. > -- > WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is > not available in your platform... using builtin-java codec where applicable > -- > > *▶Reference information* > === > *-CPU INFO (namenode :2 / datanode : 5)* > - > namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz > datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz > - > *-hadoop checknative (After source build)* > - > Native library checking: > hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 > zlib: true /lib64/libz.so.1 > zstd : false > snappy: true /lib64/libsnappy.so.1 > lz4: true revision:10301 > bzip2: false > openssl: true /lib64/libcrypto.so > *ISA-L: true /lib64/libisal.so.2* > - > *-hdfs ec -listCodecs* > Erasure Coding Codecs: Codec [Coder List] > RS [RS_NATIVE, RS_JAVA] > RS-LEGACY [RS-LEGACY_JAVA] > XOR [XOR_NATIVE, XOR_JAVA] > === > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?focusedWorklogId=484223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484223
]
ASF GitHub Bot logged work on HDFS-15492:
-
Author: ASF GitHub Bot
Created on: 14/Sep/20 22:07
Start Date: 14/Sep/20 22:07
Worklog Time Spent: 10m
Work Description: smengcl edited a comment on pull request #2176:
URL: https://github.com/apache/hadoop/pull/2176#issuecomment-692330196
> I'm late to this. But we should verify to make sure httpfs also behaves
the sames. Otherwise it'll break Hue.
hey @jojochuang, thanks for checking in!
I have written a simple test for verifying HttpFS (over WebHDFS). HttpFS
works as expected. But I haven't figured out how to add a config to the HDFS
cluster that `BaseTestHttpFSWith` launches. Currently the config change for
this test is a bit hacky. Ergo, I haven't submitted a patch for new jira to add
this test. I would file a new jira to add it once this little problem is solved.
[HDFS-15492.TestHttpFS.001.patch.txt](https://github.com/apache/hadoop/files/5221189/HDFS-15492.TestHttpFS.001.patch.txt)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484223)
Time Spent: 0.5h (was: 20m)
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enabled}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?focusedWorklogId=484216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484216
]
ASF GitHub Bot logged work on HDFS-15492:
-
Author: ASF GitHub Bot
Created on: 14/Sep/20 21:43
Start Date: 14/Sep/20 21:43
Worklog Time Spent: 10m
Work Description: smengcl commented on pull request #2176:
URL: https://github.com/apache/hadoop/pull/2176#issuecomment-692330196
> I'm late to this. But we should verify to make sure httpfs also behaves
the sames. Otherwise it'll break Hue.
hey @jojochuang, thanks for checking in!
I have written a simple test for verifying HttpFS (over WebHDFS). HttpFS
works as expected. But I haven't figured out how to add a config to the HDFS
cluster that `BaseTestHttpFSWith` launches.
Currently the config change is a bit hacky. Ergo, I haven't submitted a
patch for new jira to add this test. I would file a new jira to add it once
this little problem is solved.
[HDFS-15492.TestHttpFS.001.patch.txt](https://github.com/apache/hadoop/files/5221189/HDFS-15492.TestHttpFS.001.patch.txt)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484216)
Time Spent: 20m (was: 10m)
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enabled}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=484201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484201
]
ASF GitHub Bot logged work on HDFS-15539:
-
Author: ASF GitHub Bot
Created on: 14/Sep/20 20:31
Start Date: 14/Sep/20 20:31
Worklog Time Spent: 10m
Work Description: smengcl merged pull request #2258:
URL: https://github.com/apache/hadoop/pull/2258
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484201)
Time Spent: 2h 40m (was: 2.5h)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195711#comment-17195711
]
Stephen O'Donnell commented on HDFS-15574:
--
[~liuml07] Thanks for the review.
[~hemanthboyina] I plan to commit this tomorrow unless you have any more
concerns?
> Remove unnecessary sort of block list in DirectoryScanner
> -
>
> Key: HDFS-15574
> URL: https://issues.apache.org/jira/browse/HDFS-15574
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15574.001.patch, HDFS-15574.002.patch,
> HDFS-15574.003.patch
>
>
> These lines of code in DirectoryScanner#scan(), obtain a snapshot of the
> finalized blocks from memory, and then sort them, under the DN lock. However
> the blocks are stored in a sorted structure (FoldedTreeSet) and hence the
> sort should be unnecessary.
> {code}
> final List bl = dataset.getFinalizedBlocks(bpid);
> Collections.sort(bl); // Sort based on blockId
> {code}
> This Jira removes the sort, and renames the getFinalizedBlocks to
> getSortedFinalizedBlocks to make the intent of the method more clear.
> Also added a test, just in case the underlying block structure is ever
> changed to something unsorted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195661#comment-17195661
]
Mingliang Liu commented on HDFS-15574:
--
V3 patch looks good to me. Thanks,
> Remove unnecessary sort of block list in DirectoryScanner
> -
>
> Key: HDFS-15574
> URL: https://issues.apache.org/jira/browse/HDFS-15574
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15574.001.patch, HDFS-15574.002.patch,
> HDFS-15574.003.patch
>
>
> These lines of code in DirectoryScanner#scan(), obtain a snapshot of the
> finalized blocks from memory, and then sort them, under the DN lock. However
> the blocks are stored in a sorted structure (FoldedTreeSet) and hence the
> sort should be unnecessary.
> {code}
> final List bl = dataset.getFinalizedBlocks(bpid);
> Collections.sort(bl); // Sort based on blockId
> {code}
> This Jira removes the sort, and renames the getFinalizedBlocks to
> getSortedFinalizedBlocks to make the intent of the method more clear.
> Also added a test, just in case the underlying block structure is ever
> changed to something unsorted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[
https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=484140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484140
]
ASF GitHub Bot logged work on HDFS-15568:
-
Author: ASF GitHub Bot
Created on: 14/Sep/20 17:39
Start Date: 14/Sep/20 17:39
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2296:
URL: https://github.com/apache/hadoop/pull/2296#discussion_r488109432
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
##
@@ -133,4 +138,57 @@ public void testValidateSnapshotIDWidth() throws Exception
{
getMaxSnapshotID() < Snapshot.CURRENT_STATE_ID);
}
+ @Test
+ public void testSnapshotLimitOnRestart() throws Exception {
+final Configuration conf = new Configuration();
+final Path snapshottableDir
+= new Path("/" + getClass().getSimpleName());
+int numSnapshots = 5;
+conf.setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, numSnapshots);
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT,
+numSnapshots * 2);
+MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(0).build();
+cluster.waitActive();
+DistributedFileSystem hdfs = cluster.getFileSystem();
+hdfs.mkdirs(snapshottableDir);
+hdfs.allowSnapshot(snapshottableDir);
+for (int i = 0; i < numSnapshots; i++) {
+ hdfs.createSnapshot(snapshottableDir, "s" + i);
+}
+LambdaTestUtils.intercept(SnapshotException.class,
+"snapshot limit",
+() -> hdfs.createSnapshot(snapshottableDir, "s5"));
+
+// now change max snapshot directory limit to 2 and restart namenode
+cluster.getNameNode().getConf().setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, 2);
+cluster.restartNameNodes();
+SnapshotManager snapshotManager = cluster.getNamesystem().
+getSnapshotManager();
+
+// make sure edits of all previous 5 create snapshots are replayed
+Assert.assertEquals(numSnapshots, snapshotManager.getNumSnapshots());
+
+// make sure namenode has the new snapshot limit configured as 2
+Assert.assertEquals(2, snapshotManager.getMaxSnapshotLimit());
+
+// Any new snapshot creation should still fail
+LambdaTestUtils.intercept(SnapshotException.class,
+"snapshot limit", () -> hdfs.createSnapshot(snapshottableDir, "s5"));
+// now change max snapshot FS limit to 2 and restart namenode
+cluster.getNameNode().getConf().setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT, 2);
+cluster.restartNameNodes();
+snapshotManager = cluster.getNamesystem().
+getSnapshotManager();
+// make sure edits of all previous 5 create snapshots are replayed
+Assert.assertEquals(numSnapshots, snapshotManager.getNumSnapshots());
+
+// make sure namenode has the new snapshot limit configured as 2
+Assert.assertEquals(2, snapshotManager.getMaxSnapshotLimit());
+cluster.shutdown();
Review comment:
We may need to do it in a finally to make sure we always clean it. Also
check for null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484140)
Time Spent: 2h 40m (was: 2.5h)
> namenode start failed to start when dfs.namenode.snapshot.max.limit set
> ---
>
> Key: HDFS-15568
> URL: https://issues.apache.org/jira/browse/HDFS-15568
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> {code:java}
> 11:35:05.872 AM ERROR NameNode
> Failed to start namenode.
> org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot:
> there are already 20 snapshot(s) and the max snapshot limit is 20
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285)
> at
> org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447)
> at
>
[jira] [Commented] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195487#comment-17195487
]
Xiaoqiao He commented on HDFS-15559:
[~leosun08] Try to run failed unit tests at local, most of them are passed. It
seems not related. Please have another checks.
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch, HDFS-15559.002.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15577) Refactor TestTracing
[ https://issues.apache.org/jira/browse/HDFS-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-15577: - Component/s: test > Refactor TestTracing > > > Key: HDFS-15577 > URL: https://issues.apache.org/jira/browse/HDFS-15577 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15576) Erasure Coding: Add test addPolicies to ECAdmin
[ https://issues.apache.org/jira/browse/HDFS-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-15576: --- Summary: Erasure Coding: Add test addPolicies to ECAdmin (was: Add test addPolicies to ECAdmin) > Erasure Coding: Add test addPolicies to ECAdmin > --- > > Key: HDFS-15576 > URL: https://issues.apache.org/jira/browse/HDFS-15576 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Minor > Attachments: HDFS-15576.001.patch > > > * Add UT TestECAdmin#testAddPolicies > * Modify UT, change "RS" to "rs", because "RS" is not supported -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15577) Refactor TestTracing
[ https://issues.apache.org/jira/browse/HDFS-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-15577: Status: Patch Available (was: Open) > Refactor TestTracing > > > Key: HDFS-15577 > URL: https://issues.apache.org/jira/browse/HDFS-15577 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15577) Refactor TestTracing
[ https://issues.apache.org/jira/browse/HDFS-15577?focusedWorklogId=483911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483911 ] ASF GitHub Bot logged work on HDFS-15577: - Author: ASF GitHub Bot Created on: 14/Sep/20 10:24 Start Date: 14/Sep/20 10:24 Worklog Time Spent: 10m Work Description: tasanuma opened a new pull request #2302: URL: https://github.com/apache/hadoop/pull/2302 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 483911) Remaining Estimate: 0h Time Spent: 10m > Refactor TestTracing > > > Key: HDFS-15577 > URL: https://issues.apache.org/jira/browse/HDFS-15577 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15577) Refactor TestTracing
[ https://issues.apache.org/jira/browse/HDFS-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15577: -- Labels: pull-request-available (was: ) > Refactor TestTracing > > > Key: HDFS-15577 > URL: https://issues.apache.org/jira/browse/HDFS-15577 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15577) Refactor TestTracing
Takanobu Asanuma created HDFS-15577: --- Summary: Refactor TestTracing Key: HDFS-15577 URL: https://issues.apache.org/jira/browse/HDFS-15577 Project: Hadoop HDFS Issue Type: Improvement Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15574) Remove unnecessary sort of block list in DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195342#comment-17195342
]
Stephen O'Donnell commented on HDFS-15574:
--
Ran all the failing test classes and they all passed locally.
[~hemanthboyina] and [~liuml07] are you happy with the v003 patch to commit it?
> Remove unnecessary sort of block list in DirectoryScanner
> -
>
> Key: HDFS-15574
> URL: https://issues.apache.org/jira/browse/HDFS-15574
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15574.001.patch, HDFS-15574.002.patch,
> HDFS-15574.003.patch
>
>
> These lines of code in DirectoryScanner#scan(), obtain a snapshot of the
> finalized blocks from memory, and then sort them, under the DN lock. However
> the blocks are stored in a sorted structure (FoldedTreeSet) and hence the
> sort should be unnecessary.
> {code}
> final List bl = dataset.getFinalizedBlocks(bpid);
> Collections.sort(bl); // Sort based on blockId
> {code}
> This Jira removes the sort, and renames the getFinalizedBlocks to
> getSortedFinalizedBlocks to make the intent of the method more clear.
> Also added a test, just in case the underlying block structure is ever
> changed to something unsorted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15576) Add test addPolicies to ECAdmin
[ https://issues.apache.org/jira/browse/HDFS-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-15576: --- Attachment: HDFS-15576.001.patch > Add test addPolicies to ECAdmin > --- > > Key: HDFS-15576 > URL: https://issues.apache.org/jira/browse/HDFS-15576 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Minor > Attachments: HDFS-15576.001.patch > > > * Add UT TestECAdmin#testAddPolicies > * Modify UT, change "RS" to "rs", because "RS" is not supported -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15576) Add test addPolicies to ECAdmin
[ https://issues.apache.org/jira/browse/HDFS-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HDFS-15576: --- Status: Patch Available (was: Open) > Add test addPolicies to ECAdmin > --- > > Key: HDFS-15576 > URL: https://issues.apache.org/jira/browse/HDFS-15576 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Minor > Attachments: HDFS-15576.001.patch > > > * Add UT TestECAdmin#testAddPolicies > * Modify UT, change "RS" to "rs", because "RS" is not supported -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15576) Add test addPolicies to ECAdmin
Fei Hui created HDFS-15576: -- Summary: Add test addPolicies to ECAdmin Key: HDFS-15576 URL: https://issues.apache.org/jira/browse/HDFS-15576 Project: Hadoop HDFS Issue Type: Test Reporter: Fei Hui Assignee: Fei Hui * Add UT TestECAdmin#testAddPolicies * Modify UT, change "RS" to "rs", because "RS" is not supported -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15382) Split FsDatasetImpl from blockpool lock to blockpool volume lock
[ https://issues.apache.org/jira/browse/HDFS-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195317#comment-17195317 ] Jiang Xin commented on HDFS-15382: -- Thanks [~Aiphag0] for your proposal, seems it helps a lot on IO heavy DNs and we planned to do this recently. Would you submit a sample patch or push it forward? > Split FsDatasetImpl from blockpool lock to blockpool volume lock > - > > Key: HDFS-15382 > URL: https://issues.apache.org/jira/browse/HDFS-15382 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Aiphago >Assignee: Aiphago >Priority: Major > Fix For: 3.2.1 > > Attachments: image-2020-06-02-1.png, image-2020-06-03-1.png > > > In HDFS-15180 we split lock to blockpool grain size.But when one volume is in > heavy load and will block other request which in same blockpool but different > volume.So we split lock to two leval to avoid this happend.And to improve > datanode performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195302#comment-17195302 ] Hemanth Boyina commented on HDFS-15543: --- thanks for the comment [~elgoiri] HDFS-1 has been resolved now and [^HDFS-15543.003.patch] still applies to trunk as HDFS-1 has modified method in RouterRpcClient.java which this Jira has not changed of > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543.002.patch, > HDFS-15543.003.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15532:
---
Fix Version/s: 3.4.0
3.3.1
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
