[jira] [Commented] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178722#comment-17178722
]
Hadoop QA commented on HDFS-15533:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
0s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
14s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
43s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
55s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
32s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
35s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 18m 35s{color}
| {color:red} root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 13 new + 2043 unchanged
- 9 fixed = 2056 total (was 2052) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
38s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 38s{color}
| {color:red} root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with
JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 12 new +
1939 unchanged - 8 fixed = 1951 total (was 1947) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 42s{color} | {color:orange} root: The patch generated 2 new + 217 unchanged
- 0 fixed = 219 total (was 217) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
|
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178647#comment-17178647
]
Hadoop QA commented on HDFS-15535:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
14s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 37s{color}
| {color:red} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}
[jira] [Commented] (HDFS-15527) Error On adding new Namespace
[
https://issues.apache.org/jira/browse/HDFS-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178642#comment-17178642
]
Ayush Saxena commented on HDFS-15527:
-
{quote}The new name nodes never be part of existing name space
{quote}
Are you adding a namespace or a namenode to an existing namespace.
if namenode then did you do bootstrapStandby?
{{hdfs namenode -bootstrapStandby}}
Well this doesn't seems to be a bug, you need to follow the doc properly, if
things don't work, you can try getting help at hadoop user mailing list, with
details of your configurations and steps you followed
> Error On adding new Namespace
> -
>
> Key: HDFS-15527
> URL: https://issues.apache.org/jira/browse/HDFS-15527
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation, ha, nn
>Affects Versions: 3.0.0
>Reporter: Thangamani Murugasamy
>Priority: Blocker
>
> We have one namespace, trying to add other one, always getting below error
> message.
>
> The new name nodes never be part of existing name space, also don't see any
> "nn" directories before adding name space.
>
> 2020-08-12 04:59:53,947 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: NameNode is not formatted.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1084)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:709)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:950)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:929)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
> 2020-08-12 04:59:53,955 DEBUG
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Closing log when already
> closed
> ==
>
>
> 2020-08-12 04:59:53,976 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.io.IOException: NameNode is not formatted.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1084)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:709)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:950)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:929)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
> 2020-08-12 04:59:53,978 DEBUG org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: java.io.IOException: NameNode is not formatted.
> 1: java.io.IOException: NameNode is not formatted.
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:265)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1726)
> Caused by: java.io.IOException: NameNode is not formatted.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:237)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1084)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:709)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:950)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:929)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
> 2020-08-12 04:59:53,979 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: java.io.IOException: NameNode is not formatted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Status: Patch Available (was: Open)
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-01.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
Ayush Saxena created HDFS-15535:
---
Summary: RBF: Fix Namespace path to snapshot path resolution for
snapshot API
Key: HDFS-15535
URL: https://issues.apache.org/jira/browse/HDFS-15535
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ayush Saxena
Assignee: Ayush Saxena
Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
namespace path is replaced with mount path.
This presumes as of now that, the invokedLocation shall always be the first
one in the sequence, but there are multiple reasons, where the directory might
not be in the first location.
So, rather than replacing using firstLocation, we should replace path using
actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11187) Optimize disk access for last partial chunk checksum of Finalized replica
[
https://issues.apache.org/jira/browse/HDFS-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178491#comment-17178491
]
Jiang Xin edited comment on HDFS-11187 at 8/16/20, 2:27 PM:
Hi [~weichiu] , thanks for your patch, but I don't think it handles all
scenario, please correct me if I misunderstood.
Consider such case: A reader thread reads a finalized replica, which have not
load the LPCC in memory, so it's LPCC is null, then it comes to
getPartialChunkChecksumForFinalized() in BlockSender#init().
{code:java}
private ChunkChecksum getPartialChunkChecksumForFinalized(
FinalizedReplica finalized) throws IOException {
...
final long replicaVisibleLength = replica.getVisibleLength();
if (replicaVisibleLength % CHUNK_SIZE != 0 &&
finalized.getLastPartialChunkChecksum() == null) {
//
// reader thread blocks here, and another writer thread append this
replica and succeed.
//
// the finalized replica does not have precomputed last partial
// chunk checksum. Recompute now.
try {
finalized.loadLastPartialChunkChecksum();
return new ChunkChecksum(finalized.getVisibleLength(),
finalized.getLastPartialChunkChecksum());
} catch (FileNotFoundException e) {
...
}
{code}
At the same time, another thread append this replica and succeed. So
getPartialChunkChecksumForFinalized recompute the LPCC and set to the replica.
In this situation, the `finalized` object is old, and it's visibleLength is old
,but after loadLastPartialChunkChecksum(), it's LPCC is new, so the mismatch
happened.
I'm not sure if we need to worry about it, any advise?
was (Author: jiang xin):
Hi [~weichiu] , thanks for your patch, but I don't think it handles all
scenario, please correct me if I misunderstood.
Consider such case: A reader thread reads a finalized replica, which have not
load the LPCC in memory, so it's LPCC is null, then it comes to
getPartialChunkChecksumForFinalized() in BlockSender#init().
{code:java}
private ChunkChecksum getPartialChunkChecksumForFinalized(
FinalizedReplica finalized) throws IOException {
...
final long replicaVisibleLength = replica.getVisibleLength();
if (replicaVisibleLength % CHUNK_SIZE != 0 &&
finalized.getLastPartialChunkChecksum() == null) {
//
// reader thread blocks here, and another writer thread append this
replica and succeed.
//
// the finalized replica does not have precomputed last partial
// chunk checksum. Recompute now.
try {
finalized.loadLastPartialChunkChecksum();
return new ChunkChecksum(finalized.getVisibleLength(),
finalized.getLastPartialChunkChecksum());
} catch (FileNotFoundException e) {
...
}
{code}
At the same time, another thread append this replica and succeed. So
getPartialChunkChecksumForFinalized recompute the LPCC and set to the replica.
In this situation, the `finalized` object is old, and it's visibleLength is old
,but after loadLastPartialChunkChecksum(), it's LPCC is new, so the mismatch
happened.
I'm not sure if we need to worry about it, any advise?
> Optimize disk access for last partial chunk checksum of Finalized replica
> -
>
> Key: HDFS-11187
> URL: https://issues.apache.org/jira/browse/HDFS-11187
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.0.3
>
> Attachments: HDFS-11187-branch-2.001.patch,
> HDFS-11187-branch-2.002.patch, HDFS-11187-branch-2.003.patch,
> HDFS-11187-branch-2.004.patch, HDFS-11187-branch-2.7.001.patch,
> HDFS-11187.001.patch, HDFS-11187.002.patch, HDFS-11187.003.patch,
> HDFS-11187.004.patch, HDFS-11187.005.patch
>
>
> The patch at HDFS-11160 ensures BlockSender reads the correct version of
> metafile when there are concurrent writers.
> However, the implementation is not optimal, because it must always read the
> last partial chunk checksum from disk while holding FsDatasetImpl lock for
> every reader. It is possible to optimize this by keeping an up-to-date
> version of last partial checksum in-memory and reduce disk access.
> I am separating the optimization into a new jira, because maintaining the
> state of in-memory checksum requires a lot more work.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11187) Optimize disk access for last partial chunk checksum of Finalized replica
[
https://issues.apache.org/jira/browse/HDFS-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178491#comment-17178491
]
Jiang Xin commented on HDFS-11187:
--
Hi [~weichiu] , thanks for your patch, but I don't think it handles all
scenario, please correct me if I misunderstood.
Consider such case: A reader thread reads a finalized replica, which have not
load the LPCC in memory, so it's LPCC is null, then it comes to
getPartialChunkChecksumForFinalized() in BlockSender#init().
{code:java}
private ChunkChecksum getPartialChunkChecksumForFinalized(
FinalizedReplica finalized) throws IOException {
...
final long replicaVisibleLength = replica.getVisibleLength();
if (replicaVisibleLength % CHUNK_SIZE != 0 &&
finalized.getLastPartialChunkChecksum() == null) {
//
// reader thread blocks here, and another writer thread append this
replica and succeed.
//
// the finalized replica does not have precomputed last partial
// chunk checksum. Recompute now.
try {
finalized.loadLastPartialChunkChecksum();
return new ChunkChecksum(finalized.getVisibleLength(),
finalized.getLastPartialChunkChecksum());
} catch (FileNotFoundException e) {
...
}
{code}
At the same time, another thread append this replica and succeed. So
getPartialChunkChecksumForFinalized recompute the LPCC and set to the replica.
In this situation, the `finalized` object is old, and it's visibleLength is old
,but after loadLastPartialChunkChecksum(), it's LPCC is new, so the mismatch
happened.
I'm not sure if we need to worry about it, any advise?
> Optimize disk access for last partial chunk checksum of Finalized replica
> -
>
> Key: HDFS-11187
> URL: https://issues.apache.org/jira/browse/HDFS-11187
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.0.3
>
> Attachments: HDFS-11187-branch-2.001.patch,
> HDFS-11187-branch-2.002.patch, HDFS-11187-branch-2.003.patch,
> HDFS-11187-branch-2.004.patch, HDFS-11187-branch-2.7.001.patch,
> HDFS-11187.001.patch, HDFS-11187.002.patch, HDFS-11187.003.patch,
> HDFS-11187.004.patch, HDFS-11187.005.patch
>
>
> The patch at HDFS-11160 ensures BlockSender reads the correct version of
> metafile when there are concurrent writers.
> However, the implementation is not optimal, because it must always read the
> last partial chunk checksum from disk while holding FsDatasetImpl lock for
> every reader. It is possible to optimize this by keeping an up-to-date
> version of last partial checksum in-memory and reduce disk access.
> I am separating the optimization into a new jira, because maintaining the
> state of in-memory checksum requires a lot more work.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
