[jira] [Updated] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
[
https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-13287:
-
Fix Version/s: (was: 3.1.3)
3.1.4
> TestINodeFile#testGetBlockType results in NPE when run alone
>
>
> Key: HDFS-13287
> URL: https://issues.apache.org/jira/browse/HDFS-13287
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Minor
> Fix For: 3.3.0, 3.1.4
>
> Attachments: HDFS-13287.01.patch
>
>
> When TestINodeFile#testGetBlockType is run by itself, it results in the
> following error:
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218
> s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
> [ERROR]
> testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile) Time
> elapsed: 0.023 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12862) CacheDirective becomes invalid when NN restart or failover
[
https://issues.apache.org/jira/browse/HDFS-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-12862:
-
Fix Version/s: (was: 3.2.1)
> CacheDirective becomes invalid when NN restart or failover
> --
>
> Key: HDFS-12862
> URL: https://issues.apache.org/jira/browse/HDFS-12862
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching, hdfs
>Affects Versions: 2.7.1
> Environment:
>Reporter: Wang XL
>Assignee: Wang XL
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: HDFS-12862-branch-2.7.1.001.patch,
> HDFS-12862-trunk.002.patch, HDFS-12862-trunk.003.patch,
> HDFS-12862-trunk.004.patch, HDFS-12862.005.patch, HDFS-12862.006.patch,
> HDFS-12862.007.patch, HDFS-12862.branch-3.1.patch
>
>
> The logic in FSNDNCacheOp#modifyCacheDirective is not correct. when modify
> cacheDirective,the expiration in directive may be a relative expiryTime, and
> EditLog will serial a relative expiry time.
> {code:java}
> // Some comments here
> static void modifyCacheDirective(
> FSNamesystem fsn, CacheManager cacheManager, CacheDirectiveInfo
> directive,
> EnumSet flags, boolean logRetryCache) throws IOException {
> final FSPermissionChecker pc = getFsPermissionChecker(fsn);
> cacheManager.modifyDirective(directive, pc, flags);
> fsn.getEditLog().logModifyCacheDirectiveInfo(directive, logRetryCache);
> }
> {code}
> But when SBN replay the log ,it will invoke
> FSImageSerialization#readCacheDirectiveInfo as a absolute expiryTime.It will
> result in the inconsistency .
> {code:java}
> public static CacheDirectiveInfo readCacheDirectiveInfo(DataInput in)
> throws IOException {
> CacheDirectiveInfo.Builder builder =
> new CacheDirectiveInfo.Builder();
> builder.setId(readLong(in));
> int flags = in.readInt();
> if ((flags & 0x1) != 0) {
> builder.setPath(new Path(readString(in)));
> }
> if ((flags & 0x2) != 0) {
> builder.setReplication(readShort(in));
> }
> if ((flags & 0x4) != 0) {
> builder.setPool(readString(in));
> }
> if ((flags & 0x8) != 0) {
> builder.setExpiration(
> CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)));
> }
> if ((flags & ~0xF) != 0) {
> throw new IOException("unknown flags set in " +
> "ModifyCacheDirectiveInfoOp: " + flags);
> }
> return builder.build();
> }
> {code}
> In other words， fsn.getEditLog().logModifyCacheDirectiveInfo(directive,
> logRetryCache) may serial a relative expiry time,But
> builder.setExpiration(CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)))
>read it as a absolute expiryTime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12862) CacheDirective becomes invalid when NN restart or failover
[
https://issues.apache.org/jira/browse/HDFS-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935543#comment-16935543
]
Rohith Sharma K S commented on HDFS-12862:
--
branch-3.2 points to 3.2.2 version.. Removed 3.2.1 as it got released. If it
fixed in branch-3.2, please update to 3.2.2
> CacheDirective becomes invalid when NN restart or failover
> --
>
> Key: HDFS-12862
> URL: https://issues.apache.org/jira/browse/HDFS-12862
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching, hdfs
>Affects Versions: 2.7.1
> Environment:
>Reporter: Wang XL
>Assignee: Wang XL
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: HDFS-12862-branch-2.7.1.001.patch,
> HDFS-12862-trunk.002.patch, HDFS-12862-trunk.003.patch,
> HDFS-12862-trunk.004.patch, HDFS-12862.005.patch, HDFS-12862.006.patch,
> HDFS-12862.007.patch, HDFS-12862.branch-3.1.patch
>
>
> The logic in FSNDNCacheOp#modifyCacheDirective is not correct. when modify
> cacheDirective,the expiration in directive may be a relative expiryTime, and
> EditLog will serial a relative expiry time.
> {code:java}
> // Some comments here
> static void modifyCacheDirective(
> FSNamesystem fsn, CacheManager cacheManager, CacheDirectiveInfo
> directive,
> EnumSet flags, boolean logRetryCache) throws IOException {
> final FSPermissionChecker pc = getFsPermissionChecker(fsn);
> cacheManager.modifyDirective(directive, pc, flags);
> fsn.getEditLog().logModifyCacheDirectiveInfo(directive, logRetryCache);
> }
> {code}
> But when SBN replay the log ,it will invoke
> FSImageSerialization#readCacheDirectiveInfo as a absolute expiryTime.It will
> result in the inconsistency .
> {code:java}
> public static CacheDirectiveInfo readCacheDirectiveInfo(DataInput in)
> throws IOException {
> CacheDirectiveInfo.Builder builder =
> new CacheDirectiveInfo.Builder();
> builder.setId(readLong(in));
> int flags = in.readInt();
> if ((flags & 0x1) != 0) {
> builder.setPath(new Path(readString(in)));
> }
> if ((flags & 0x2) != 0) {
> builder.setReplication(readShort(in));
> }
> if ((flags & 0x4) != 0) {
> builder.setPool(readString(in));
> }
> if ((flags & 0x8) != 0) {
> builder.setExpiration(
> CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)));
> }
> if ((flags & ~0xF) != 0) {
> throw new IOException("unknown flags set in " +
> "ModifyCacheDirectiveInfoOp: " + flags);
> }
> return builder.build();
> }
> {code}
> In other words， fsn.getEditLog().logModifyCacheDirectiveInfo(directive,
> logRetryCache) may serial a relative expiry time,But
> builder.setExpiration(CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)))
>read it as a absolute expiryTime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13732) ECAdmin should print the policy name when an EC policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935541#comment-16935541 ] Rohith Sharma K S commented on HDFS-13732: -- branch-3.2 points to 3.2.2 version.. Removed 3.2.1 as it got released. > ECAdmin should print the policy name when an EC policy is set > - > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
[
https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-13287:
-
Fix Version/s: (was: 3.2.1)
> TestINodeFile#testGetBlockType results in NPE when run alone
>
>
> Key: HDFS-13287
> URL: https://issues.apache.org/jira/browse/HDFS-13287
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Minor
> Fix For: 3.3.0, 3.1.3
>
> Attachments: HDFS-13287.01.patch
>
>
> When TestINodeFile#testGetBlockType is run by itself, it results in the
> following error:
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218
> s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
> [ERROR]
> testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile) Time
> elapsed: 0.023 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13732) ECAdmin should print the policy name when an EC policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935542#comment-16935542 ] Rohith Sharma K S commented on HDFS-13732: -- branch-3.2 points to 3.2.2 version.. Removed 3.2.1 as it got released. If it fixed in branch-3.2, please update to 3.2.2 > ECAdmin should print the policy name when an EC policy is set > - > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13732) ECAdmin should print the policy name when an EC policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated HDFS-13732: - Fix Version/s: (was: 3.2.1) > ECAdmin should print the policy name when an EC policy is set > - > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14509) DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-14509:
-
Target Version/s: 2.10.0, 3.3.0, 3.1.3, 3.2.2 (was: 2.10.0, 3.3.0, 3.2.1,
3.1.3)
> DN throws InvalidToken due to inequality of password when upgrade NN 2.x to
> 3.x
> ---
>
> Key: HDFS-14509
> URL: https://issues.apache.org/jira/browse/HDFS-14509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yuxuan Wang
>Priority: Blocker
> Attachments: HDFS-14509-001.patch
>
>
> According to the doc, if we want to upgrade cluster from 2.x to 3.x, we need
> upgrade NN first. And there will be a intermediate state that NN is 3.x and
> DN is 2.x. At that moment, if a client reads (or writes) a block, it will get
> a block token from NN and then deliver the token to DN who can verify the
> token. But the verification in the code now is :
> {code:title=BlockTokenSecretManager.java|borderStyle=solid}
> public void checkAccess(...)
> {
> ...
> id.readFields(new DataInputStream(new
> ByteArrayInputStream(token.getIdentifier(;
> ...
> if (!Arrays.equals(retrievePassword(id), token.getPassword())) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't have the correct token password");
> }
> }
> {code}
> And {{retrievePassword(id)}} is:
> {code}
> public byte[] retrievePassword(BlockTokenIdentifier identifier)
> {
> ...
> return createPassword(identifier.getBytes(), key.getKey());
> }
> {code}
> So, if NN's identifier add new fields, DN will lose the fields and compute
> wrong password.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14340) Lower the log level when can't get postOpAttr
[
https://issues.apache.org/jira/browse/HDFS-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925315#comment-16925315
]
Rohith Sharma K S commented on HDFS-14340:
--
Removed Flag which was set as *Important*!
> Lower the log level when can't get postOpAttr
> -
>
> Key: HDFS-14340
> URL: https://issues.apache.org/jira/browse/HDFS-14340
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Anuhan Torgonshar
>Assignee: Anuhan Torgonshar
>Priority: Minor
> Labels: easyfix
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14340.trunk.patch
>
>
> I think should lower the log level when can't get postOpAttr in
> _*hadoop-2.8.5-src/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/**RpcProgramNfs3.java*_.
>
>
> {code:java}
> **The fisrt code snippet***
> //the problematic log level ERROR, at line 1044
> try {
>dirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpDirAttr),
>dfsClient, dirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.error("Can't get postOpDirAttr for dirFileId: "
>+ dirHandle.getFileId(), e1);
> }
> **The second code snippet***
> //other practice in similar code snippets, line number is 475, the log
> assigned with INFO level
> try {
>wccData = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpAttr),
> dfsClient, fileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpAttr for fileIdPath: " + fileIdPath, e1);
> }
> **The third code snippet***
> //other practice in similar code snippets, line number is 1405, the log
> assigned with INFO level
> try {
>fromDirWcc = Nfs3Utils.createWccData(
>Nfs3Utils.getWccAttr(fromPreOpAttr), dfsClient, fromDirFileIdPath,iug);
>toDirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(toPreOpAttr),
>dfsClient, toDirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpDirAttr for " + fromDirFileIdPath + " or"
>+ toDirFileIdPath, e1);
> }
> {code}
> Therefore, I think the logging practices should be consistent in similar
> contexts. When the code catches _*IOException*_ for *_getWccAttr()_* method,
> it more likely prints a log message with _*INFO*_ level, a lower level.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14340) Lower the log level when can't get postOpAttr
[
https://issues.apache.org/jira/browse/HDFS-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-14340:
-
Hadoop Flags: Reviewed
> Lower the log level when can't get postOpAttr
> -
>
> Key: HDFS-14340
> URL: https://issues.apache.org/jira/browse/HDFS-14340
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Anuhan Torgonshar
>Assignee: Anuhan Torgonshar
>Priority: Minor
> Labels: easyfix
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14340.trunk.patch
>
>
> I think should lower the log level when can't get postOpAttr in
> _*hadoop-2.8.5-src/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/**RpcProgramNfs3.java*_.
>
>
> {code:java}
> **The fisrt code snippet***
> //the problematic log level ERROR, at line 1044
> try {
>dirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpDirAttr),
>dfsClient, dirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.error("Can't get postOpDirAttr for dirFileId: "
>+ dirHandle.getFileId(), e1);
> }
> **The second code snippet***
> //other practice in similar code snippets, line number is 475, the log
> assigned with INFO level
> try {
>wccData = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpAttr),
> dfsClient, fileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpAttr for fileIdPath: " + fileIdPath, e1);
> }
> **The third code snippet***
> //other practice in similar code snippets, line number is 1405, the log
> assigned with INFO level
> try {
>fromDirWcc = Nfs3Utils.createWccData(
>Nfs3Utils.getWccAttr(fromPreOpAttr), dfsClient, fromDirFileIdPath,iug);
>toDirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(toPreOpAttr),
>dfsClient, toDirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpDirAttr for " + fromDirFileIdPath + " or"
>+ toDirFileIdPath, e1);
> }
> {code}
> Therefore, I think the logging practices should be consistent in similar
> contexts. When the code catches _*IOException*_ for *_getWccAttr()_* method,
> it more likely prints a log message with _*INFO*_ level, a lower level.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14340) Lower the log level when can't get postOpAttr
[
https://issues.apache.org/jira/browse/HDFS-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-14340:
-
Flags: (was: Important)
> Lower the log level when can't get postOpAttr
> -
>
> Key: HDFS-14340
> URL: https://issues.apache.org/jira/browse/HDFS-14340
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Anuhan Torgonshar
>Assignee: Anuhan Torgonshar
>Priority: Minor
> Labels: easyfix
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14340.trunk.patch
>
>
> I think should lower the log level when can't get postOpAttr in
> _*hadoop-2.8.5-src/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/**RpcProgramNfs3.java*_.
>
>
> {code:java}
> **The fisrt code snippet***
> //the problematic log level ERROR, at line 1044
> try {
>dirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpDirAttr),
>dfsClient, dirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.error("Can't get postOpDirAttr for dirFileId: "
>+ dirHandle.getFileId(), e1);
> }
> **The second code snippet***
> //other practice in similar code snippets, line number is 475, the log
> assigned with INFO level
> try {
>wccData = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpAttr),
> dfsClient, fileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpAttr for fileIdPath: " + fileIdPath, e1);
> }
> **The third code snippet***
> //other practice in similar code snippets, line number is 1405, the log
> assigned with INFO level
> try {
>fromDirWcc = Nfs3Utils.createWccData(
>Nfs3Utils.getWccAttr(fromPreOpAttr), dfsClient, fromDirFileIdPath,iug);
>toDirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(toPreOpAttr),
>dfsClient, toDirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpDirAttr for " + fromDirFileIdPath + " or"
>+ toDirFileIdPath, e1);
> }
> {code}
> Therefore, I think the logging practices should be consistent in similar
> contexts. When the code catches _*IOException*_ for *_getWccAttr()_* method,
> it more likely prints a log message with _*INFO*_ level, a lower level.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14340) Lower the log level when can't get postOpAttr
[
https://issues.apache.org/jira/browse/HDFS-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925129#comment-16925129
]
Rohith Sharma K S commented on HDFS-14340:
--
[~jojochuang] [~OneisAll] [~starphin] Could anyone update the release note
since it is marked as *Important* in Flags for release 3.2.1? If it nothing
breaks backward compatible, better to remove the flag 'Important'.
> Lower the log level when can't get postOpAttr
> -
>
> Key: HDFS-14340
> URL: https://issues.apache.org/jira/browse/HDFS-14340
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Anuhan Torgonshar
>Assignee: Anuhan Torgonshar
>Priority: Major
> Labels: easyfix
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14340.trunk.patch
>
>
> I think should lower the log level when can't get postOpAttr in
> _*hadoop-2.8.5-src/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/**RpcProgramNfs3.java*_.
>
>
> {code:java}
> **The fisrt code snippet***
> //the problematic log level ERROR, at line 1044
> try {
>dirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpDirAttr),
>dfsClient, dirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.error("Can't get postOpDirAttr for dirFileId: "
>+ dirHandle.getFileId(), e1);
> }
> **The second code snippet***
> //other practice in similar code snippets, line number is 475, the log
> assigned with INFO level
> try {
>wccData = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(preOpAttr),
> dfsClient, fileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpAttr for fileIdPath: " + fileIdPath, e1);
> }
> **The third code snippet***
> //other practice in similar code snippets, line number is 1405, the log
> assigned with INFO level
> try {
>fromDirWcc = Nfs3Utils.createWccData(
>Nfs3Utils.getWccAttr(fromPreOpAttr), dfsClient, fromDirFileIdPath,iug);
>toDirWcc = Nfs3Utils.createWccData(Nfs3Utils.getWccAttr(toPreOpAttr),
>dfsClient, toDirFileIdPath, iug);
> } catch (IOException e1) {
>LOG.info("Can't get postOpDirAttr for " + fromDirFileIdPath + " or"
>+ toDirFileIdPath, e1);
> }
> {code}
> Therefore, I think the logging practices should be consistent in similar
> contexts. When the code catches _*IOException*_ for *_getWccAttr()_* method,
> it more likely prints a log message with _*INFO*_ level, a lower level.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14074) DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace.
[
https://issues.apache.org/jira/browse/HDFS-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925128#comment-16925128
]
Rohith Sharma K S commented on HDFS-14074:
--
[~jojochuang] [~luguangyi] [~arp] Could anyone update the release note since it
is marked as incompatible change for release 3.2.1?
> DataNode runs async disk checks maybe throws NullPointerException, and
> DataNode failed to register to NameSpace.
> --
>
> Key: HDFS-14074
> URL: https://issues.apache.org/jira/browse/HDFS-14074
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0, 3.0.0
> Environment: hadoop-2.7.3, hadoop-2.8.0
>Reporter: guangyi lu
>Assignee: guangyi lu
>Priority: Major
> Labels: HDFS, HDFS-4
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14074-latest.patch, HDFS-14074.patch,
> WechatIMG83.jpeg
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> In ThrottledAsyncChecker class，it members of the completedChecks is
> WeakHashMap, its definition is as follows：
> this.completedChecks = new WeakHashMap<>();
> and one of its uses is as follows in schedule method:
> if (completedChecks.containsKey(target)) {
> // here may be happen garbage collection，and result may be null.
> final LastCheckResult result = completedChecks.get(target);
>
> final long msSinceLastCheck = timer.monotonicNow() -
> result.completedAt;
>
> }
> after "completedChecks.containsKey(target)"， may be happen garbage
> collection， and result may be null.
> the solution is：
> this.completedChecks = new ReferenceMap(1, 1);
> or
> this.completedChecks = new HashMap<>();
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14018) Compilation fails in branch-3.0
[
https://issues.apache.org/jira/browse/HDFS-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-14018:
-
Affects Version/s: 3.0.4
> Compilation fails in branch-3.0
> ---
>
> Key: HDFS-14018
> URL: https://issues.apache.org/jira/browse/HDFS-14018
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.4
>Reporter: Rohith Sharma K S
>Priority: Blocker
>
> HDFS branch-3.0 compilation fails.
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-hdfs: Compilation failure
> [ERROR]
> /Users/rsharmaks/Repos/Apache/Commit_Repos/branch-3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java:[306,9]
> cannot find symbol
> [ERROR] symbol: variable ArrayUtils
> [ERROR] location: class
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14019) Update ATSv2 hbase.two.version to 2.0.2
Rohith Sharma K S created HDFS-14019: Summary: Update ATSv2 hbase.two.version to 2.0.2 Key: HDFS-14019 URL: https://issues.apache.org/jira/browse/HDFS-14019 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.2.0, 3.1.2, 3.3.0 Reporter: Rohith Sharma K S Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14018) Compilation fails in branch-3.0
[
https://issues.apache.org/jira/browse/HDFS-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-14018:
-
Summary: Compilation fails in branch-3.0 (was: Compilation falure in
branch-3.0)
> Compilation fails in branch-3.0
> ---
>
> Key: HDFS-14018
> URL: https://issues.apache.org/jira/browse/HDFS-14018
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Blocker
>
> HDFS branch-3.0 compilation fails.
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-hdfs: Compilation failure
> [ERROR]
> /Users/rsharmaks/Repos/Apache/Commit_Repos/branch-3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java:[306,9]
> cannot find symbol
> [ERROR] symbol: variable ArrayUtils
> [ERROR] location: class
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14018) Compilation falure in branch-3.0
Rohith Sharma K S created HDFS-14018:
Summary: Compilation falure in branch-3.0
Key: HDFS-14018
URL: https://issues.apache.org/jira/browse/HDFS-14018
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Rohith Sharma K S
HDFS branch-3.0 compilation fails.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-hdfs: Compilation failure
[ERROR]
/Users/rsharmaks/Repos/Apache/Commit_Repos/branch-3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java:[306,9]
cannot find symbol
[ERROR] symbol: variable ArrayUtils
[ERROR] location: class
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager
[ERROR]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14018) Compilation falure in branch-3.0
[
https://issues.apache.org/jira/browse/HDFS-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-14018:
-
Priority: Blocker (was: Major)
> Compilation falure in branch-3.0
>
>
> Key: HDFS-14018
> URL: https://issues.apache.org/jira/browse/HDFS-14018
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Blocker
>
> HDFS branch-3.0 compilation fails.
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-hdfs: Compilation failure
> [ERROR]
> /Users/rsharmaks/Repos/Apache/Commit_Repos/branch-3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java:[306,9]
> cannot find symbol
> [ERROR] symbol: variable ArrayUtils
> [ERROR] location: class
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HDFS-13596:
-
Priority: Blocker (was: Critical)
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Priority: Blocker
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resetLastInodeId(FSDirectory.java:1945)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at
[jira] [Commented] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510975#comment-16510975
]
Rohith Sharma K S commented on HDFS-13596:
--
Thanks [~hanishakoneru] for reporting this. I do see this error while rolling
upgrade testing Hadoop cluster. Especially creating a file while rolling
upgrade is in progress and finalizing the upgrade on upgraded version. This
blocks rolling upgrade functionality.
I am bumping the JIRA to blocker as it blocks rolling upgrade.
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Priority: Critical
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resetLastInodeId(FSDirectory.java:1945)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
>
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and nameservice quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471739#comment-16471739
]
Rohith Sharma K S commented on HDFS-13346:
--
This issue fix breaks compilation in branch-2. [~linyiqun] could you check this
please?
{code}
ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-hdfs-rbf: Compilation failure
[ERROR]
/Users/rsharmaks/Repos/Apache/Commit_Repos/branch-2/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java:[114,58]
incompatible types: java.lang.Object cannot be converted to
org.apache.hadoop.fs.StorageType
{code}
> RBF: Fix synchronization of router quota and nameservice quota
> --
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: liuhongtong
>Assignee: Yiqun Lin
>Priority: Major
> Labels: incompatible
> Fix For: 2.10.0, 3.2.0, 3.1.1
>
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch, HDFS-13346.004.patch, HDFS-13346.005.patch,
> HDFS-13346.006.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13242) HMaster start fails with IllegalStateException when HADOOP_HOME is set
Rohith Sharma K S created HDFS-13242:
Summary: HMaster start fails with IllegalStateException when
HADOOP_HOME is set
Key: HDFS-13242
URL: https://issues.apache.org/jira/browse/HDFS-13242
Project: Hadoop HDFS
Issue Type: Bug
Environment: HBase-2.0-beta1.
Hadoop trunk version.
java version "1.8.0_144"
Reporter: Rohith Sharma K S
It is observed that HMaster start is failed when HADOOP_HOME is set as env
while starting HMaster. HADOOP_HOME is pointing to Hadoop trunk version.
{noformat}
2018-03-07 16:59:52,654 ERROR [master//10.200.4.200:16000] master.HMaster:
Failed to become active master
java.lang.IllegalStateException: The procedure WAL relies on the ability to
hsync for proper operation during component failures, but the underlying
filesystem does not support doing so. Please check the config value of
'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness
and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount
that can provide it.
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
at
org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232)
at
org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
at java.lang.Thread.run(Thread.java:748)
{noformat}
The same configs is working in HBase-1.2.6 build properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-12403) The size of dataQueue and ackQueue in DataStreamer has no limit when writer thread is interrupted
[
https://issues.apache.org/jira/browse/HDFS-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S moved YARN-7168 to HDFS-12403:
Component/s: (was: client)
Key: HDFS-12403 (was: YARN-7168)
Project: Hadoop HDFS (was: Hadoop YARN)
> The size of dataQueue and ackQueue in DataStreamer has no limit when writer
> thread is interrupted
> -
>
> Key: HDFS-12403
> URL: https://issues.apache.org/jira/browse/HDFS-12403
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jiandan Yang
> Attachments: mat.jpg
>
>
> In our cluster, when found NodeManager frequently FullGC when decommissioning
> NodeManager, and we found the biggest object is dataQueue of DataStreamer, it
> has almost 6w DFSPacket, and every DFSPacket is about 64k, as shown below.
> The root reason is that the size of dataQueue and ackQueue in DataStreamer
> has no limit when writer thread is interrupted.
> DFSOutputStream#waitAndQueuePacket does not wait when writer thread is
> interrupted. I know NodeManager may stop writing when interruped, but
> DFSOutputStream also could do something to avoid Infinite growth of dataQueue.
> {code:java}
> while (!streamerClosed && dataQueue.size() + ackQueue.size() >
> dfsClient.getConf().getWriteMaxPackets()) {
> if (firstWait) {
> Span span = Tracer.getCurrentSpan();
> if (span != null) {
> span.addTimelineAnnotation("dataQueue.wait");
> }
> firstWait = false;
> }
> try {
> dataQueue.wait();
> } catch (InterruptedException e) {
> // If we get interrupted while waiting to queue data, we still
> need to get rid
> // of the current packet. This is because we have an invariant
> that if
> // currentPacket gets full, it will get queued before the next
> writeChunk.
> //
> // Rather than wait around for space in the queue, we should
> instead try to
> // return to the caller as soon as possible, even though we
> slightly overrun
> // the MAX_PACKETS length.
> Thread.currentThread().interrupt();
> break;
> }
> }
> } finally {
> Span span = Tracer.getCurrentSpan();
> if ((span != null) && (!firstWait)) {
> span.addTimelineAnnotation("end.wait");
> }
> }
> {code}
> !mat.jpg|memory_analysis!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8132) Namenode Startup Failing When we add Jcarder.jar in class Path
[
https://issues.apache.org/jira/browse/HDFS-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891673#comment-15891673
]
Rohith Sharma K S commented on HDFS-8132:
-
[~vijaykalluru] JCarder need not to update its build for attaching to Hadoop
version. You can download existing JCarder-2.0 build and attache to hadoop.
While attaching agent, make sure to add *-noverify* flag i.e {{-noverify
-javaagent:/jcarder.jar=outputdir=/jcarder/rm}}
> Namenode Startup Failing When we add Jcarder.jar in class Path
> --
>
> Key: HDFS-8132
> URL: https://issues.apache.org/jira/browse/HDFS-8132
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> *{color:blue}Namenode while Startup Args{color}* ( Just added the jcarder
> args)
> exec /home/hdfs/jdk1.7.0_72/bin/java -Dproc_namenode -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Dhadoop.log.dir=/opt/ClusterSetup/Hadoop2.7/install/hadoop/namenode/logs
> -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/opt/ClusterSetup/Hadoop2.7/install/hadoop/namenode
> -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/opt/ClusterSetup/Hadoop2.7/install/hadoop/namenode/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
> {color:red}-javaagent:/opt/Jcarder/jcarder.jar=outputdir=/opt/Jcarder/Output/nn-jcarder{color}
> -Dhadoop.security.logger=INFO,NullAppender
> org.apache.hadoop.hdfs.server.namenode.NameNode
> Setting outputdir to /opt/Jcarder/Output/nn-jcarder
> Starting JCarder (2.0.0/6) agent
> Opening for writing: /opt/Jcarder/Output/nn-jcarder/jcarder_events.db
> Opening for writing: /opt/Jcarder/Output/nn-jcarder/jcarder_contexts.db
> Not instrumenting standard library classes (AWT, Swing, etc.)
> JCarder agent initialized
> *{color:red}ERROR{color}*
> {noformat}
> Exception in thread "main" java.lang.VerifyError: Expecting a stackmap frame
> at branch target 21
> Exception Details:
> Location:
>
> org/apache/hadoop/hdfs/server/namenode/NameNode.createHAState(Lorg/apache/hadoop/hdfs/server/common/HdfsServerConstants$StartupOption;)Lorg/apache/hadoop/hdfs/server/namenode/ha/HAState;
> @4: ifeq
> Reason:
> Expected stackmap frame at this location.
> Bytecode:
> 000: 2ab4 02d2 9900 112b b203 08a5 000a 2bb2
> 010: 030b a600 07b2 030d b0b2 030f b0
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
> at java.lang.Class.getMethod0(Class.java:2856)
> at java.lang.Class.getMethod(Class.java:1668)
> at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
