[jira] [Commented] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942840#comment-13942840
]
Hadoop QA commented on HDFS-6130:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635948/HDFS-6130.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6462//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6462//console
This message is automatically generated.
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
[jira] [Commented] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942835#comment-13942835 ] Jing Zhao commented on HDFS-5138: - Created HDFS-6135 for the issue. Also tried to use a unit test to reproduce the issue there. > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Assignee: Aaron T. Myers >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > hdfs-5138-branch-2.txt > > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6135) In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back
[
https://issues.apache.org/jira/browse/HDFS-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6135:
Attachment: HDFS-6135.test.txt
The attached patch mimics the LV change while upgrade and can reproduce the
exception on a unit test.
> In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump
> when rolling back
> --
>
> Key: HDFS-6135
> URL: https://issues.apache.org/jira/browse/HDFS-6135
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Jing Zhao
> Attachments: HDFS-6135.test.txt
>
>
> While doing HDFS upgrade with HA setup, if the layoutversion gets changed in
> the upgrade, the rollback may trigger the following exception in JournalNodes
> (suppose the new software bumped the layoutversion from -55 to -56):
> {code}
> 14/03/21 01:01:53 FATAL namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not check if
> roll back possible for one or more JournalNodes. 1 exceptions thrown:
> Unexpected version of storage directory /grid/1/tmp/journal/mycluster.
> Reported: -56. Expecting = -55.
> at
> org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:203)
> at
> org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:156)
> at
> org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:135)
> at
> org.apache.hadoop.hdfs.qjournal.server.JNStorage.analyzeStorage(JNStorage.java:202)
> at
> org.apache.hadoop.hdfs.qjournal.server.JNStorage.(JNStorage.java:73)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.(Journal.java:142)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:87)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNode.canRollBack(JournalNode.java:304)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.canRollBack(JournalNodeRpcServer.java:228)
> {code}
> Looks like for rollback JN with old software cannot handle future
> layoutversion brought by new software.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: HDFSDataAtRestEncryption.pdf This proposal (PDF attached) discusses 4 possible designs for HDFS file encryption. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6135) In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back
Jing Zhao created HDFS-6135:
---
Summary: In HDFS upgrade with HA setup, JournalNode cannot handle
layout version bump when rolling back
Key: HDFS-6135
URL: https://issues.apache.org/jira/browse/HDFS-6135
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Jing Zhao
While doing HDFS upgrade with HA setup, if the layoutversion gets changed in
the upgrade, the rollback may trigger the following exception in JournalNodes
(suppose the new software bumped the layoutversion from -55 to -56):
{code}
14/03/21 01:01:53 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not check if roll
back possible for one or more JournalNodes. 1 exceptions thrown:
Unexpected version of storage directory /grid/1/tmp/journal/mycluster.
Reported: -56. Expecting = -55.
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:203)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:156)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:135)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.analyzeStorage(JNStorage.java:202)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.(JNStorage.java:73)
at
org.apache.hadoop.hdfs.qjournal.server.Journal.(Journal.java:142)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:87)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.canRollBack(JournalNode.java:304)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.canRollBack(JournalNodeRpcServer.java:228)
{code}
Looks like for rollback JN with old software cannot handle future layoutversion
brought by new software.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6134) Transparent data at rest encryption
Alejandro Abdelnur created HDFS-6134: Summary: Transparent data at rest encryption Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6133) Make Balancer support don't move blocks belongs to Hbase
[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942811#comment-13942811 ] stack commented on HDFS-6133: - HDFS-4420 looks related? I skimmed the patch. Looks good. You should probably change the issue subject to exclude specified paths from consideration rather than talk of hbase (your patch is of use to more than just hbase installs).It looks like I can pass multiple -excludes when I run the balancer? Is the balancer run automatically by HDFS on a period or is it only run manually? > Make Balancer support don't move blocks belongs to Hbase > > > Key: HDFS-6133 > URL: https://issues.apache.org/jira/browse/HDFS-6133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer, namenode >Reporter: zhaoyunjiong >Assignee: zhaoyunjiong > Attachments: HDFS-6133.patch > > > Currently, run Balancer will destroying Regionserver's data locality. > If getBlocks could exclude blocks belongs to files which have specific path > prefix, like "/hbase", then we can run Balancer without destroying > Regionserver's data locality. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5978) Create a tool to take fsimage and expose read-only WebHDFS API
[
https://issues.apache.org/jira/browse/HDFS-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942806#comment-13942806
]
Hadoop QA commented on HDFS-5978:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635938/HDFS-5978.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6461//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6461//console
This message is automatically generated.
> Create a tool to take fsimage and expose read-only WebHDFS API
> --
>
> Key: HDFS-5978
> URL: https://issues.apache.org/jira/browse/HDFS-5978
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HDFS-5978.2.patch, HDFS-5978.3.patch, HDFS-5978.4.patch,
> HDFS-5978.patch
>
>
> Suggested in HDFS-5975.
> Add an option to exposes the read-only version of WebHDFS API for
> OfflineImageViewer. You can imagine it looks very similar to jhat.
> That way we can allow the operator to use the existing command-line tool, or
> even the web UI to debug the fsimage. It also allows the operator to
> interactively browsing the file system, figuring out what goes wrong.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-6130:
--
Attachment: (was: HDFS-6130.patch)
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942777#comment-13942777
]
Suresh Srinivas commented on HDFS-6130:
---
bq. So I don't think this was caused by cdh release
I suspect this is due to some layout version mismatch issues. Given apache
releases do not have this issue, I suggest closing this as invalid. If you want
to explore fixing this further, I suggest looking at layout version issue,
possibly related to inode id feature introduction and later features.
Certainly turning of ha and upgrading is one option. The current patch is not a
choice as it will ignore many paths on encountering an error.
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
> Attachments: HDFS-6130.patch
>
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5196) Provide more snapshot information in WebUI
[
https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942768#comment-13942768
]
Haohui Mai commented on HDFS-5196:
--
What I meant was to move {{SnapshotInfo.Bean}} and
{{SnapshottableDirectoryStatus.Bean}} into {{SnapshotStatsMXBean}}, but it's
your call.
> Provide more snapshot information in WebUI
> --
>
> Key: HDFS-5196
> URL: https://issues.apache.org/jira/browse/HDFS-5196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 3.0.0
>Reporter: Haohui Mai
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HDFS-5196-2.patch, HDFS-5196-3.patch, HDFS-5196-4.patch,
> HDFS-5196-5.patch, HDFS-5196-6.patch, HDFS-5196-7.patch, HDFS-5196.patch,
> HDFS-5196.patch, HDFS-5196.patch, snapshot-new-webui.png,
> snapshottable-directoryList.png, snapshotteddir.png
>
>
> The WebUI should provide more detailed information about snapshots, such as
> all snapshottable directories and corresponding number of snapshots
> (suggested in HDFS-4096).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5196) Provide more snapshot information in WebUI
[ https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942765#comment-13942765 ] Shinichi Yamashita commented on HDFS-5196: -- Thank you for your comment. bq. Do you think it makes sense to move both definitions of the beans to SnapshotStatsMXBean? You mean that SnapshotStatsMXBean is not neccessary ? Please give me more details about this. > Provide more snapshot information in WebUI > -- > > Key: HDFS-5196 > URL: https://issues.apache.org/jira/browse/HDFS-5196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Haohui Mai >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: HDFS-5196-2.patch, HDFS-5196-3.patch, HDFS-5196-4.patch, > HDFS-5196-5.patch, HDFS-5196-6.patch, HDFS-5196-7.patch, HDFS-5196.patch, > HDFS-5196.patch, HDFS-5196.patch, snapshot-new-webui.png, > snapshottable-directoryList.png, snapshotteddir.png > > > The WebUI should provide more detailed information about snapshots, such as > all snapshottable directories and corresponding number of snapshots > (suggested in HDFS-4096). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942764#comment-13942764
]
Fengdong Yu commented on HDFS-6130:
---
cacel patch, It loss data sometimes.
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
> Attachments: HDFS-6130.patch
>
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-6130:
--
Status: Open (was: Patch Available)
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
> Attachments: HDFS-6130.patch
>
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-6130:
--
Status: Patch Available (was: Open)
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
> Attachments: HDFS-6130.patch
>
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-6130:
--
Attachment: (was: HDFS-6130.patch)
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
> Attachments: HDFS-6130.patch
>
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-6130:
--
Attachment: HDFS-6130.patch
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
> Attachments: HDFS-6130.patch
>
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-6130:
--
Attachment: HDFS-6130.patch
I am not sure I resolved the root casue, but It works now.
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
> Attachments: HDFS-6130.patch
>
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6124) Add final modifier to class members
[
https://issues.apache.org/jira/browse/HDFS-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942743#comment-13942743
]
Hadoop QA commented on HDFS-6124:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635925/HDFS-6124.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 139
new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6460//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6460//console
This message is automatically generated.
> Add final modifier to class members
> ---
>
> Key: HDFS-6124
> URL: https://issues.apache.org/jira/browse/HDFS-6124
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.4.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFS-6124.1.patch, HDFS-6124.patch
>
>
> Many of the member variables declaration for classes are missing final
> modifier in HDFS. This jira adds final modifier where possible.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5978) Create a tool to take fsimage and expose read-only WebHDFS API
[
https://issues.apache.org/jira/browse/HDFS-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942682#comment-13942682
]
Akira AJISAKA commented on HDFS-5978:
-
Thanks for the comments! I renewed a patch.
bq. It is better to avoid the dependency. Can you experiement with
{{Jackson#ObjectMapper}}?
I suppose it's better to use {{JsonUtil}} because it is used in WebHDFS
({{NameNodeWebHdfsMethods.java}}). Nevertheless, I'll try to use
{{ObjectMapper}} and attach a patch later.
> Create a tool to take fsimage and expose read-only WebHDFS API
> --
>
> Key: HDFS-5978
> URL: https://issues.apache.org/jira/browse/HDFS-5978
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HDFS-5978.2.patch, HDFS-5978.3.patch, HDFS-5978.4.patch,
> HDFS-5978.patch
>
>
> Suggested in HDFS-5975.
> Add an option to exposes the read-only version of WebHDFS API for
> OfflineImageViewer. You can imagine it looks very similar to jhat.
> That way we can allow the operator to use the existing command-line tool, or
> even the web UI to debug the fsimage. It also allows the operator to
> interactively browsing the file system, figuring out what goes wrong.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5978) Create a tool to take fsimage and expose read-only WebHDFS API
[ https://issues.apache.org/jira/browse/HDFS-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5978: Attachment: HDFS-5978.4.patch > Create a tool to take fsimage and expose read-only WebHDFS API > -- > > Key: HDFS-5978 > URL: https://issues.apache.org/jira/browse/HDFS-5978 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: tools >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Labels: newbie > Attachments: HDFS-5978.2.patch, HDFS-5978.3.patch, HDFS-5978.4.patch, > HDFS-5978.patch > > > Suggested in HDFS-5975. > Add an option to exposes the read-only version of WebHDFS API for > OfflineImageViewer. You can imagine it looks very similar to jhat. > That way we can allow the operator to use the existing command-line tool, or > even the web UI to debug the fsimage. It also allows the operator to > interactively browsing the file system, figuring out what goes wrong. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942639#comment-13942639
]
Fengdong Yu commented on HDFS-6130:
---
[~sureshms] maybe you are right, I have three test clusters, only this one is
cdh release. others are all Apache release.
but cdh old release can upgrade to Apache 2.2.0 successfully. so I don't think
this was caused by cdh release.
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HDFS-5138) Support HDFS upgrade in HA
[
https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942638#comment-13942638
]
Jing Zhao edited comment on HDFS-5138 at 3/21/14 1:36 AM:
--
So I did a simple test for HDFS upgrade with HA, and hit the following
exception while doing rollback (with layoutversion change in the upgrade):
{code}
14/03/21 01:01:53 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not check if roll
back possible for one or more JournalNodes. 1 exceptions thrown:
Unexpected version of storage directory /grid/1/tmp/journal/mycluster.
Reported: -56. Expecting = -55.
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.analyzeStorage(JNStorage.java:202)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.(JNStorage.java:73)
at
org.apache.hadoop.hdfs.qjournal.server.Journal.(Journal.java:142)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:87)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.canRollBack(JournalNode.java:309)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.canRollBack(JournalNodeRpcServer.java:228)
{code}
In my HA upgrade test, the new software bumped the layoutversion from -55 to
-56. I stopped all the services and restarted JNs with old software. Then I run
"namenode -rollback" and hit the above exception. Looks like for rollback JN
with old software cannot handle future layoutversion brought by new software.
was (Author: jingzhao):
So I did a simple test for HDFS upgrade with HA, and hit the following
exception while doing rollback (with layoutversion change in the upgrade):
{code}
14/03/21 01:01:53 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not check if roll
back possible for one or more JournalNodes. 1 exceptions thrown:
Unexpected version of storage directory /grid/1/tmp/journal/mycluster.
Reported: -56. Expecting = -55.
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.analyzeStorage(JNStorage.java:202)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.(JNStorage.java:73)
at
org.apache.hadoop.hdfs.qjournal.server.Journal.(Journal.java:142)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:87)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.canRollBack(JournalNode.java:309)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.canRollBack(JournalNodeRpcServer.java:228)
{code}
In my HA upgrade test, the new software bumped the layoutversion from -55 to
-56. Then I stopped all the services and restarted JNs with old software. Then
I run "namenode -rollback" and hit the above exception. Looks like for rollback
JN with old software cannot handle future layoutversion brought by new software.
> Support HDFS upgrade in HA
> --
>
> Key: HDFS-5138
> URL: https://issues.apache.org/jira/browse/HDFS-5138
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.1.1-beta
>Reporter: Kihwal Lee
>Assignee: Aaron T. Myers
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch,
> HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch,
> HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch,
> hdfs-5138-branch-2.txt
>
>
> With HA enabled, NN wo't start with "-upgrade". Since there has been a layout
> version change between 2.0.x and 2.1.x, starting NN in upgrade mode was
> necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way
> to get around this was to disable HA and upgrade.
> The NN and the cluster cannot be flipped back to HA until the upgrade is
> finalized. If HA is disabled only on NN for layout upgrade and HA is turned
> back on without involving DNs, things will work, but finaliizeUpgrade won't
> work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade
> snapshots won't get removed.
> We will need a different ways of doing layout upgrade and
[jira] [Commented] (HDFS-5138) Support HDFS upgrade in HA
[
https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942638#comment-13942638
]
Jing Zhao commented on HDFS-5138:
-
So I did a simple test for HDFS upgrade with HA, and hit the following
exception while doing rollback (with layoutversion change in the upgrade):
{code}
14/03/21 01:01:53 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not check if roll
back possible for one or more JournalNodes. 1 exceptions thrown:
Unexpected version of storage directory /grid/1/tmp/journal/mycluster.
Reported: -56. Expecting = -55.
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.analyzeStorage(JNStorage.java:202)
at
org.apache.hadoop.hdfs.qjournal.server.JNStorage.(JNStorage.java:73)
at
org.apache.hadoop.hdfs.qjournal.server.Journal.(Journal.java:142)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:87)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNode.canRollBack(JournalNode.java:309)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.canRollBack(JournalNodeRpcServer.java:228)
{code}
In my HA upgrade test, the new software bumped the layoutversion from -55 to
-56. Then I stopped all the services and restarted JNs with old software. Then
I run "namenode -rollback" and hit the above exception. Looks like for rollback
JN with old software cannot handle future layoutversion brought by new software.
> Support HDFS upgrade in HA
> --
>
> Key: HDFS-5138
> URL: https://issues.apache.org/jira/browse/HDFS-5138
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.1.1-beta
>Reporter: Kihwal Lee
>Assignee: Aaron T. Myers
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch,
> HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch,
> HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch,
> hdfs-5138-branch-2.txt
>
>
> With HA enabled, NN wo't start with "-upgrade". Since there has been a layout
> version change between 2.0.x and 2.1.x, starting NN in upgrade mode was
> necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way
> to get around this was to disable HA and upgrade.
> The NN and the cluster cannot be flipped back to HA until the upgrade is
> finalized. If HA is disabled only on NN for layout upgrade and HA is turned
> back on without involving DNs, things will work, but finaliizeUpgrade won't
> work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade
> snapshots won't get removed.
> We will need a different ways of doing layout upgrade and upgrade snapshot.
> I am marking this as a 2.1.1-beta blocker based on feedback from others. If
> there is a reasonable workaround that does not increase maintenance window
> greatly, we can lower its priority from blocker to critical.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-5138: --- > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Assignee: Aaron T. Myers >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > hdfs-5138-branch-2.txt > > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942636#comment-13942636 ] Suresh Srinivas commented on HDFS-5138: --- Reopening this jira to make sure it gets tracked as blocker for 2.4. > Support HDFS upgrade in HA > -- > > Key: HDFS-5138 > URL: https://issues.apache.org/jira/browse/HDFS-5138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Kihwal Lee >Assignee: Aaron T. Myers >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, > hdfs-5138-branch-2.txt > > > With HA enabled, NN wo't start with "-upgrade". Since there has been a layout > version change between 2.0.x and 2.1.x, starting NN in upgrade mode was > necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way > to get around this was to disable HA and upgrade. > The NN and the cluster cannot be flipped back to HA until the upgrade is > finalized. If HA is disabled only on NN for layout upgrade and HA is turned > back on without involving DNs, things will work, but finaliizeUpgrade won't > work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade > snapshots won't get removed. > We will need a different ways of doing layout upgrade and upgrade snapshot. > I am marking this as a 2.1.1-beta blocker based on feedback from others. If > there is a reasonable workaround that does not increase maintenance window > greatly, we can lower its priority from blocker to critical. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942634#comment-13942634
]
Suresh Srinivas commented on HDFS-6130:
---
>From what I gather, between Apache releases the upgrade works, right? If so,
>cdh release related issues perhaps are best taken up in a forum related to
>that?
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942627#comment-13942627
]
Fengdong Yu commented on HDFS-6130:
---
[~andrew.wang] , can you also take a look?
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6120) Fix and improve safe mode log messages
[
https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942606#comment-13942606
]
Hadoop QA commented on HDFS-6120:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635884/HDFS-6120.03.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSafeMode
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6458//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6458//console
This message is automatically generated.
> Fix and improve safe mode log messages
> --
>
> Key: HDFS-6120
> URL: https://issues.apache.org/jira/browse/HDFS-6120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6120.01.patch, HDFS-6120.02.patch,
> HDFS-6120.03.patch, HDFS-6120.04.patch
>
>
> In HA mode the SBN can enter safe-mode extension and stay there even after
> the extension period has elapsed but continue to return the safemode message
> stating that "The threshold has been reached" and "safe mode will be turned
> off soon".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5988) Bad fsimage always generated after upgrade
[ https://issues.apache.org/jira/browse/HDFS-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942603#comment-13942603 ] Fengdong Yu commented on HDFS-5988: --- [~andrew.wang] , please take a look HDFS-6130, still failed upgrade from an old release. > Bad fsimage always generated after upgrade > -- > > Key: HDFS-5988 > URL: https://issues.apache.org/jira/browse/HDFS-5988 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Fix For: 2.4.0 > > Attachments: hdfs-5988-1.patch > > > Internal testing revealed an issue where, after upgrading from an earlier > release, we always fail to save a correct PB-based fsimage (namely, missing > inodes leading to an inconsistent namespace). This results in substantial > data loss, since the upgraded fsimage is broken, as well as the fsimages > generated by saveNamespace and checkpointing. > This ended up being a bug in the old fsimage loading code, patch coming. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942602#comment-13942602
]
Fengdong Yu commented on HDFS-6130:
---
[~szetszwo], I've included the HDFS-5988, I tested trunk from Revision: 1579559
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5910) Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
[
https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942583#comment-13942583
]
Hadoop QA commented on HDFS-5910:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635876/HDFS-5910.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6456//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6456//console
This message is automatically generated.
> Enhance DataTransferProtocol to allow per-connection choice of
> encryption/plain-text
>
>
> Key: HDFS-5910
> URL: https://issues.apache.org/jira/browse/HDFS-5910
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-5910.patch, HDFS-5910.patch
>
>
> It is possible to enable encryption of DataTransferProtocol.
> In some use cases, it is required to encrypt data transfer with some clients
> , but communicate in plain text with some other clients and data nodes.
> A sample use case will be that any data transfer inside a firewall can be in
> plain text whereas any data transfer from clients outside the firewall needs
> to be encrypted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6124) Add final modifier to class members
[ https://issues.apache.org/jira/browse/HDFS-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-6124: -- Attachment: HDFS-6124.1.patch findbug issue is addressed in the latest patch. > Add final modifier to class members > --- > > Key: HDFS-6124 > URL: https://issues.apache.org/jira/browse/HDFS-6124 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.4.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-6124.1.patch, HDFS-6124.patch > > > Many of the member variables declaration for classes are missing final > modifier in HDFS. This jira adds final modifier where possible. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6132) TestBlockToken fails on JDK 7
[
https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942574#comment-13942574
]
Hadoop QA commented on HDFS-6132:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635870/HDFS-6132.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.qjournal.TestNNWithQJM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6455//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6455//console
This message is automatically generated.
> TestBlockToken fails on JDK 7
> -
>
> Key: HDFS-6132
> URL: https://issues.apache.org/jira/browse/HDFS-6132
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Siqi Li
> Attachments: HDFS-6132.patch
>
>
> Currently, UserGroupInformation.setConfiguration(conf) does not reset
> loginUser. This is causing this test case to fail.
> For now, the work around solution is to use UserGroupInformation.reset()
> before UserGroupInformation.setConfiguration(conf). However, in general, we
> should do this reset whenever we call
> UserGroupInformation.setConfiguration(conf)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6120) Fix and improve safe mode log messages
[ https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6120: Attachment: HDFS-6120.04.patch That was careless of me, updated patch attached. Thanks Jing! > Fix and improve safe mode log messages > -- > > Key: HDFS-6120 > URL: https://issues.apache.org/jira/browse/HDFS-6120 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6120.01.patch, HDFS-6120.02.patch, > HDFS-6120.03.patch, HDFS-6120.04.patch > > > In HA mode the SBN can enter safe-mode extension and stay there even after > the extension period has elapsed but continue to return the safemode message > stating that "The threshold has been reached" and "safe mode will be turned > off soon". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
[ https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942570#comment-13942570 ] Hudson commented on HDFS-6038: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5367 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5367/]) HDFS-6038. Allow JournalNode to handle editlog produced by new release with future layoutversion. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579813) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DataOutputBuffer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLogger.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocol/QJournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/QJournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/QJournalProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/RedundantEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/BinaryEditsVisitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/Off
[jira] [Commented] (HDFS-5910) Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
[
https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942523#comment-13942523
]
Arpit Agarwal commented on HDFS-5910:
-
Thanks for the clarifications [~benoyantony].
Few comments on the patch:
# {{isSecureOnClient}} may also want to use the peer's address to make a
decision. e.g. intra-cluster transfer vs. distcp to remote cluster.
# Related to #1, {{isSecureOnClient}} and {{isSecureOnServer}} look awkward.
How about replacing both with {{isTrustedChannel}} that takes the peer's IP
address? We should probably avoid overloading the term secure in this context
since there is a related concept of {{Peer#hasSecureChannel()}}.
# Could you please update the documentation for {{dfs.encrypt.data.transfer}}
to state that per-connection override is possible via a custom resolver which
can be configured with {{dfs.securechannel.resolver.class}}. Also move the two
settings to be next to each other in the docs?
# Is the {{InetAddress.getByName}} call in {{DataXceiver#getClientAddress}}
necessary? If it were necessary it would have been a security hole since DNS
resolution may yield a different IP address than the one used by the client. It
turns out for the kinds of Peers we are interested in this will be an IP
address, so let's just remove the call.
The patch looks fine otherwise.
> Enhance DataTransferProtocol to allow per-connection choice of
> encryption/plain-text
>
>
> Key: HDFS-5910
> URL: https://issues.apache.org/jira/browse/HDFS-5910
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-5910.patch, HDFS-5910.patch
>
>
> It is possible to enable encryption of DataTransferProtocol.
> In some use cases, it is required to encrypt data transfer with some clients
> , but communicate in plain text with some other clients and data nodes.
> A sample use case will be that any data transfer inside a firewall can be in
> plain text whereas any data transfer from clients outside the firewall needs
> to be encrypted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6120) Fix and improve safe mode log messages
[
https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942502#comment-13942502
]
Jing Zhao commented on HDFS-6120:
-
{code}
+ msg += (reached == 0) ? "In safe mode extension. " : "";
{code}
I guess your mean "reached > 0" here? :)
Other than this +1.
> Fix and improve safe mode log messages
> --
>
> Key: HDFS-6120
> URL: https://issues.apache.org/jira/browse/HDFS-6120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6120.01.patch, HDFS-6120.02.patch,
> HDFS-6120.03.patch
>
>
> In HA mode the SBN can enter safe-mode extension and stay there even after
> the extension period has elapsed but continue to return the safemode message
> stating that "The threshold has been reached" and "safe mode will be turned
> off soon".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
[
https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6038:
Resolution: Fixed
Fix Version/s: 2.4.0
Status: Resolved (was: Patch Available)
I've committed the patch to trunk, branch-2, and branch-2.4. Thanks Nicholas
and Todd for the review!
> Allow JournalNode to handle editlog produced by new release with future
> layoutversion
> -
>
> Key: HDFS-6038
> URL: https://issues.apache.org/jira/browse/HDFS-6038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: journal-node, namenode
>Reporter: Haohui Mai
>Assignee: Jing Zhao
> Fix For: 2.4.0
>
> Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch,
> HDFS-6038.002.patch, HDFS-6038.003.patch, HDFS-6038.004.patch,
> HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch,
> HDFS-6038.008.patch, editsStored
>
>
> In HA setup, the JNs receive edit logs (blob) from the NN and write into edit
> log files. In order to write well-formed edit log files, the JNs prepend a
> header for each edit log file. The problem is that the JN hard-codes the
> version (i.e., {{NameNodeLayoutVersion}} in the edit log, therefore it
> generates incorrect edit logs when the newer release bumps the
> {{NameNodeLayoutVersion}} during rolling upgrade.
> In the meanwhile, currently JN tries to decode the in-progress editlog
> segment in order to know the last txid in the segment. In the rolling upgrade
> scenario, the JN with the old software may not be able to correctly decode
> the editlog generated by the new software.
> This jira makes the following changes to allow JN to handle editlog produced
> by software with future layoutversion:
> 1. Change the NN--JN startLogSegment RPC signature and let NN specify the
> layoutversion for the new editlog segment.
> 2. Persist a length field for each editlog op to indicate the total length of
> the op. Instead of calling EditLogFileInputStream#validateEditLog to get the
> last txid of an in-progress editlog segment, a new method scanEditLog is
> added and used by JN which does not decode each editlog op but uses the
> length to quickly jump to the next op.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-4192) LocalFileSystem does not implement getFileChecksum()
[ https://issues.apache.org/jira/browse/HDFS-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Uhlig updated HDFS-4192: - Affects Version/s: (was: 2.0.2-alpha) 2.4.0 2.2.0 2.3.0 > LocalFileSystem does not implement getFileChecksum() > > > Key: HDFS-4192 > URL: https://issues.apache.org/jira/browse/HDFS-4192 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.2.0, 2.3.0, 2.4.0 >Reporter: Hans Uhlig > > LocalFileSystem decendent of ChecksumFileSystem does not implement > getFileChecksum(). I would like to be able to compare checksums for files in > HDFS to files on local disk to files in HDFS quickly for differences. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
[
https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942461#comment-13942461
]
Jing Zhao commented on HDFS-6038:
-
Thanks for the review, Nicholas! I will commit the patch shortly.
> Allow JournalNode to handle editlog produced by new release with future
> layoutversion
> -
>
> Key: HDFS-6038
> URL: https://issues.apache.org/jira/browse/HDFS-6038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: journal-node, namenode
>Reporter: Haohui Mai
>Assignee: Jing Zhao
> Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch,
> HDFS-6038.002.patch, HDFS-6038.003.patch, HDFS-6038.004.patch,
> HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch,
> HDFS-6038.008.patch, editsStored
>
>
> In HA setup, the JNs receive edit logs (blob) from the NN and write into edit
> log files. In order to write well-formed edit log files, the JNs prepend a
> header for each edit log file. The problem is that the JN hard-codes the
> version (i.e., {{NameNodeLayoutVersion}} in the edit log, therefore it
> generates incorrect edit logs when the newer release bumps the
> {{NameNodeLayoutVersion}} during rolling upgrade.
> In the meanwhile, currently JN tries to decode the in-progress editlog
> segment in order to know the last txid in the segment. In the rolling upgrade
> scenario, the JN with the old software may not be able to correctly decode
> the editlog generated by the new software.
> This jira makes the following changes to allow JN to handle editlog produced
> by software with future layoutversion:
> 1. Change the NN--JN startLogSegment RPC signature and let NN specify the
> layoutversion for the new editlog segment.
> 2. Persist a length field for each editlog op to indicate the total length of
> the op. Instead of calling EditLogFileInputStream#validateEditLog to get the
> last txid of an in-progress editlog segment, a new method scanEditLog is
> added and used by JN which does not decode each editlog op but uses the
> length to quickly jump to the next op.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6133) Make Balancer support don't move blocks belongs to Hbase
[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyunjiong updated HDFS-6133: --- Attachment: HDFS-6133.patch This patch make Balancer support don't move blocks belongs to Hbase > Make Balancer support don't move blocks belongs to Hbase > > > Key: HDFS-6133 > URL: https://issues.apache.org/jira/browse/HDFS-6133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer, namenode >Reporter: zhaoyunjiong >Assignee: zhaoyunjiong > Attachments: HDFS-6133.patch > > > Currently, run Balancer will destroying Regionserver's data locality. > If getBlocks could exclude blocks belongs to files which have specific path > prefix, like "/hbase", then we can run Balancer without destroying > Regionserver's data locality. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5910) Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
[ https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942438#comment-13942438 ] Benoy Antony commented on HDFS-5910: BTW, this is in similar lines of HADOOP-10221 where custom logic can be used to determine the QOP for a Sasl connection. > Enhance DataTransferProtocol to allow per-connection choice of > encryption/plain-text > > > Key: HDFS-5910 > URL: https://issues.apache.org/jira/browse/HDFS-5910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5910.patch, HDFS-5910.patch > > > It is possible to enable encryption of DataTransferProtocol. > In some use cases, it is required to encrypt data transfer with some clients > , but communicate in plain text with some other clients and data nodes. > A sample use case will be that any data transfer inside a firewall can be in > plain text whereas any data transfer from clients outside the firewall needs > to be encrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5910) Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
[ https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942436#comment-13942436 ] Benoy Antony commented on HDFS-5910: [~arpitagarwal] I have added the sample use case as part of the description. The title looks accurate. Thanks. > Enhance DataTransferProtocol to allow per-connection choice of > encryption/plain-text > > > Key: HDFS-5910 > URL: https://issues.apache.org/jira/browse/HDFS-5910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5910.patch, HDFS-5910.patch > > > It is possible to enable encryption of DataTransferProtocol. > In some use cases, it is required to encrypt data transfer with some clients > , but communicate in plain text with some other clients and data nodes. > A sample use case will be that any data transfer inside a firewall can be in > plain text whereas any data transfer from clients outside the firewall needs > to be encrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6133) Make Balancer support don't move blocks belongs to Hbase
zhaoyunjiong created HDFS-6133: -- Summary: Make Balancer support don't move blocks belongs to Hbase Key: HDFS-6133 URL: https://issues.apache.org/jira/browse/HDFS-6133 Project: Hadoop HDFS Issue Type: Improvement Components: balancer, namenode Reporter: zhaoyunjiong Assignee: zhaoyunjiong Currently, run Balancer will destroying Regionserver's data locality. If getBlocks could exclude blocks belongs to files which have specific path prefix, like "/hbase", then we can run Balancer without destroying Regionserver's data locality. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5910) Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
[ https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-5910: --- Description: It is possible to enable encryption of DataTransferProtocol. In some use cases, it is required to encrypt data transfer with some clients , but communicate in plain text with some other clients and data nodes. A sample use case will be that any data transfer inside a firewall can be in plain text whereas any data transfer from clients outside the firewall needs to be encrypted. was: It is possible to enable encryption of DataTransferProtocol. In some use cases, it is required to encrypt data transfer with some clients , but communicate in plain text with some other clients and datanodes. > Enhance DataTransferProtocol to allow per-connection choice of > encryption/plain-text > > > Key: HDFS-5910 > URL: https://issues.apache.org/jira/browse/HDFS-5910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5910.patch, HDFS-5910.patch > > > It is possible to enable encryption of DataTransferProtocol. > In some use cases, it is required to encrypt data transfer with some clients > , but communicate in plain text with some other clients and data nodes. > A sample use case will be that any data transfer inside a firewall can be in > plain text whereas any data transfer from clients outside the firewall needs > to be encrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6050) NFS OpenFileCtx does not handle exceptions correctly
[
https://issues.apache.org/jira/browse/HDFS-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942432#comment-13942432
]
Hadoop QA commented on HDFS-6050:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635881/HDFS-6050.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6457//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6457//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6457//console
This message is automatically generated.
> NFS OpenFileCtx does not handle exceptions correctly
>
>
> Key: HDFS-6050
> URL: https://issues.apache.org/jira/browse/HDFS-6050
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Reporter: Brock Noland
>Assignee: Brandon Li
> Attachments: HDFS-6050.patch
>
>
> I noticed this file does not log exceptions appropriately in multiple
> locations.
> Not logging the stack of Throwable:
> https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L364
> Printing exceptions to stderr:
> https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1160
> https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1149
> Not logging the stack trace:
> https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1062
> https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L966
> https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L961
> https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L680
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5910) Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
[ https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942408#comment-13942408 ] Arpit Agarwal commented on HDFS-5910: - I updated the Jira title to reflect the content of the change more accurately. Let me know if it's inaccurate. > Enhance DataTransferProtocol to allow per-connection choice of > encryption/plain-text > > > Key: HDFS-5910 > URL: https://issues.apache.org/jira/browse/HDFS-5910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5910.patch, HDFS-5910.patch > > > It is possible to enable encryption of DataTransferProtocol. > In some use cases, it is required to encrypt data transfer with some clients > , but communicate in plain text with some other clients and datanodes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5910) Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text
[ https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5910: Summary: Enhance DataTransferProtocol to allow per-connection choice of encryption/plain-text (was: Enhance DataTransferProtocol to support encrypted and plain-text communication) > Enhance DataTransferProtocol to allow per-connection choice of > encryption/plain-text > > > Key: HDFS-5910 > URL: https://issues.apache.org/jira/browse/HDFS-5910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5910.patch, HDFS-5910.patch > > > It is possible to enable encryption of DataTransferProtocol. > In some use cases, it is required to encrypt data transfer with some clients > , but communicate in plain text with some other clients and datanodes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5910) Enhance DataTransferProtocol to support encrypted and plain-text communication
[
https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942405#comment-13942405
]
Arpit Agarwal commented on HDFS-5910:
-
Hi Benoy, do you have an example use case for when we might want to let the
{{SecureChannelResolver}} override the encryption setting?
> Enhance DataTransferProtocol to support encrypted and plain-text communication
> --
>
> Key: HDFS-5910
> URL: https://issues.apache.org/jira/browse/HDFS-5910
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-5910.patch, HDFS-5910.patch
>
>
> It is possible to enable encryption of DataTransferProtocol.
> In some use cases, it is required to encrypt data transfer with some clients
> , but communicate in plain text with some other clients and datanodes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6120) Fix and improve safe mode log messages
[ https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6120: Summary: Fix and improve safe mode log messages (was: Cleanup safe mode error messages) > Fix and improve safe mode log messages > -- > > Key: HDFS-6120 > URL: https://issues.apache.org/jira/browse/HDFS-6120 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6120.01.patch, HDFS-6120.02.patch, > HDFS-6120.03.patch > > > In HA mode the SBN can enter safe-mode extension and stay there even after > the extension period has elapsed but continue to return the safemode message > stating that "The threshold has been reached" and "safe mode will be turned > off soon". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6120) Cleanup safe mode error messages
[ https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6120: Attachment: HDFS-6120.03.patch Update hard-coded strings in the test. Also updated the safe mode tip to make it more useful, hopefully. > Cleanup safe mode error messages > > > Key: HDFS-6120 > URL: https://issues.apache.org/jira/browse/HDFS-6120 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6120.01.patch, HDFS-6120.02.patch, > HDFS-6120.03.patch > > > In HA mode the SBN can enter safe-mode extension and stay there even after > the extension period has elapsed but continue to return the safemode message > stating that "The threshold has been reached" and "safe mode will be turned > off soon". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6050) NFS OpenFileCtx does not handle exceptions correctly
[ https://issues.apache.org/jira/browse/HDFS-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6050: - Status: Patch Available (was: Open) > NFS OpenFileCtx does not handle exceptions correctly > > > Key: HDFS-6050 > URL: https://issues.apache.org/jira/browse/HDFS-6050 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Brock Noland >Assignee: Brandon Li > Attachments: HDFS-6050.patch > > > I noticed this file does not log exceptions appropriately in multiple > locations. > Not logging the stack of Throwable: > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L364 > Printing exceptions to stderr: > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1160 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1149 > Not logging the stack trace: > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1062 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L966 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L961 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L680 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6050) NFS OpenFileCtx does not handle exceptions correctly
[ https://issues.apache.org/jira/browse/HDFS-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6050: - Attachment: HDFS-6050.patch Uploaded a patch to fix the exception handling issue along with some code clean up. > NFS OpenFileCtx does not handle exceptions correctly > > > Key: HDFS-6050 > URL: https://issues.apache.org/jira/browse/HDFS-6050 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Brock Noland >Assignee: Brandon Li > Attachments: HDFS-6050.patch > > > I noticed this file does not log exceptions appropriately in multiple > locations. > Not logging the stack of Throwable: > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L364 > Printing exceptions to stderr: > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1160 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1149 > Not logging the stack trace: > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L1062 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L966 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L961 > https://github.com/apache/hadoop-common/blob/f567f09091368fe800f3f70605da38a69c953fe3/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java#L680 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5196) Provide more snapshot information in WebUI
[
https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942342#comment-13942342
]
Haohui Mai commented on HDFS-5196:
--
The patch mostly looks good.
{code}
+ public static class Bean {...
{code}
Do you think it makes sense to move both definitions of the beans to
{{SnapshotStatsMXBean}}? You can convert the constructor into a function. For
example:
{code}
private static SnapshotDirectoryMXBean toBean(SnapshottableDirectoryStatus s) {
return new SnapshotDirectoryMXBean(...);
}
{code}
{code}
+ /**
+ * @return The number of snapshottale directories in the system
+ */
+ public int getNumSnapshottableDirs();
+
+ /**
+ * @return The number of directories that have been snapshotted
+ */
+ public int getNumSnapshots();
{code}
Since the code now returns the full lists of snapshottable directories and the
snapshots. I think we can get rid of these two calls. I think you can either
get rid of the count on the UI, or use a helper to count them (since the
{{size}} helper in dustjs looks buggy). It's your call.
{code}
+'permission_tostring' : function(v) {
+ var p = ['---', '--x', '-w-', '-wx', 'r--', 'r-x', 'rw-', 'rwx'];
+ return p[(v >>> 6) & 7] + p[(v >>> 3) & 7] + p[v & 7];
}
{code}
Let's move the function {{helper_to_permission}} from {{explorer.js}} here and
use it instead.
It might be worthwhile to bring back the unit test in the v6 patch.
nit: there are a few places that have trailing whitespaces.
> Provide more snapshot information in WebUI
> --
>
> Key: HDFS-5196
> URL: https://issues.apache.org/jira/browse/HDFS-5196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 3.0.0
>Reporter: Haohui Mai
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HDFS-5196-2.patch, HDFS-5196-3.patch, HDFS-5196-4.patch,
> HDFS-5196-5.patch, HDFS-5196-6.patch, HDFS-5196-7.patch, HDFS-5196.patch,
> HDFS-5196.patch, HDFS-5196.patch, snapshot-new-webui.png,
> snapshottable-directoryList.png, snapshotteddir.png
>
>
> The WebUI should provide more detailed information about snapshots, such as
> all snapshottable directories and corresponding number of snapshots
> (suggested in HDFS-4096).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6120) Cleanup safe mode error messages
[
https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942324#comment-13942324
]
Hadoop QA commented on HDFS-6120:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635841/HDFS-6120.02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSafeMode
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6454//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6454//console
This message is automatically generated.
> Cleanup safe mode error messages
>
>
> Key: HDFS-6120
> URL: https://issues.apache.org/jira/browse/HDFS-6120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6120.01.patch, HDFS-6120.02.patch
>
>
> In HA mode the SBN can enter safe-mode extension and stay there even after
> the extension period has elapsed but continue to return the safemode message
> stating that "The threshold has been reached" and "safe mode will be turned
> off soon".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5910) Enhance DataTransferProtocol to support encrypted and plain-text communication
[ https://issues.apache.org/jira/browse/HDFS-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-5910: --- Attachment: HDFS-5910.patch Attaching a new patch . The approach taken is different from the approach mentioned in the previous comment. The DataTransfer does not support the full SASL QOP values. It supports encryption and plainText. To achieve the goal of determining whether to encrypt based on each connection, _SecureChannelResolver_ (new interface) is defined. The interface can be plugged in via configuration. The server side (_DataXceiver_) consults _SecureChannelResolver_ instance to determine whether the channel is secure or not. If the _SecureChannelResolver_ returns true (channel is secure) , then encryption is not done. Similar process is done at the client side also while attempting the data transfer. The default implementation of _SecureChannelResolver_ returns false signaling that the channel is not secure and encryption is required. The _SecureChannelResolver_ is consulted only if dfs.encrypt.data.transfer is set to true. > Enhance DataTransferProtocol to support encrypted and plain-text communication > -- > > Key: HDFS-5910 > URL: https://issues.apache.org/jira/browse/HDFS-5910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-5910.patch, HDFS-5910.patch > > > It is possible to enable encryption of DataTransferProtocol. > In some use cases, it is required to encrypt data transfer with some clients > , but communicate in plain text with some other clients and datanodes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6093) Expose more caching information for debugging by users
[
https://issues.apache.org/jira/browse/HDFS-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942306#comment-13942306
]
Colin Patrick McCabe commented on HDFS-6093:
{code}
With the locking issues resolved, is it okay to just leave it with a single set
of variables? I could switch it over to AtomicLongs or something, but I think
it's all under the FSN lock anyway.
{code}
I think putting more things under the big lock is the wrong direction to go.
In particular, we will eventually need to release the big lock from time to
time while doing the CacheReplicationMonitor scan. When we do that, having
just one set of counters is not going to work. It seems simple enough just to
have a {{CacheManager#Counters}} object with its own lock, and set it at the
end of the scan. There's other ways to do this too (atomics, etc.)
This would also make it easier to modify the pending cache count in
{{processCacheReportImpl}}. It's easy to understand the concept of modifying a
copy of the stats, harder to understand all the locking interactions of
modifying the counter that the CRM is actually using. At least for me.
With regard to the {{processCacheReportImpl}} changes, I think there are still
some issues here. I don't like the fact that we are now potentially allocating
a TreeMap of size NUM_PENDING_UNCACHED blocks in every cache report. There are
a few different ways to handle this without a huge memory blowup. The simplest
is probably to remove the "final" on {{DatanodeDescriptor#pendingUncached}}.
Then you just create a new list in {{processCacheReportImpl}}, and selectively
add the still-need-to-be-uncached blocks to that. Then at the end, you throw
away the old list and make {{DatanodeDescriptor}} use the new list.
+1 once all that is addressed
> Expose more caching information for debugging by users
> --
>
> Key: HDFS-6093
> URL: https://issues.apache.org/jira/browse/HDFS-6093
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: caching
>Affects Versions: 2.4.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-6093-1.patch, hdfs-6093-2.patch, hdfs-6093-3.patch,
> hdfs-6093-4.patch
>
>
> When users submit a new cache directive, it's unclear if the NN has
> recognized it and is actively trying to cache it, or if it's hung for some
> other reason. It'd be nice to expose a "pending caching/uncaching" count the
> same way we expose pending replication work.
> It'd also be nice to display the aggregate cache capacity and usage in
> dfsadmin -report, since we already have have it as a metric and expose it
> per-DN in report output.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6132) TestBlockToken fails on JDK 7
[ https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-6132: -- Status: Patch Available (was: Open) > TestBlockToken fails on JDK 7 > - > > Key: HDFS-6132 > URL: https://issues.apache.org/jira/browse/HDFS-6132 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li > Attachments: HDFS-6132.patch > > > Currently, UserGroupInformation.setConfiguration(conf) does not reset > loginUser. This is causing this test case to fail. > For now, the work around solution is to use UserGroupInformation.reset() > before UserGroupInformation.setConfiguration(conf). However, in general, we > should do this reset whenever we call > UserGroupInformation.setConfiguration(conf) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6132) TestBlockToken fails on JDK 7
[ https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-6132: -- Attachment: HDFS-6132.patch > TestBlockToken fails on JDK 7 > - > > Key: HDFS-6132 > URL: https://issues.apache.org/jira/browse/HDFS-6132 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li > Attachments: HDFS-6132.patch > > > Currently, UserGroupInformation.setConfiguration(conf) does not reset > loginUser. This is causing this test case to fail. > For now, the work around solution is to use UserGroupInformation.reset() > before UserGroupInformation.setConfiguration(conf). However, in general, we > should do this reset whenever we call > UserGroupInformation.setConfiguration(conf) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6132) TestBlockToken fails on JDK 7
[ https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-6132: -- Status: Open (was: Patch Available) > TestBlockToken fails on JDK 7 > - > > Key: HDFS-6132 > URL: https://issues.apache.org/jira/browse/HDFS-6132 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li > Attachments: HDFS-6132.patch > > > Currently, UserGroupInformation.setConfiguration(conf) does not reset > loginUser. This is causing this test case to fail. > For now, the work around solution is to use UserGroupInformation.reset() > before UserGroupInformation.setConfiguration(conf). However, in general, we > should do this reset whenever we call > UserGroupInformation.setConfiguration(conf) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6132) TestBlockToken fails on JDK 7
[ https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-6132: -- Attachment: (was: HDFS-6132.patch) > TestBlockToken fails on JDK 7 > - > > Key: HDFS-6132 > URL: https://issues.apache.org/jira/browse/HDFS-6132 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li > Attachments: HDFS-6132.patch > > > Currently, UserGroupInformation.setConfiguration(conf) does not reset > loginUser. This is causing this test case to fail. > For now, the work around solution is to use UserGroupInformation.reset() > before UserGroupInformation.setConfiguration(conf). However, in general, we > should do this reset whenever we call > UserGroupInformation.setConfiguration(conf) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6132) TestBlockToken fails on JDK 7
[ https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-6132: -- Description: Currently, UserGroupInformation.setConfiguration(conf) does not reset loginUser. This is causing this test case to fail. For now, the work around solution is to use UserGroupInformation.reset() before UserGroupInformation.setConfiguration(conf). However, in general, we should do this reset whenever we call UserGroupInformation.setConfiguration(conf) > TestBlockToken fails on JDK 7 > - > > Key: HDFS-6132 > URL: https://issues.apache.org/jira/browse/HDFS-6132 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li > Attachments: HDFS-6132.patch > > > Currently, UserGroupInformation.setConfiguration(conf) does not reset > loginUser. This is causing this test case to fail. > For now, the work around solution is to use UserGroupInformation.reset() > before UserGroupInformation.setConfiguration(conf). However, in general, we > should do this reset whenever we call > UserGroupInformation.setConfiguration(conf) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6132) TestBlockToken fails on JDK 7
[ https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-6132: -- Attachment: HDFS-6132.patch > TestBlockToken fails on JDK 7 > - > > Key: HDFS-6132 > URL: https://issues.apache.org/jira/browse/HDFS-6132 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li > Attachments: HDFS-6132.patch > > > Currently, UserGroupInformation.setConfiguration(conf) does not reset > loginUser. This is causing this test case to fail. > For now, the work around solution is to use UserGroupInformation.reset() > before UserGroupInformation.setConfiguration(conf). However, in general, we > should do this reset whenever we call > UserGroupInformation.setConfiguration(conf) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6132) TestBlockToken fails on JDK 7
[ https://issues.apache.org/jira/browse/HDFS-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-6132: -- Status: Patch Available (was: Open) > TestBlockToken fails on JDK 7 > - > > Key: HDFS-6132 > URL: https://issues.apache.org/jira/browse/HDFS-6132 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siqi Li > Attachments: HDFS-6132.patch > > > Currently, UserGroupInformation.setConfiguration(conf) does not reset > loginUser. This is causing this test case to fail. > For now, the work around solution is to use UserGroupInformation.reset() > before UserGroupInformation.setConfiguration(conf). However, in general, we > should do this reset whenever we call > UserGroupInformation.setConfiguration(conf) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6132) TestBlockToken fails on JDK 7
Siqi Li created HDFS-6132: - Summary: TestBlockToken fails on JDK 7 Key: HDFS-6132 URL: https://issues.apache.org/jira/browse/HDFS-6132 Project: Hadoop HDFS Issue Type: Bug Reporter: Siqi Li -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-6131. - Resolution: Fixed Fix Version/s: 2.4.0 Hadoop Flags: Reviewed Thanks very much to Arpit for the review! I've committed this to branch-2 and branch-2.4.0. > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.4.0 > > Attachments: HDFS-6131.000.patch, HDFS-6131.001.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5196) Provide more snapshot information in WebUI
[
https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942262#comment-13942262
]
Hadoop QA commented on HDFS-5196:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635829/HDFS-5196-7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.tools.TestJMXGet
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.TestFileCreation
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6453//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6453//console
This message is automatically generated.
> Provide more snapshot information in WebUI
> --
>
> Key: HDFS-5196
> URL: https://issues.apache.org/jira/browse/HDFS-5196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 3.0.0
>Reporter: Haohui Mai
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HDFS-5196-2.patch, HDFS-5196-3.patch, HDFS-5196-4.patch,
> HDFS-5196-5.patch, HDFS-5196-6.patch, HDFS-5196-7.patch, HDFS-5196.patch,
> HDFS-5196.patch, HDFS-5196.patch, snapshot-new-webui.png,
> snapshottable-directoryList.png, snapshotteddir.png
>
>
> The WebUI should provide more detailed information about snapshots, such as
> all snapshottable directories and corresponding number of snapshots
> (suggested in HDFS-4096).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942258#comment-13942258 ] Arpit Agarwal commented on HDFS-6131: - +1 for the updated patch, staged site and verified links work as expected. > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6131.000.patch, HDFS-6131.001.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6131: Attachment: HDFS-6131.001.patch > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6131.000.patch, HDFS-6131.001.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942212#comment-13942212 ] Jing Zhao commented on HDFS-6131: - Ahh, yes. Thanks Arpit! Let me update the patch. > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6131.000.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6130) NPE during namenode startup
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6130:
--
Summary: NPE during namenode startup (was: NPE during upgrade using trunk
after RU merged)
Fengdong, do you have HDFS-5988 in your build? If not, could you try again
with the latest version of trunk?
> This looks like not related with RU, ...
Let's revise the summary.
> NPE during namenode startup
> ---
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6093) Expose more caching information for debugging by users
[ https://issues.apache.org/jira/browse/HDFS-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942169#comment-13942169 ] Arpit Agarwal commented on HDFS-6093: - +1 from me, nice diagnosability improvements. > Expose more caching information for debugging by users > -- > > Key: HDFS-6093 > URL: https://issues.apache.org/jira/browse/HDFS-6093 > Project: Hadoop HDFS > Issue Type: Improvement > Components: caching >Affects Versions: 2.4.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-6093-1.patch, hdfs-6093-2.patch, hdfs-6093-3.patch, > hdfs-6093-4.patch > > > When users submit a new cache directive, it's unclear if the NN has > recognized it and is actively trying to cache it, or if it's hung for some > other reason. It'd be nice to expose a "pending caching/uncaching" count the > same way we expose pending replication work. > It'd also be nice to display the aggregate cache capacity and usage in > dfsadmin -report, since we already have have it as a metric and expose it > per-DN in report output. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[
https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942149#comment-13942149
]
Arpit Agarwal commented on HDFS-6131:
-
Hi Jing,
I think we also need to update the links in hadoop-project/src/site/site.xml.
{code}
Move HDFSHighAvailabilityWithNFS.apt.vm and
> HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
>
>
> Key: HDFS-6131
> URL: https://issues.apache.org/jira/browse/HDFS-6131
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.3.0
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-6131.000.patch
>
>
> Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and
> HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should
> move them to HDFS just like in trunk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[
https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942149#comment-13942149
]
Arpit Agarwal edited comment on HDFS-6131 at 3/20/14 7:06 PM:
--
Hi Jing,
I think we also need to update the links in hadoop-project/src/site/site.xml.
{code}
Move HDFSHighAvailabilityWithNFS.apt.vm and
> HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
>
>
> Key: HDFS-6131
> URL: https://issues.apache.org/jira/browse/HDFS-6131
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.3.0
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-6131.000.patch
>
>
> Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and
> HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should
> move them to HDFS just like in trunk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6120) Cleanup safe mode error messages
[
https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-6120:
Attachment: HDFS-6120.02.patch
Thanks Jing! Thanks for catching #1. I also updated {{getTurnOffTip}} to show
{{reached}} status by printing whether in extension.
> Cleanup safe mode error messages
>
>
> Key: HDFS-6120
> URL: https://issues.apache.org/jira/browse/HDFS-6120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6120.01.patch, HDFS-6120.02.patch
>
>
> In HA mode the SBN can enter safe-mode extension and stay there even after
> the extension period has elapsed but continue to return the safemode message
> stating that "The threshold has been reached" and "safe mode will be turned
> off soon".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6131: Status: Patch Available (was: Open) > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6131.000.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942128#comment-13942128 ] Jing Zhao commented on HDFS-6131: - Upload the patch for branch-2. > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6131.000.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6131: Attachment: HDFS-6131.000.patch > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6131.000.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6131: Status: Open (was: Patch Available) > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6131.000.patch > > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
[ https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao moved HADOOP-10415 to HDFS-6131: -- Component/s: (was: documentation) documentation Affects Version/s: (was: 2.3.0) 2.3.0 Key: HDFS-6131 (was: HADOOP-10415) Project: Hadoop HDFS (was: Hadoop Common) > Move HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS > > > Key: HDFS-6131 > URL: https://issues.apache.org/jira/browse/HDFS-6131 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.3.0 >Reporter: Jing Zhao >Assignee: Jing Zhao > > Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and > HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should > move them to HDFS just like in trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6130) NPE during upgrade using trunk after RU merged
[
https://issues.apache.org/jira/browse/HDFS-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942117#comment-13942117
]
Haohui Mai commented on HDFS-6130:
--
I believe that this is a duplicate of HDFS-5988.
> NPE during upgrade using trunk after RU merged
> --
>
> Key: HDFS-6130
> URL: https://issues.apache.org/jira/browse/HDFS-6130
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Fengdong Yu
>
> I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance,
> I can upgrade successfully if I don't configurage HA, but if HA enabled,
> there is NPE when I run ' hdfs namenode -initializeSharedEdits'
> {code}
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is
> enabled
> 14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
> total heap and retry cache entry expiry time is 60 millis
> 14/03/20 15:06:41 INFO util.GSet: Computing capacity for map
> NameNodeRetryCache
> 14/03/20 15:06:41 INFO util.GSet: VM type = 64-bit
> 14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB =
> 275.3 KB
> 14/03/20 15:06:41 INFO util.GSet: capacity = 2^15 = 32768 entries
> 14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
> 14/03/20 15:06:41 INFO common.Storage: Lock on
> /data/hadoop/data1/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO common.Storage: Lock on
> /data/hadoop/data2/dfs/name/in_use.lock acquired by nodename
> 7326@10-150-170-176
> 14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
> 14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
> 14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
> 14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
> 14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
> /
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5196) Provide more snapshot information in WebUI
[ https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated HDFS-5196: - Attachment: HDFS-5196-7.patch I attach the patch file which reflected former discussion. > Provide more snapshot information in WebUI > -- > > Key: HDFS-5196 > URL: https://issues.apache.org/jira/browse/HDFS-5196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Haohui Mai >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: HDFS-5196-2.patch, HDFS-5196-3.patch, HDFS-5196-4.patch, > HDFS-5196-5.patch, HDFS-5196-6.patch, HDFS-5196-7.patch, HDFS-5196.patch, > HDFS-5196.patch, HDFS-5196.patch, snapshot-new-webui.png, > snapshottable-directoryList.png, snapshotteddir.png > > > The WebUI should provide more detailed information about snapshots, such as > all snapshottable directories and corresponding number of snapshots > (suggested in HDFS-4096). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6089) Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended
[ https://issues.apache.org/jira/browse/HDFS-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942059#comment-13942059 ] Hudson commented on HDFS-6089: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5365 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5365/]) HDFS-6089. Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended. Contributed by Jing Zhao. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579692) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java > Standby NN while transitioning to active throws a connection refused error > when the prior active NN process is suspended > > > Key: HDFS-6089 > URL: https://issues.apache.org/jira/browse/HDFS-6089 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Arpit Gupta >Assignee: Jing Zhao > Fix For: 2.4.0 > > Attachments: HDFS-6089.000.patch, HDFS-6089.001.patch, > HDFS-6089.002.patch > > > The following scenario was tested: > * Determine Active NN and suspend the process (kill -19) > * Wait about 60s to let the standby transition to active > * Get the service state for nn1 and nn2 and make sure nn2 has transitioned to > active. > What was noticed that some times the call to get the service state of nn2 got > a socket time out exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6120) Cleanup safe mode error messages
[
https://issues.apache.org/jira/browse/HDFS-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942052#comment-13942052
]
Jing Zhao commented on HDFS-6120:
-
Thanks for the cleanup, Arpit! The patch looks good to me. Some minors:
# I guess here we should use "if (needEnter())" ?
{code}
+ if (!needEnter()) {
+reportStatus("STATE* Safe mode ON, thresholds not met.", false);
+ }
{code}
# For getTurnOffTip(), do you think we need an extra msg to report the value of
"reached" and the corresponding status (safe mode off, or on, or in extension)?
After the NN comes into the safemode extension, the safe block count may still
fall below the threshold.
> Cleanup safe mode error messages
>
>
> Key: HDFS-6120
> URL: https://issues.apache.org/jira/browse/HDFS-6120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6120.01.patch
>
>
> In HA mode the SBN can enter safe-mode extension and stay there even after
> the extension period has elapsed but continue to return the safemode message
> stating that "The threshold has been reached" and "safe mode will be turned
> off soon".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6089) Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended
[ https://issues.apache.org/jira/browse/HDFS-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6089: -- Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) Committed this back through 2.4, thanks again Jing for the patch and Arpit for the report. > Standby NN while transitioning to active throws a connection refused error > when the prior active NN process is suspended > > > Key: HDFS-6089 > URL: https://issues.apache.org/jira/browse/HDFS-6089 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Arpit Gupta >Assignee: Jing Zhao > Fix For: 2.4.0 > > Attachments: HDFS-6089.000.patch, HDFS-6089.001.patch, > HDFS-6089.002.patch > > > The following scenario was tested: > * Determine Active NN and suspend the process (kill -19) > * Wait about 60s to let the standby transition to active > * Get the service state for nn1 and nn2 and make sure nn2 has transitioned to > active. > What was noticed that some times the call to get the service state of nn2 got > a socket time out exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6089) Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended
[ https://issues.apache.org/jira/browse/HDFS-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942022#comment-13942022 ] Andrew Wang commented on HDFS-6089: --- +1 looks good to me. Will commit shortly, thanks Jing. > Standby NN while transitioning to active throws a connection refused error > when the prior active NN process is suspended > > > Key: HDFS-6089 > URL: https://issues.apache.org/jira/browse/HDFS-6089 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Arpit Gupta >Assignee: Jing Zhao > Attachments: HDFS-6089.000.patch, HDFS-6089.001.patch, > HDFS-6089.002.patch > > > The following scenario was tested: > * Determine Active NN and suspend the process (kill -19) > * Wait about 60s to let the standby transition to active > * Get the service state for nn1 and nn2 and make sure nn2 has transitioned to > active. > What was noticed that some times the call to get the service state of nn2 got > a socket time out exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6121) Support of "mount" onto HDFS directories
[ https://issues.apache.org/jira/browse/HDFS-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941936#comment-13941936 ] Chris Nauroth commented on HDFS-6121: - bq. This leaves room for random I/O between different threads. I seem to recall prior discussion of a "global I/O scheduler" idea. This would be something aware of all such threads (i.e. all of the map tasks on the same node trying to scan different HDFS blocks on the same physical disk), and it would try to maximize overall throughput by keeping one thread scanning forward while blocking the others. Could that potentially be a different solution to the problem? Unfortunately, I can't find any prior jiras or mailing list discussions related to this idea, so I don't know what details have been discussed already. There are of course potential negative side effects to consider, like unpredictable latency. I think this never got prioritized, because the problem hasn't caused a large impact in practice. (Doesn't mean we can't do it, just that it hasn't been prioritized.) > Support of "mount" onto HDFS directories > > > Key: HDFS-6121 > URL: https://issues.apache.org/jira/browse/HDFS-6121 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yan > > Currently, HDFS configuration can only create HDFS on one or several existing > local file system directories. This pretty much abstracts physical disk > drives away from HDFS users. > While it may provide conveniences in data > movement/manipulation/management/formatting, it could deprive users a way to > access physical disks in a more directly controlled manner. > For instance, a multi-threaded server may wish to access its disk blocks > sequentially per thread for fear of random I/O otherwise. If the cluster > boxes have multiple physical disk drives, and the server load is pretty much > I/O-bound, then it will be quite reasonable to hope for disk performance > typical of sequential I/O. Disk read-ahead and/or buffering at various layers > may alleviate the problem to some degree, but it couldn't totally eliminate > it. This could hurt hard performance of workloads than need to scan data. > Map/Reduce may experience the same problem as well. > For instance, HBase region servers may wish to scan disk data for each region > in a sequential way, again, to avoid random I/O. HBase incapability in this > regard aside, one major obstacle is with HDFS's incapability to specify > mappings of local directories to HDFS directories. Specifically, the > "dfs.data.dir" configuration setting only allows for the mapping from one or > multiple local directories to the HDFS root directory. In the case of data > nodes of multiple disk drives mounted as multiple local file system > directories per node, the HDFS data will be spread on all disk drives in a > pretty random manner, potentially resulting random I/O from a multi-threaded > server reading multiple data blocks from each thread. > A seemingly simple enhancement is an introduction of mappings from one or > multiple local FS directories to a single HDFS directory, plus necessary > sanity checks, replication policies, advices of best practices, ..., etc, of > course. Note that this should be an one-to-one or many-to-one mapping from > local to HDFS directories. The other way around, though probably feasible, > won't serve our purpose at all. This is similar to the mounting of different > disks onto different local FS directories, and will give the users an option > to place and access their data in a more controlled and efficient way. > Conceptually this option will allow for local physical data partition in a > distributed environment for application data on HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6129) When a replica is not found for deletion, do not throw exception.
[ https://issues.apache.org/jira/browse/HDFS-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941871#comment-13941871 ] Hudson commented on HDFS-6129: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5364 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5364/]) HDFS-6129. When a replica is not found for deletion, do not throw an exception. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579670) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java > When a replica is not found for deletion, do not throw exception. > - > > Key: HDFS-6129 > URL: https://issues.apache.org/jira/browse/HDFS-6129 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.4.0 > > Attachments: h6129_20140319.patch > > > It is actually a valid case if a replica is not found for deletion -- the > replica may be deleted earlier. So that we should not throw exceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6129) When a replica is not found for deletion, do not throw exception.
[ https://issues.apache.org/jira/browse/HDFS-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-6129: -- Resolution: Fixed Fix Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Fengdong and Kihwal for reviewing the patch. I have committed this. > When a replica is not found for deletion, do not throw exception. > - > > Key: HDFS-6129 > URL: https://issues.apache.org/jira/browse/HDFS-6129 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.4.0 > > Attachments: h6129_20140319.patch > > > It is actually a valid case if a replica is not found for deletion -- the > replica may be deleted earlier. So that we should not throw exceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6123) Improve datanode error messages
[
https://issues.apache.org/jira/browse/HDFS-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941802#comment-13941802
]
Hudson commented on HDFS-6123:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1732 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1732/])
HDFS-6123. Do not log stack trace for ReplicaAlreadyExistsException and
SocketTimeoutException. (szetszwo:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579396)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
> Improve datanode error messages
> ---
>
> Key: HDFS-6123
> URL: https://issues.apache.org/jira/browse/HDFS-6123
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 2.4.0
>
> Attachments: 6123_20140318.patch
>
>
> [~yeshavora] found two cases that there are unnecessary exception stack trace
> in datanode log:
> - SocketTimeoutException
> {noformat}
> 2014-03-07 03:30:44,567 INFO datanode.DataNode
> (BlockSender.java:sendPacket(563)) - exception:
> java.net.SocketTimeoutException: 48 millis timeout while waiting for
> channel to be ready for write. ch : java.nio.channels.SocketChannel[connected
> local=/xx.xx.xx.xx:1019 remote=/xx.xx.xx.xx:37997]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> ...
> {noformat}
> - ReplicaAlreadyExistsException
> {noformat}
> 2014-03-07 03:02:39,334 ERROR datanode.DataNode (DataXceiver.java:run(234)) -
> xx.xx.xx.xx:1019:DataXceiver error processing WRITE_BLOCK operation src:
> /xx.xx.xx.xx:32959 dest: /xx.xx.xx.xx:1019
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block
> BP-1409640778-xx.xx.xx.xx-1394150965191:blk_1073742158_1334 already exists in
> state TEMPORARY and thus cannot be created.
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:874)
> ...
> {noformat}
> Both cases are normal. They are not bugs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6127) WebHDFS tokens cannot be renewed in HA setup
[
https://issues.apache.org/jira/browse/HDFS-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941805#comment-13941805
]
Hudson commented on HDFS-6127:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1732 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1732/])
HDFS-6127. WebHDFS tokens cannot be renewed in HA setup. Contributed by Haohui
Mai. (wheat9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579546)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/TokenAspect.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/resources
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/resources/TestDatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
> WebHDFS tokens cannot be renewed in HA setup
>
>
> Key: HDFS-6127
> URL: https://issues.apache.org/jira/browse/HDFS-6127
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.4.0
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Fix For: 2.4.0
>
> Attachments: HDFS-6127.000.patch, HDFS-6127.001.patch
>
>
> {{TokenAspect}} class assumes that the service name of the token is alway a
> host-ip pair. In HA setup, however, the service name becomes the name service
> id, which breaks the assumption. As a result, WebHDFS tokens cannot be
> renewed in HA setup.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6100) DataNodeWebHdfsMethods does not failover in HA mode
[
https://issues.apache.org/jira/browse/HDFS-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941806#comment-13941806
]
Hudson commented on HDFS-6100:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1732 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1732/])
HDFS-6100. DataNodeWebHdfsMethods does not failover in HA mode. Contributed by
Haohui Mai. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579301)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/InetSocketAddressParam.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/NamenodeAddressParam.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/NamenodeRpcAddressParam.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> DataNodeWebHdfsMethods does not failover in HA mode
> ---
>
> Key: HDFS-6100
> URL: https://issues.apache.org/jira/browse/HDFS-6100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.4.0
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Fix For: 2.4.0
>
> Attachments: HDFS-6100.000.patch, HDFS-6100.001.patch
>
>
> In {{DataNodeWebHdfsMethods}}, the code creates a {{DFSClient}} to connect to
> the NN, so that it can access the files in the cluster.
> {{DataNodeWebHdfsMethods}} relies on the address passed in the URL to locate
> the NN. This implementation has two problems:
> # The {{DFSClient}} only knows about the current active NN, thus it does not
> support failover.
> # The delegation token is based on the active NN, therefore the {{DFSClient}}
> will fail to authenticate of the standby NN in secure HA setup.
> Currently the parameter {{namenoderpcaddress}} in the URL stores the host-ip
> pair that corresponds to the active NN. To fix this bug, this jira proposes
> to store the name service id in the parameter in HA setup (yet the parameter
> stays the same in non-HA setup).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6105) NN web UI for DN list loads the same jmx page multiple times.
[ https://issues.apache.org/jira/browse/HDFS-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941803#comment-13941803 ] Hudson commented on HDFS-6105: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1732 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1732/]) HDFS-6105. NN web UI for DN list loads the same jmx page multiple times. Contributed by Haohui Mai. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579468) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js > NN web UI for DN list loads the same jmx page multiple times. > - > > Key: HDFS-6105 > URL: https://issues.apache.org/jira/browse/HDFS-6105 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai > Fix For: 2.4.0 > > Attachments: HDFS-6105.000.patch, datanodes-tab.png > > > When loading "Datanodes" page of the NN web UI, the same jmx query is made > multiple times. For a big cluster, that's a lot of data and overhead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6100) DataNodeWebHdfsMethods does not failover in HA mode
[
https://issues.apache.org/jira/browse/HDFS-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941734#comment-13941734
]
Hudson commented on HDFS-6100:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1707 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1707/])
HDFS-6100. DataNodeWebHdfsMethods does not failover in HA mode. Contributed by
Haohui Mai. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579301)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/InetSocketAddressParam.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/NamenodeAddressParam.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/NamenodeRpcAddressParam.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> DataNodeWebHdfsMethods does not failover in HA mode
> ---
>
> Key: HDFS-6100
> URL: https://issues.apache.org/jira/browse/HDFS-6100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.4.0
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Fix For: 2.4.0
>
> Attachments: HDFS-6100.000.patch, HDFS-6100.001.patch
>
>
> In {{DataNodeWebHdfsMethods}}, the code creates a {{DFSClient}} to connect to
> the NN, so that it can access the files in the cluster.
> {{DataNodeWebHdfsMethods}} relies on the address passed in the URL to locate
> the NN. This implementation has two problems:
> # The {{DFSClient}} only knows about the current active NN, thus it does not
> support failover.
> # The delegation token is based on the active NN, therefore the {{DFSClient}}
> will fail to authenticate of the standby NN in secure HA setup.
> Currently the parameter {{namenoderpcaddress}} in the URL stores the host-ip
> pair that corresponds to the active NN. To fix this bug, this jira proposes
> to store the name service id in the parameter in HA setup (yet the parameter
> stays the same in non-HA setup).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6105) NN web UI for DN list loads the same jmx page multiple times.
[ https://issues.apache.org/jira/browse/HDFS-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941731#comment-13941731 ] Hudson commented on HDFS-6105: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1707 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1707/]) HDFS-6105. NN web UI for DN list loads the same jmx page multiple times. Contributed by Haohui Mai. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579468) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js > NN web UI for DN list loads the same jmx page multiple times. > - > > Key: HDFS-6105 > URL: https://issues.apache.org/jira/browse/HDFS-6105 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai > Fix For: 2.4.0 > > Attachments: HDFS-6105.000.patch, datanodes-tab.png > > > When loading "Datanodes" page of the NN web UI, the same jmx query is made > multiple times. For a big cluster, that's a lot of data and overhead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6127) WebHDFS tokens cannot be renewed in HA setup
[
https://issues.apache.org/jira/browse/HDFS-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941733#comment-13941733
]
Hudson commented on HDFS-6127:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1707 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1707/])
HDFS-6127. WebHDFS tokens cannot be renewed in HA setup. Contributed by Haohui
Mai. (wheat9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579546)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/TokenAspect.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/resources
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/resources/TestDatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
> WebHDFS tokens cannot be renewed in HA setup
>
>
> Key: HDFS-6127
> URL: https://issues.apache.org/jira/browse/HDFS-6127
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.4.0
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Fix For: 2.4.0
>
> Attachments: HDFS-6127.000.patch, HDFS-6127.001.patch
>
>
> {{TokenAspect}} class assumes that the service name of the token is alway a
> host-ip pair. In HA setup, however, the service name becomes the name service
> id, which breaks the assumption. As a result, WebHDFS tokens cannot be
> renewed in HA setup.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6123) Improve datanode error messages
[
https://issues.apache.org/jira/browse/HDFS-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941730#comment-13941730
]
Hudson commented on HDFS-6123:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1707 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1707/])
HDFS-6123. Do not log stack trace for ReplicaAlreadyExistsException and
SocketTimeoutException. (szetszwo:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579396)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
> Improve datanode error messages
> ---
>
> Key: HDFS-6123
> URL: https://issues.apache.org/jira/browse/HDFS-6123
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 2.4.0
>
> Attachments: 6123_20140318.patch
>
>
> [~yeshavora] found two cases that there are unnecessary exception stack trace
> in datanode log:
> - SocketTimeoutException
> {noformat}
> 2014-03-07 03:30:44,567 INFO datanode.DataNode
> (BlockSender.java:sendPacket(563)) - exception:
> java.net.SocketTimeoutException: 48 millis timeout while waiting for
> channel to be ready for write. ch : java.nio.channels.SocketChannel[connected
> local=/xx.xx.xx.xx:1019 remote=/xx.xx.xx.xx:37997]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> ...
> {noformat}
> - ReplicaAlreadyExistsException
> {noformat}
> 2014-03-07 03:02:39,334 ERROR datanode.DataNode (DataXceiver.java:run(234)) -
> xx.xx.xx.xx:1019:DataXceiver error processing WRITE_BLOCK operation src:
> /xx.xx.xx.xx:32959 dest: /xx.xx.xx.xx:1019
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block
> BP-1409640778-xx.xx.xx.xx-1394150965191:blk_1073742158_1334 already exists in
> state TEMPORARY and thus cannot be created.
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:874)
> ...
> {noformat}
> Both cases are normal. They are not bugs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6129) When a replica is not found for deletion, do not throw exception.
[ https://issues.apache.org/jira/browse/HDFS-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941716#comment-13941716 ] Kihwal Lee commented on HDFS-6129: -- bq. move v = after the 'if' block. v is also used for constructing the error message in the 'if' block. +1 the patch looks good. > When a replica is not found for deletion, do not throw exception. > - > > Key: HDFS-6129 > URL: https://issues.apache.org/jira/browse/HDFS-6129 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h6129_20140319.patch > > > It is actually a valid case if a replica is not found for deletion -- the > replica may be deleted earlier. So that we should not throw exceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941698#comment-13941698 ] Daryn Sharp commented on HDFS-4564: --- The httpfs test failure is caused by the dependency on HADOOP-9363. > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
