[jira] [Updated] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5291:
Attachment: HDFS-5291.002.patch
Add a new unit test to make sure client retry happens.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 2013-10-02 00:13:49,875 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recover
> {code}
> And then we get into safemode
> {code}
> Construction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,277 INFO BlockStateChange
> (BlockManager.java:logAddStoredBlock(2237)) - BLOCK* addStoredBlock: blockMap
> updated: IP:1019 is added to blk_IP157{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[IP:1019|RBW],
> ReplicaUnderConstruction[172.18.145.96:1019|RBW], ReplicaUnde
> rConstruction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,279 INFO hdfs.StateChange
> (FSNamesystem.java:reportStatus(4703)) - STATE* Safe mode ON.
>
[jira] [Commented] (HDFS-5257) addBlock() retry should return LocatedBlock with locations else client will get AIOBE
[
https://issues.apache.org/jira/browse/HDFS-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787967#comment-13787967
]
Hadoop QA commented on HDFS-5257:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607130/HDFS-5257.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5119//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5119//console
This message is automatically generated.
> addBlock() retry should return LocatedBlock with locations else client will
> get AIOBE
> -
>
> Key: HDFS-5257
> URL: https://issues.apache.org/jira/browse/HDFS-5257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Critical
> Attachments: HDFS-5257.patch, HDFS-5257.patch, HDFS-5257.patch
>
>
> {{addBlock()}} call retry should return the LocatedBlock with locations if
> the block was created in previous call and failover/restart of namenode
> happened.
> otherwise client will get {{ArrayIndexOutOfBoundsException}} while creating
> the block and write will fail.
> {noformat}java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1118)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:511){noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787976#comment-13787976
]
Vinay commented on HDFS-5291:
-
bq. This is seen in our nightlies where we see other services being impacted by
namenode being in safemode. In our tests we are killing the active namenode
every 5 mins and some times we see that after the transition from standby to
active the namenode goes into safemode.
In this case since failovers are frequent NN entering into safemode is
possible. In this case safemode is entering to extention after reading the
complete edits during failover.
I think, title of the issue is confusing here. May be we can change to "Client
needs to retry on startup SafeModeException" ..?
But One more query, if NN stays in safemode forever due to some missing blocks,
then also client have to retry till all failovers are complete?
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787989#comment-13787989
]
Hadoop QA commented on HDFS-5291:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607128/HDFS-5291.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5120//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5120//console
This message is automatically generated.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoo
[jira] [Commented] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[
https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788023#comment-13788023
]
Hadoop QA commented on HDFS-5283:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607132/HDFS-5283.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5121//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5121//console
This message is automatically generated.
> NN not coming out of startup safemode due to under construction blocks only
> inside snapshots also counted in safemode threshhold
>
>
> Key: HDFS-5283
> URL: https://issues.apache.org/jira/browse/HDFS-5283
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0, 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Blocker
> Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch,
> HDFS-5283.patch
>
>
> This is observed in one of our env:
> 1. A MR Job was running which has created some temporary files and was
> writing to them.
> 2. Snapshot was taken
> 3. And Job was killed and temporary files were deleted.
> 4. Namenode restarted.
> 5. After restart Namenode was in safemode waiting for blocks
> Analysis
> -
> 1. Since the snapshot taken also includes the temporary files which were
> open, and later original files are deleted.
> 2. UnderConstruction blocks count was taken from leases. not considered the
> UC blocks only inside snapshots
> 3. So safemode threshold count was more and NN did not come out of safemode
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5257) addBlock() retry should return LocatedBlock with locations else client will get AIOBE
[
https://issues.apache.org/jira/browse/HDFS-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788049#comment-13788049
]
Hadoop QA commented on HDFS-5257:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607131/HDFS-5257.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5122//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5122//console
This message is automatically generated.
> addBlock() retry should return LocatedBlock with locations else client will
> get AIOBE
> -
>
> Key: HDFS-5257
> URL: https://issues.apache.org/jira/browse/HDFS-5257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Vinay
>Assignee: Vinay
>Priority: Critical
> Attachments: HDFS-5257.patch, HDFS-5257.patch, HDFS-5257.patch
>
>
> {{addBlock()}} call retry should return the LocatedBlock with locations if
> the block was created in previous call and failover/restart of namenode
> happened.
> otherwise client will get {{ArrayIndexOutOfBoundsException}} while creating
> the block and write will fail.
> {noformat}java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1118)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:511){noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5259) Support client which combines appended data with old data before sends it to NFS server
[ https://issues.apache.org/jira/browse/HDFS-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788069#comment-13788069 ] Hudson commented on HDFS-5259: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #355 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/355/]) HDFS-5259. Support client which combines appended data with old data before sends it to NFS server. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529730) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/WRITE3Request.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Support client which combines appended data with old data before sends it to > NFS server > --- > > Key: HDFS-5259 > URL: https://issues.apache.org/jira/browse/HDFS-5259 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Yesha Vora >Assignee: Brandon Li > Fix For: 2.2.0 > > Attachments: HDFS-5259.000.patch, HDFS-5259.001.patch, > HDFS-5259.003.patch, HDFS-5259.004.patch, HDFS-5259.005.patch, > HDFS-5259.006.patch, HDFS-5259.007.patch, HDFS-5259.008.patch > > > The append does not work with some Linux client. The Client gets > "Input/output Error" when it tries to append. And NFS server considers it as > random write and fails the request. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788068#comment-13788068
]
Hudson commented on HDFS-4817:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #355 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/355/])
HDFS-4817. Moving changelog to Release 2.2.0 section to reflect the backport.
(acmurthy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529751)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> make HDFS advisory caching configurable on a per-file basis
> ---
>
> Key: HDFS-4817
> URL: https://issues.apache.org/jira/browse/HDFS-4817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
> HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch,
> HDFS-4817.008.patch, HDFS-4817.009.patch, HDFS-4817.010.patch,
> HDFS-4817-b2.1.001.patch
>
>
> HADOOP-7753 and related JIRAs introduced some performance optimizations for
> the DataNode. One of them was readahead. When readahead is enabled, the
> DataNode starts reading the next bytes it thinks it will need in the block
> file, before the client requests them. This helps hide the latency of
> rotational media and send larger reads down to the device. Another
> optimization was "drop-behind." Using this optimization, we could remove
> files from the Linux page cache after they were no longer needed.
> Using {{dfs.datanode.drop.cache.behind.writes}} and
> {{dfs.datanode.drop.cache.behind.reads}} can improve performance
> substantially on many MapReduce jobs. In our internal benchmarks, we have
> seen speedups of 40% on certain workloads. The reason is because if we know
> the block data will not be read again any time soon, keeping it out of memory
> allows more memory to be used by the other processes on the system. See
> HADOOP-7714 for more benchmarks.
> We would like to turn on these configurations on a per-file or per-client
> basis, rather than on the DataNode as a whole. This will allow more users to
> actually make use of them. It would also be good to add unit tests for the
> drop-cache code path, to ensure that it is functioning as we expect.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5299) DFS client hangs in updatePipeline RPC when failover happened
[ https://issues.apache.org/jira/browse/HDFS-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788065#comment-13788065 ] Hudson commented on HDFS-5299: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #355 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/355/]) HDFS-5299. DFS client hangs in updatePipeline RPC when failover happened. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529660) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java > DFS client hangs in updatePipeline RPC when failover happened > - > > Key: HDFS-5299 > URL: https://issues.apache.org/jira/browse/HDFS-5299 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Fix For: 2.2.0 > > Attachments: HDFS-5299.000.patch, HDFS-5299.patch > > > DFSClient got hanged in updatedPipeline call to namenode when the failover > happened at exactly sametime. > When we digged down, issue found to be with handling the RetryCache in > updatePipeline. > Here are the steps : > 1. Client was writing slowly. > 2. One of the datanode was down and updatePipeline was called to ANN. > 3. Call reached the ANN, while processing updatePipeline call it got shutdown. > 3. Now Client retried (Since the api marked as AtMostOnce) to another > NameNode. at that time still NN was in STANDBY and got StandbyException. > 4. Now one more time client failover happened. > 5. Now SNN became Active. > 6. Client called to current ANN again for updatePipeline, > Now client call got hanged in NN, waiting for the cached call with same > callid to be over. But this cached call is already got over last time with > StandbyException. > Conclusion : > Always whenever the new entry is added to cache we need to update the result > of the call before returning the call or throwing exception. > I can see similar issue multiple RPCs in FSNameSystem. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788100#comment-13788100
]
Hadoop QA commented on HDFS-5291:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607136/HDFS-5291.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5123//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5123//console
This message is automatically generated.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788137#comment-13788137
]
Hudson commented on HDFS-4817:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1545 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1545/])
HDFS-4817. Moving changelog to Release 2.2.0 section to reflect the backport.
(acmurthy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529751)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> make HDFS advisory caching configurable on a per-file basis
> ---
>
> Key: HDFS-4817
> URL: https://issues.apache.org/jira/browse/HDFS-4817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
> HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch,
> HDFS-4817.008.patch, HDFS-4817.009.patch, HDFS-4817.010.patch,
> HDFS-4817-b2.1.001.patch
>
>
> HADOOP-7753 and related JIRAs introduced some performance optimizations for
> the DataNode. One of them was readahead. When readahead is enabled, the
> DataNode starts reading the next bytes it thinks it will need in the block
> file, before the client requests them. This helps hide the latency of
> rotational media and send larger reads down to the device. Another
> optimization was "drop-behind." Using this optimization, we could remove
> files from the Linux page cache after they were no longer needed.
> Using {{dfs.datanode.drop.cache.behind.writes}} and
> {{dfs.datanode.drop.cache.behind.reads}} can improve performance
> substantially on many MapReduce jobs. In our internal benchmarks, we have
> seen speedups of 40% on certain workloads. The reason is because if we know
> the block data will not be read again any time soon, keeping it out of memory
> allows more memory to be used by the other processes on the system. See
> HADOOP-7714 for more benchmarks.
> We would like to turn on these configurations on a per-file or per-client
> basis, rather than on the DataNode as a whole. This will allow more users to
> actually make use of them. It would also be good to add unit tests for the
> drop-cache code path, to ensure that it is functioning as we expect.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5299) DFS client hangs in updatePipeline RPC when failover happened
[ https://issues.apache.org/jira/browse/HDFS-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788134#comment-13788134 ] Hudson commented on HDFS-5299: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1545 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1545/]) HDFS-5299. DFS client hangs in updatePipeline RPC when failover happened. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529660) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java > DFS client hangs in updatePipeline RPC when failover happened > - > > Key: HDFS-5299 > URL: https://issues.apache.org/jira/browse/HDFS-5299 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Fix For: 2.2.0 > > Attachments: HDFS-5299.000.patch, HDFS-5299.patch > > > DFSClient got hanged in updatedPipeline call to namenode when the failover > happened at exactly sametime. > When we digged down, issue found to be with handling the RetryCache in > updatePipeline. > Here are the steps : > 1. Client was writing slowly. > 2. One of the datanode was down and updatePipeline was called to ANN. > 3. Call reached the ANN, while processing updatePipeline call it got shutdown. > 3. Now Client retried (Since the api marked as AtMostOnce) to another > NameNode. at that time still NN was in STANDBY and got StandbyException. > 4. Now one more time client failover happened. > 5. Now SNN became Active. > 6. Client called to current ANN again for updatePipeline, > Now client call got hanged in NN, waiting for the cached call with same > callid to be over. But this cached call is already got over last time with > StandbyException. > Conclusion : > Always whenever the new entry is added to cache we need to update the result > of the call before returning the call or throwing exception. > I can see similar issue multiple RPCs in FSNameSystem. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5259) Support client which combines appended data with old data before sends it to NFS server
[ https://issues.apache.org/jira/browse/HDFS-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788138#comment-13788138 ] Hudson commented on HDFS-5259: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1545 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1545/]) HDFS-5259. Support client which combines appended data with old data before sends it to NFS server. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529730) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/WRITE3Request.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Support client which combines appended data with old data before sends it to > NFS server > --- > > Key: HDFS-5259 > URL: https://issues.apache.org/jira/browse/HDFS-5259 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Yesha Vora >Assignee: Brandon Li > Fix For: 2.2.0 > > Attachments: HDFS-5259.000.patch, HDFS-5259.001.patch, > HDFS-5259.003.patch, HDFS-5259.004.patch, HDFS-5259.005.patch, > HDFS-5259.006.patch, HDFS-5259.007.patch, HDFS-5259.008.patch > > > The append does not work with some Linux client. The Client gets > "Input/output Error" when it tries to append. And NFS server considers it as > random write and fails the request. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5315) [HDFS-Snapshot] Return code of SnapshotDiff of two invalid snapshots is 0
Brahma Reddy Battula created HDFS-5315: -- Summary: [HDFS-Snapshot] Return code of SnapshotDiff of two invalid snapshots is 0 Key: HDFS-5315 URL: https://issues.apache.org/jira/browse/HDFS-5315 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5315) [HDFS-Snapshot] Return code of SnapshotDiff of two invalid snapshots is 0
[
https://issues.apache.org/jira/browse/HDFS-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788145#comment-13788145
]
Brahma Reddy Battula commented on HDFS-5315:
This jira is dupliucate of following
https://issues.apache.org/jira/browse/HDFS-4759
As even after above defect fix return code will be "0", since we are not doing
"System.exit(1);" for following..
{code}
catch (IOException e) {
String[] content = e.getLocalizedMessage().split("\n");
System.err.println("snapshotDiff: " + content[0]);
}
{code}
> [HDFS-Snapshot] Return code of SnapshotDiff of two invalid snapshots is 0
> -
>
> Key: HDFS-5315
> URL: https://issues.apache.org/jira/browse/HDFS-5315
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4759) snapshotDiff of two invalid snapshots but with same name returns success
[ https://issues.apache.org/jira/browse/HDFS-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788146#comment-13788146 ] Brahma Reddy Battula commented on HDFS-4759: Even after defect fix return code will be Zero(for the mentioned scenario), since we are not returning the return code in following catch exception.. Hence handling the as part of the https://issues.apache.org/jira/i#browse/HDFS-5315 > snapshotDiff of two invalid snapshots but with same name returns success > > > Key: HDFS-4759 > URL: https://issues.apache.org/jira/browse/HDFS-4759 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Ramya Sunil >Assignee: Jing Zhao >Priority: Minor > Fix For: Snapshot (HDFS-2802) > > Attachments: HDFS-4759.001.patch > > > snapshotDiff of two invalid snapshots which have the same names returns a > success. > $ hadoop dfs -ls /user/foo/hdfs-snapshots/.snapshot > Found 1 items > drwx-- - foo foo 0 2013-04-26 00:53 > /user/foo/hdfs-snapshots/.snapshot/s1 > $ hadoop snapshotDiff /user/foo/hdfs-snapshots invalid invalid > Difference between snapshot invalid and snapshot invalid under directory > /user/foo/hdfs-snapshots: > -bash-4.1$ echo $? > 0 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5299) DFS client hangs in updatePipeline RPC when failover happened
[ https://issues.apache.org/jira/browse/HDFS-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788174#comment-13788174 ] Hudson commented on HDFS-5299: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1571 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1571/]) HDFS-5299. DFS client hangs in updatePipeline RPC when failover happened. Contributed by Vinay. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529660) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java > DFS client hangs in updatePipeline RPC when failover happened > - > > Key: HDFS-5299 > URL: https://issues.apache.org/jira/browse/HDFS-5299 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Fix For: 2.2.0 > > Attachments: HDFS-5299.000.patch, HDFS-5299.patch > > > DFSClient got hanged in updatedPipeline call to namenode when the failover > happened at exactly sametime. > When we digged down, issue found to be with handling the RetryCache in > updatePipeline. > Here are the steps : > 1. Client was writing slowly. > 2. One of the datanode was down and updatePipeline was called to ANN. > 3. Call reached the ANN, while processing updatePipeline call it got shutdown. > 3. Now Client retried (Since the api marked as AtMostOnce) to another > NameNode. at that time still NN was in STANDBY and got StandbyException. > 4. Now one more time client failover happened. > 5. Now SNN became Active. > 6. Client called to current ANN again for updatePipeline, > Now client call got hanged in NN, waiting for the cached call with same > callid to be over. But this cached call is already got over last time with > StandbyException. > Conclusion : > Always whenever the new entry is added to cache we need to update the result > of the call before returning the call or throwing exception. > I can see similar issue multiple RPCs in FSNameSystem. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788177#comment-13788177
]
Hudson commented on HDFS-4817:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1571 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1571/])
HDFS-4817. Moving changelog to Release 2.2.0 section to reflect the backport.
(acmurthy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529751)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> make HDFS advisory caching configurable on a per-file basis
> ---
>
> Key: HDFS-4817
> URL: https://issues.apache.org/jira/browse/HDFS-4817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
> HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch,
> HDFS-4817.008.patch, HDFS-4817.009.patch, HDFS-4817.010.patch,
> HDFS-4817-b2.1.001.patch
>
>
> HADOOP-7753 and related JIRAs introduced some performance optimizations for
> the DataNode. One of them was readahead. When readahead is enabled, the
> DataNode starts reading the next bytes it thinks it will need in the block
> file, before the client requests them. This helps hide the latency of
> rotational media and send larger reads down to the device. Another
> optimization was "drop-behind." Using this optimization, we could remove
> files from the Linux page cache after they were no longer needed.
> Using {{dfs.datanode.drop.cache.behind.writes}} and
> {{dfs.datanode.drop.cache.behind.reads}} can improve performance
> substantially on many MapReduce jobs. In our internal benchmarks, we have
> seen speedups of 40% on certain workloads. The reason is because if we know
> the block data will not be read again any time soon, keeping it out of memory
> allows more memory to be used by the other processes on the system. See
> HADOOP-7714 for more benchmarks.
> We would like to turn on these configurations on a per-file or per-client
> basis, rather than on the DataNode as a whole. This will allow more users to
> actually make use of them. It would also be good to add unit tests for the
> drop-cache code path, to ensure that it is functioning as we expect.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5259) Support client which combines appended data with old data before sends it to NFS server
[ https://issues.apache.org/jira/browse/HDFS-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788178#comment-13788178 ] Hudson commented on HDFS-5259: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1571 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1571/]) HDFS-5259. Support client which combines appended data with old data before sends it to NFS server. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1529730) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/request/WRITE3Request.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Support client which combines appended data with old data before sends it to > NFS server > --- > > Key: HDFS-5259 > URL: https://issues.apache.org/jira/browse/HDFS-5259 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Yesha Vora >Assignee: Brandon Li > Fix For: 2.2.0 > > Attachments: HDFS-5259.000.patch, HDFS-5259.001.patch, > HDFS-5259.003.patch, HDFS-5259.004.patch, HDFS-5259.005.patch, > HDFS-5259.006.patch, HDFS-5259.007.patch, HDFS-5259.008.patch > > > The append does not work with some Linux client. The Client gets > "Input/output Error" when it tries to append. And NFS server considers it as > random write and fails the request. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5294) DistributedFileSystem#getFileLinkStatus should not fully qualify the link target
[
https://issues.apache.org/jira/browse/HDFS-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788190#comment-13788190
]
Daryn Sharp commented on HDFS-5294:
---
Yes, that's correct. The symlink target should simply be returned as-is.
Agreed that it should be the behavior for all filesystems.
> DistributedFileSystem#getFileLinkStatus should not fully qualify the link
> target
>
>
> Key: HDFS-5294
> URL: https://issues.apache.org/jira/browse/HDFS-5294
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>
> The NN returns a {{FileStatus}} containing the exact link target as specified
> by the user at creation. However,
> {{DistributedFileSystem#getFileLinkStatus}} explicit overwrites the target
> with the fully scheme qualified path lookup. This causes multiple issues
> such as:
> # Prevents clients from discerning if the target is relative or absolute
> # Mangles a target that is not intended to be a path
> # Causes incorrect resolution with multi-layered filesystems - ie. the link
> should be resolved relative to a higher level fs (ie. viewfs, chroot,
> filtered, etc)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4531) Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788223#comment-13788223 ] Ravi Prakash commented on HDFS-4531: We've agreed to exclude the hamlet code from code coverage results as part HADOOP-9494, so I am going to close this JIRA since its not needed any more. Thanks Vadim > Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests > -- > > Key: HDFS-4531 > URL: https://issues.apache.org/jira/browse/HDFS-4531 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Vadim Bondarev > Attachments: HADOOP-4531-branch-0.23-a.patch, > HADOOP-4531-branch-2-a.patch, HADOOP-4531-trunk-a.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4531) Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-4531: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) > Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests > -- > > Key: HDFS-4531 > URL: https://issues.apache.org/jira/browse/HDFS-4531 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Vadim Bondarev > Attachments: HADOOP-4531-branch-0.23-a.patch, > HADOOP-4531-branch-2-a.patch, HADOOP-4531-trunk-a.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4531) Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788240#comment-13788240 ] Ivan A. Veselovsky commented on HDFS-4531: -- Hi, Ravi, in the last comment of (https://issues.apache.org/jira/browse/HADOOP-9494?focusedCommentId=13786558&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13786558) Jonathan E. said: **/hamlet/*.java not sure this should be excluded since it is not generated. Remove this. As I understand, he means to remove the exclude, that is, to calculate the code coverage for hamlet package. Am I correct? > Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests > -- > > Key: HDFS-4531 > URL: https://issues.apache.org/jira/browse/HDFS-4531 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Vadim Bondarev > Attachments: HADOOP-4531-branch-0.23-a.patch, > HADOOP-4531-branch-2-a.patch, HADOOP-4531-trunk-a.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788264#comment-13788264
]
Suresh Srinivas commented on HDFS-5291:
---
Some comments:
# RetriableException - javadoc could say along the lines of "Exception thrown
by a server typically to indicate that server is in a state where request
cannot be processed temporarily (such as still starting up). Client may retry
the request. If server reaches a state whrere the service is up, one of the
retries may succeed.
# Why add getRetryDecision() to RetriableException? Do you foresee adding other
kinds of retries later? Also any exception member variables cannot be
communicated over RPC.
# checkNamenodeSafeMode could just be named checkSafeMode()? Also @throws
should be followed by the excetion name. Lets add specific exception to throws
clause instead of generic IOException.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedT
[jira] [Comment Edited] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788264#comment-13788264
]
Suresh Srinivas edited comment on HDFS-5291 at 10/7/13 4:04 PM:
Some comments:
# RetriableException - javadoc could say along the lines of "Exception thrown
by a server typically to indicate that server is in a state where request
cannot be processed temporarily (such as still starting up). Client may retry
the request. If the service is up, the server may be able to process a retried
request.
# Why add getRetryDecision() to RetriableException? Do you foresee adding other
kinds of retries later? Also any exception member variables cannot be
communicated over RPC.
# checkNamenodeSafeMode could just be named checkSafeMode()? Also @throws
should be followed by the excetion name. Lets add specific exception to throws
clause instead of generic IOException.
was (Author: sureshms):
Some comments:
# RetriableException - javadoc could say along the lines of "Exception thrown
by a server typically to indicate that server is in a state where request
cannot be processed temporarily (such as still starting up). Client may retry
the request. If server reaches a state whrere the service is up, one of the
retries may succeed.
# Why add getRetryDecision() to RetriableException? Do you foresee adding other
kinds of retries later? Also any exception member variables cannot be
communicated over RPC.
# checkNamenodeSafeMode could just be named checkSafeMode()? Also @throws
should be followed by the excetion name. Lets add specific exception to throws
clause instead of generic IOException.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientPr
[jira] [Commented] (HDFS-4531) Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788288#comment-13788288 ] Ravi Prakash commented on HDFS-4531: Hi Ivan! I'm sorry if that is the case, but the consensus here in the team is that time would be better spent unit testing other stuff. We will be letting that exclude stay. > Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests > -- > > Key: HDFS-4531 > URL: https://issues.apache.org/jira/browse/HDFS-4531 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Vadim Bondarev > Attachments: HADOOP-4531-branch-0.23-a.patch, > HADOOP-4531-branch-2-a.patch, HADOOP-4531-trunk-a.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4531) Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788297#comment-13788297 ] Ivan A. Veselovsky commented on HDFS-4531: -- Ok, thanks for the clarification. > Cover package org.apache.hadoop.yarn.webapp.hamlet with unit tests > -- > > Key: HDFS-4531 > URL: https://issues.apache.org/jira/browse/HDFS-4531 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Vadim Bondarev > Attachments: HADOOP-4531-branch-0.23-a.patch, > HADOOP-4531-branch-2-a.patch, HADOOP-4531-trunk-a.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5291:
Attachment: HDFS-5291.003.patch
Thanks for the review Suresh! Update the patch to address your comments.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 2013-10-02 00:13:49,875 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recover
> {code}
> And then we get into safemode
> {code}
> Construction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,277 INFO BlockStateChange
> (BlockManager.java:logAddStoredBlock(2237)) - BLOCK* addStoredBlock: blockMap
> updated: IP:1019 is added to blk_IP157{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[IP:1019|RBW],
> ReplicaUnderConstruction[172.18.145.96:1019|RBW], ReplicaUnde
> rConstruction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,279 INFO hdfs.StateChange
> (FSNamesystem.java:report
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788353#comment-13788353
]
Jing Zhao commented on HDFS-5291:
-
bq. checkNamenodeSafeMode could just be named checkSafeMode()
There is already a method named checkSafeMode in FSNamesystem.java. So I name
the new method as checkNameNodeSafeMode to distinguish with the old one.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 2013-10-02 00:13:49,875 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recover
> {code}
> And then we get into safemode
> {code}
> Construction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,277 INFO BlockStateChange
> (BlockManager.java:logAddStoredBlock(2237)) - BLOCK* addStoredBlock: blockMap
> updated: IP:1019 is added to blk_IP157{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[IP:1019|RBW],
> ReplicaUnderCon
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788371#comment-13788371
]
Andrew Wang commented on HDFS-5224:
---
Hey Chris, thanks for picking this up,
I agree with your thinking here. We shouldn't be using a {{Path}} to the
server-side, since we're supposed to handle {{Pathy}}-y things like the scheme
and relative paths on the client. The NN should only be seeing absolute path
strings.
Adding some new classes dealing in strings is likely the right call, though it
does unfortunately increase the class-bloat. It's already kind of complicated
having the similar-ish {{Directives}}, {{Descriptors}}, and {{Entries}}, and
now we'll have 2 more (though both internal to HDFS). Internal complexity we
can deal with, but simplifying all of this is an end-goal of mine.
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788386#comment-13788386
]
Suresh Srinivas commented on HDFS-5291:
---
+1 for the patch.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 2013-10-02 00:13:49,875 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recover
> {code}
> And then we get into safemode
> {code}
> Construction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,277 INFO BlockStateChange
> (BlockManager.java:logAddStoredBlock(2237)) - BLOCK* addStoredBlock: blockMap
> updated: IP:1019 is added to blk_IP157{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[IP:1019|RBW],
> ReplicaUnderConstruction[172.18.145.96:1019|RBW], ReplicaUnde
> rConstruction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,279 INFO hdfs.StateChange
> (FSNamesystem.java:reportStatus(4703)) - STATE*
[jira] [Created] (HDFS-5316) Namenode ignore the default https port
Suresh Srinivas created HDFS-5316:
-
Summary: Namenode ignore the default https port
Key: HDFS-5316
URL: https://issues.apache.org/jira/browse/HDFS-5316
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Suresh Srinivas
When dfs.https.enable is true and dfs.https.port is not configured, namenode
does not pickup the default https port (50470), instead picks up random port.
See:
{code}
boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
":" + conf.get(
DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
Configuration sslConf = new Configuration(false);
if (certSSL) {
sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
"ssl-server.xml"));
}
{code}
Unless https port is specifically configured as 0, we should not be picking
random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5316) Namenode ignores the default https port
[
https://issues.apache.org/jira/browse/HDFS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-5316:
--
Affects Version/s: 2.0.0-alpha
> Namenode ignores the default https port
> ---
>
> Key: HDFS-5316
> URL: https://issues.apache.org/jira/browse/HDFS-5316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha
>Reporter: Suresh Srinivas
>
> When dfs.https.enable is true and dfs.https.port is not configured, namenode
> does not pickup the default https port (50470), instead picks up random port.
> See:
> {code}
> boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
> InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
> ":" + conf.get(
> DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
> Configuration sslConf = new Configuration(false);
> if (certSSL) {
>
> sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
> "ssl-server.xml"));
> }
> {code}
> Unless https port is specifically configured as 0, we should not be picking
> random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5316) Namenode ignores the default https port
[
https://issues.apache.org/jira/browse/HDFS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-5316:
--
Summary: Namenode ignores the default https port (was: Namenode ignore the
default https port)
> Namenode ignores the default https port
> ---
>
> Key: HDFS-5316
> URL: https://issues.apache.org/jira/browse/HDFS-5316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha
>Reporter: Suresh Srinivas
>
> When dfs.https.enable is true and dfs.https.port is not configured, namenode
> does not pickup the default https port (50470), instead picks up random port.
> See:
> {code}
> boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
> InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
> ":" + conf.get(
> DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
> Configuration sslConf = new Configuration(false);
> if (certSSL) {
>
> sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
> "ssl-server.xml"));
> }
> {code}
> Unless https port is specifically configured as 0, we should not be picking
> random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5316) Namenode ignores the default https port
[
https://issues.apache.org/jira/browse/HDFS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-5316:
--
Target Version/s: 2.2.0
> Namenode ignores the default https port
> ---
>
> Key: HDFS-5316
> URL: https://issues.apache.org/jira/browse/HDFS-5316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha
>Reporter: Suresh Srinivas
>Priority: Critical
>
> When dfs.https.enable is true and dfs.https.port is not configured, namenode
> does not pickup the default https port (50470), instead picks up random port.
> See:
> {code}
> boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
> InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
> ":" + conf.get(
> DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
> Configuration sslConf = new Configuration(false);
> if (certSSL) {
>
> sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
> "ssl-server.xml"));
> }
> {code}
> Unless https port is specifically configured as 0, we should not be picking
> random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5316) Namenode ignores the default https port
[
https://issues.apache.org/jira/browse/HDFS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-5316:
--
Priority: Critical (was: Major)
> Namenode ignores the default https port
> ---
>
> Key: HDFS-5316
> URL: https://issues.apache.org/jira/browse/HDFS-5316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha
>Reporter: Suresh Srinivas
>Priority: Critical
>
> When dfs.https.enable is true and dfs.https.port is not configured, namenode
> does not pickup the default https port (50470), instead picks up random port.
> See:
> {code}
> boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
> InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
> ":" + conf.get(
> DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
> Configuration sslConf = new Configuration(false);
> if (certSSL) {
>
> sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
> "ssl-server.xml"));
> }
> {code}
> Unless https port is specifically configured as 0, we should not be picking
> random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5303) A symlink within a snapshot pointing to a target outside the snapshot root can cause the snapshot contents to appear to change.
[ https://issues.apache.org/jira/browse/HDFS-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788413#comment-13788413 ] Colin Patrick McCabe commented on HDFS-5303: bq. With a whole-volume snapshot, I believe the user still can assume immutability as long as the symlinks don't jump out of the volume. But symlinks can jump out of the volume. Symlinks on ZFS can point anywhere. bq. At this point, I propose that the scope of this jira is to update the snapshot documentation to warn end users about the pitfalls of combining snapshots and symlinks. (No code changes.) agree > A symlink within a snapshot pointing to a target outside the snapshot root > can cause the snapshot contents to appear to change. > --- > > Key: HDFS-5303 > URL: https://issues.apache.org/jira/browse/HDFS-5303 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Chris Nauroth > > A snapshot is supposed to represent the point-in-time state of a directory. > However, if the directory contains a symlink that targets a file outside the > snapshot root, then the snapshot contents will appear to change if someone > changes the target file (i.e. delete or append). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5307) Support both HTTP and HTTPS in jsp pages
[ https://issues.apache.org/jira/browse/HDFS-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788414#comment-13788414 ] Brandon Li commented on HDFS-5307: -- +1. The patch looks good. > Support both HTTP and HTTPS in jsp pages > > > Key: HDFS-5307 > URL: https://issues.apache.org/jira/browse/HDFS-5307 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5307.000.patch > > > Adopt the new APIs provided by HDFS-5306 to support both HTTP and HTTPS in > the jsp pages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
Suresh Srinivas created HDFS-5317: - Summary: Go back to DFS Home link does not work on datanode webUI Key: HDFS-5317 URL: https://issues.apache.org/jira/browse/HDFS-5317 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Priority: Critical Here is how to reproduce this problem: # Goto https://namenode:https_port/dfshealth.jsp # Click on "Live Nodes" link # On the Live Nodes page, click one of the links for the datanodes. This link has always hardcodes the http link instead of http/https based on the context of the webpage. On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5313) NameNode hangs during startup trying to apply OP_ADD_PATH_BASED_CACHE_DIRECTIVE.
[
https://issues.apache.org/jira/browse/HDFS-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788417#comment-13788417
]
Colin Patrick McCabe commented on HDFS-5313:
This code path is changing in HDFS-5096 to fix this and other issues, so we
should probably dupe this JIRA to that one.
> NameNode hangs during startup trying to apply
> OP_ADD_PATH_BASED_CACHE_DIRECTIVE.
>
>
> Key: HDFS-5313
> URL: https://issues.apache.org/jira/browse/HDFS-5313
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>
> During namenode startup, if the edits contain a
> {{OP_ADD_PATH_BASED_CACHE_DIRECTIVE}} for an existing file, then the process
> hangs while trying to apply the op. This is because of a call to
> {{FSDirectory#setCacheReplication}}, which calls
> {{FSDirectory#waitForReady}}, but of course nothing is ever going to mark the
> directory ready, because it's still in the process of loading.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788428#comment-13788428
]
Colin Patrick McCabe commented on HDFS-5224:
Thanks for looking at this, Chris.
I don't think we should create another class. We already have
{{PathBasedCacheDirective}} (client-facing), {{PathBasedCacheDescriptor}} (a
directive with an ID associated), and {{PathBasedCacheEntry}} (server-side).
Adding more classes would just be too many!
It would make more sense to add a method like
{{DistributedFileSystem#createPathBasedCacheDirective(Path path)}} which
returns a {{PathBasedCacheDirective}}. Keep in mind that we want the
{{DistributedFileSystem}} to supply its current working directory, and validate
that the path matches its URI. Then, we should make the fields in
{{PathBasedCacheDirective}} mutable and provide setters, so clients can do
something like this:
{code}
fs.addPathBasedCacheDirective(fs.createPathBasedCacheDirective(new
Path("/foo/bar")).
setReplication(2).
setPool("pool1")));
{code}
This way:
* we don't have more class bloat
* DFS provides its working directory and validates the path
* it's easy to see what's being set
* we can add more fields to the directive later without breaking compatibility
or getting into constructor overload hell
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HDFS-5316) Namenode ignores the default https port
[
https://issues.apache.org/jira/browse/HDFS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai reassigned HDFS-5316:
Assignee: Haohui Mai
> Namenode ignores the default https port
> ---
>
> Key: HDFS-5316
> URL: https://issues.apache.org/jira/browse/HDFS-5316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha
>Reporter: Suresh Srinivas
>Assignee: Haohui Mai
>Priority: Critical
>
> When dfs.https.enable is true and dfs.https.port is not configured, namenode
> does not pickup the default https port (50470), instead picks up random port.
> See:
> {code}
> boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
> InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
> ":" + conf.get(
> DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
> Configuration sslConf = new Configuration(false);
> if (certSSL) {
>
> sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
> "ssl-server.xml"));
> }
> {code}
> Unless https port is specifically configured as 0, we should not be picking
> random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-5317: Assignee: Haohui Mai > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5292) clean up output of `dfs -du -s`
[
https://issues.apache.org/jira/browse/HDFS-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788436#comment-13788436
]
Chris Nauroth commented on HDFS-5292:
-
Hi, [~ndimiduk]. Do you have any specific ideas in mind on what you'd like to
see in the output? More Unix-like du options like -h for human-readable and -c
for a total? Something else? Thanks!
> clean up output of `dfs -du -s`
> ---
>
> Key: HDFS-5292
> URL: https://issues.apache.org/jira/browse/HDFS-5292
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.1.1-beta
>Reporter: Nick Dimiduk
>Priority: Minor
>
> This could be formatted a little nicer:
> {noformat}
> $ hdfs dfs -du -s /apps/hbase/data/data/default/*
> 22604541341 /apps/hbase/data/data/default/IntegrationTestBulkLoad
> 896656491 /apps/hbase/data/data/default/IntegrationTestIngest
> 33776145312 /apps/hbase/data/data/default/IntegrationTestLoadAndVerify
> 83512463 /apps/hbase/data/data/default/SendTracesTable
> 532898 /apps/hbase/data/data/default/TestAcidGuarantees
> 27294 /apps/hbase/data/data/default/demo_table
> 1410 /apps/hbase/data/data/default/example
> 2531532801 /apps/hbase/data/data/default/loadtest_d1
> 901 /apps/hbase/data/data/default/table_qho71mpvj8
> 1433 /apps/hbase/data/data/default/tcreatetbl
> 1690 /apps/hbase/data/data/default/tdelrowtbl
> 360 /apps/hbase/data/data/default/testtbl1
> 360 /apps/hbase/data/data/default/testtbl2
> 360 /apps/hbase/data/data/default/testtbl3
> 1515 /apps/hbase/data/data/default/tquerytbl
> 1513 /apps/hbase/data/data/default/tscantbl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5307) Support both HTTP and HTTPS in jsp pages
[ https://issues.apache.org/jira/browse/HDFS-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788443#comment-13788443 ] Brandon Li commented on HDFS-5307: -- I've committed the patch. Thank you, Haohui, for the contribution! > Support both HTTP and HTTPS in jsp pages > > > Key: HDFS-5307 > URL: https://issues.apache.org/jira/browse/HDFS-5307 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5307.000.patch > > > Adopt the new APIs provided by HDFS-5306 to support both HTTP and HTTPS in > the jsp pages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5292) clean up output of `dfs -du -s`
[
https://issues.apache.org/jira/browse/HDFS-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788449#comment-13788449
]
Nick Dimiduk commented on HDFS-5292:
The jagged output is what caught my attention as it differs from the unix-like
rendering.
{noformat}
$ du -s ./*
768 ./CHANGES.txt
8 ./HDP-CHANGES.txt
24 ./LICENSE.txt
8 ./NOTICE.txt
8 ./README.txt
328 ./bin
64 ./conf
424 ./dev-support
176 ./hbase-assembly
14184 ./hbase-client
8112./hbase-common
6368./hbase-examples
1000./hbase-hadoop-compat
1616./hbase-hadoop1-compat
1664./hbase-hadoop2-compat
2120./hbase-hbase-master-hor17n34.gq1.ygridcore.net.log.gz
3104./hbase-it
2632./hbase-prefix-tree
46904 ./hbase-protocol
88432 ./hbase-server
1344./hbase-shell
16096 ./hbase-thrift
2872./logs
152 ./pom.xml
10184 ./small1-run-hdp2.0.tgz
10480 ./small2-run-hdp1.3.tgz
76328 ./small2.tgz
1832./src
80 ./target
{noformat}
> clean up output of `dfs -du -s`
> ---
>
> Key: HDFS-5292
> URL: https://issues.apache.org/jira/browse/HDFS-5292
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.1.1-beta
>Reporter: Nick Dimiduk
>Priority: Minor
>
> This could be formatted a little nicer:
> {noformat}
> $ hdfs dfs -du -s /apps/hbase/data/data/default/*
> 22604541341 /apps/hbase/data/data/default/IntegrationTestBulkLoad
> 896656491 /apps/hbase/data/data/default/IntegrationTestIngest
> 33776145312 /apps/hbase/data/data/default/IntegrationTestLoadAndVerify
> 83512463 /apps/hbase/data/data/default/SendTracesTable
> 532898 /apps/hbase/data/data/default/TestAcidGuarantees
> 27294 /apps/hbase/data/data/default/demo_table
> 1410 /apps/hbase/data/data/default/example
> 2531532801 /apps/hbase/data/data/default/loadtest_d1
> 901 /apps/hbase/data/data/default/table_qho71mpvj8
> 1433 /apps/hbase/data/data/default/tcreatetbl
> 1690 /apps/hbase/data/data/default/tdelrowtbl
> 360 /apps/hbase/data/data/default/testtbl1
> 360 /apps/hbase/data/data/default/testtbl2
> 360 /apps/hbase/data/data/default/testtbl3
> 1515 /apps/hbase/data/data/default/tquerytbl
> 1513 /apps/hbase/data/data/default/tscantbl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5307) Support both HTTP and HTTPS in jsp pages
[ https://issues.apache.org/jira/browse/HDFS-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788463#comment-13788463 ] Hudson commented on HDFS-5307: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4560 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4560/]) HDFS-5307. Support both HTTP and HTTPS in jsp pages. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1530027) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DatanodeJspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ClusterJspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeJsp.java > Support both HTTP and HTTPS in jsp pages > > > Key: HDFS-5307 > URL: https://issues.apache.org/jira/browse/HDFS-5307 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5307.000.patch > > > Adopt the new APIs provided by HDFS-5306 to support both HTTP and HTTPS in > the jsp pages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5318) Account for shared storage in NameNode replica counting algorithm
Eric Sirianni created HDFS-5318:
---
Summary: Account for shared storage in NameNode replica counting
algorithm
Key: HDFS-5318
URL: https://issues.apache.org/jira/browse/HDFS-5318
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.3.0
Reporter: Eric Sirianni
There are several use cases for using shared-storage for datanode block storage
in an HDFS environment (storing cold blocks on a NAS device, Amazon S3, etc.).
With shared-storage, there is a distinction between:
# a distinct physical copy of a block
# an access-path to that block via a datanode.
A single 'replication count' metric cannot accurately capture both aspects of
the block. However, for most of the current uses of 'replication count' in the
Namenode, the "number of physical copies" aspect seems to be the appropriate
semantic.
I propose altering the replication counting algorithm in the Namenode to
accurately infer distinct physical copies in a shared storage environment.
With HDFS-5115, a {{StorageID}} is a UUID. I propose associating some minor
additional semantics to the {{StorageID}} - namely that multiple datanodes
attaching to the same physical shared storage pool should report the same
{{StorageID}} for that pool. A minor modification would be required in the
DataNode to enable the generation of {{StorageID}} s to be pluggable behind the
{{FsDatasetSpi}} interface.
With those semantics in place, the number of physical copies of a block in a
shared storage environment can be calculated as the number of _distinct_
{{StorageID}} s associated with that block.
Consider the following combinations for two {{(DataNode ID, Storage ID)}} pairs
{{(DN_A, S_A) (DN_B, S_B)}} for a given block B:
* {{DN_A != DN_B && S_A != S_B}} - *different* access paths to *different*
physical replicas (i.e. the traditional HDFS case with local disks)
** → Block B has {{ReplicationCount == 2}}
* {{DN_A != DN_B && S_A == S_B}} - *different* access paths to the *same*
physical replica (e.g. HDFS datanodes mounting the same NAS share)
** → Block B has {{ReplicationCount == 1}}
For example, if block B has the following location tuples:
* {{DN_1, STORAGE_A}}
* {{DN_2, STORAGE_A}}
* {{DN_3, STORAGE_B}}
* {{DN_4, STORAGE_B}},
the effect of this proposed change would be to calculate the replication factor
in the namenode as *2* instead of *4*.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5318) Account for shared storage in NameNode replica counting algorithm
[
https://issues.apache.org/jira/browse/HDFS-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sirianni updated HDFS-5318:
Description:
There are several use cases for using shared-storage for datanode block storage
in an HDFS environment (storing cold blocks on a NAS device, Amazon S3, etc.).
With shared-storage, there is a distinction between:
# a distinct physical copy of a block
# an access-path to that block via a datanode.
A single 'replication count' metric cannot accurately capture both aspects.
However, for most of the current uses of 'replication count' in the Namenode,
the "number of physical copies" aspect seems to be the appropriate semantic.
I propose altering the replication counting algorithm in the Namenode to
accurately infer distinct physical copies in a shared storage environment.
With HDFS-5115, a {{StorageID}} is a UUID. I propose associating some minor
additional semantics to the {{StorageID}} - namely that multiple datanodes
attaching to the same physical shared storage pool should report the same
{{StorageID}} for that pool. A minor modification would be required in the
DataNode to enable the generation of {{StorageID}} s to be pluggable behind the
{{FsDatasetSpi}} interface.
With those semantics in place, the number of physical copies of a block in a
shared storage environment can be calculated as the number of _distinct_
{{StorageID}} s associated with that block.
Consider the following combinations for two {{(DataNode ID, Storage ID)}} pairs
{{(DN_A, S_A) (DN_B, S_B)}} for a given block B:
* {{DN_A != DN_B && S_A != S_B}} - *different* access paths to *different*
physical replicas (i.e. the traditional HDFS case with local disks)
** → Block B has {{ReplicationCount == 2}}
* {{DN_A != DN_B && S_A == S_B}} - *different* access paths to the *same*
physical replica (e.g. HDFS datanodes mounting the same NAS share)
** → Block B has {{ReplicationCount == 1}}
For example, if block B has the following location tuples:
* {{DN_1, STORAGE_A}}
* {{DN_2, STORAGE_A}}
* {{DN_3, STORAGE_B}}
* {{DN_4, STORAGE_B}},
the effect of this proposed change would be to calculate the replication factor
in the namenode as *2* instead of *4*.
was:
There are several use cases for using shared-storage for datanode block storage
in an HDFS environment (storing cold blocks on a NAS device, Amazon S3, etc.).
With shared-storage, there is a distinction between:
# a distinct physical copy of a block
# an access-path to that block via a datanode.
A single 'replication count' metric cannot accurately capture both aspects of
the block. However, for most of the current uses of 'replication count' in the
Namenode, the "number of physical copies" aspect seems to be the appropriate
semantic.
I propose altering the replication counting algorithm in the Namenode to
accurately infer distinct physical copies in a shared storage environment.
With HDFS-5115, a {{StorageID}} is a UUID. I propose associating some minor
additional semantics to the {{StorageID}} - namely that multiple datanodes
attaching to the same physical shared storage pool should report the same
{{StorageID}} for that pool. A minor modification would be required in the
DataNode to enable the generation of {{StorageID}} s to be pluggable behind the
{{FsDatasetSpi}} interface.
With those semantics in place, the number of physical copies of a block in a
shared storage environment can be calculated as the number of _distinct_
{{StorageID}} s associated with that block.
Consider the following combinations for two {{(DataNode ID, Storage ID)}} pairs
{{(DN_A, S_A) (DN_B, S_B)}} for a given block B:
* {{DN_A != DN_B && S_A != S_B}} - *different* access paths to *different*
physical replicas (i.e. the traditional HDFS case with local disks)
** → Block B has {{ReplicationCount == 2}}
* {{DN_A != DN_B && S_A == S_B}} - *different* access paths to the *same*
physical replica (e.g. HDFS datanodes mounting the same NAS share)
** → Block B has {{ReplicationCount == 1}}
For example, if block B has the following location tuples:
* {{DN_1, STORAGE_A}}
* {{DN_2, STORAGE_A}}
* {{DN_3, STORAGE_B}}
* {{DN_4, STORAGE_B}},
the effect of this proposed change would be to calculate the replication factor
in the namenode as *2* instead of *4*.
> Account for shared storage in NameNode replica counting algorithm
> -
>
> Key: HDFS-5318
> URL: https://issues.apache.org/jira/browse/HDFS-5318
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Eric Sirianni
>
> There are several use cases for using shared-storage for datanode block
> storage in an HDFS environment (storing cold blocks on a NAS device, Amazon
> S3, etc.).
> With shared-storage, there is
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788466#comment-13788466
]
Andrew Wang commented on HDFS-5224:
---
Good idea, only change I'd like is to keep the directive immutable and use a
builder instead. We create the descriptor from the input directive in the
ClientNamenode PB translator, so it could possibly race here, and I don't think
mutating a directive after create time makes much sense anyway (esp after it's
used in {{addPathBasedCacheDirective}}).
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5307) Support both HTTP and HTTPS in jsp pages
[ https://issues.apache.org/jira/browse/HDFS-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5307: - Fix Version/s: 2.2.0 > Support both HTTP and HTTPS in jsp pages > > > Key: HDFS-5307 > URL: https://issues.apache.org/jira/browse/HDFS-5307 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.2.0 > > Attachments: HDFS-5307.000.patch > > > Adopt the new APIs provided by HDFS-5306 to support both HTTP and HTTPS in > the jsp pages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5307) Support both HTTP and HTTPS in jsp pages
[ https://issues.apache.org/jira/browse/HDFS-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5307: - Resolution: Fixed Status: Resolved (was: Patch Available) > Support both HTTP and HTTPS in jsp pages > > > Key: HDFS-5307 > URL: https://issues.apache.org/jira/browse/HDFS-5307 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.2.0 > > Attachments: HDFS-5307.000.patch > > > Adopt the new APIs provided by HDFS-5306 to support both HTTP and HTTPS in > the jsp pages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788516#comment-13788516
]
Chris Nauroth commented on HDFS-5224:
-
[~cmccabe], if I understand correctly, you're advocating that we go ahead and
pass around {{PathBasedCacheDirective/Descriptor}} at all layers (client,
protocol, and NN implementation all the way down to {{CacheManager}}). The
path member will be a {{Path}}, not a {{String}}, and the implementation in the
NN will only use the path part and ignore all other URI components like scheme
and authority. This is accomplished by calling {{getPath().toUri().getPath()}}
in any NN code that needs to work with it. Am I understanding the proposal
correctly? If so, then the consequence is that we have a slightly odd data
type in the NN implementation (most components of {{Path}} will be ignored),
but the benefit is that we have fewer total classes.
(BTW, I'm not objecting to anything above, just trying to understand
trade-offs.)
bq. Keep in mind that we want the DistributedFileSystem to supply its current
working directory, and validate that the path matches its URI.
Yes, understood. The current patch already does this, and now I think we're
just working out the final state for refactoring this code.
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788519#comment-13788519
]
Hadoop QA commented on HDFS-5291:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607193/HDFS-5291.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5124//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5124//console
This message is automatically generated.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Succes
[jira] [Commented] (HDFS-5314) Do not expose CachePool type in AddCachePoolOp
[
https://issues.apache.org/jira/browse/HDFS-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788545#comment-13788545
]
Andrew Wang commented on HDFS-5314:
---
Hey Colin, patch makes sense to me. +1 after the following are addressed:
* Add some more class Javadoc for CachePoolInfo. I'd like to see mention of
where it's used (client-side and serialization in general in the edit log) and
where defaults are supplied for missing values.
* Add/fix Javadoc for CachePool, the clients comment is no longer entirely
accurate per above
* Can we make the CachePool constructor private? Else it'd benefit from a
javadoc pointer to the new static methods.
* Can rename {{createFromPoolInfoAndDefaults}} to {{fromInfoWithDefaults}},
{{createFromPoolInfo}} to {{fromInfo}} to be more concise.
> Do not expose CachePool type in AddCachePoolOp
> --
>
> Key: HDFS-5314
> URL: https://issues.apache.org/jira/browse/HDFS-5314
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5314-caching.001.patch
>
>
> {{CachePool}} is an internal type which should not be exposed or used in the
> {{FSEditLog}}. Because {{CachePool}} is mutable, storing a reference to it
> in an edit log operation could lead to the edit log operation changing as a
> result of the {{CachePool}} object being changed. Instead, we should store
> {{CachePoolInfo}}.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788548#comment-13788548
]
Colin Patrick McCabe commented on HDFS-5224:
bq. Colin Patrick McCabe, if I understand correctly, you're advocating that we
go ahead and pass around PathBasedCacheDirective/Descriptor at all layers
(client, protocol, and NN implementation all the way down to CacheManager). The
path member will be a Path, not a String, and the implementation in the NN will
only use the path part and ignore all other URI components like scheme and
authority.
Right, exactly. If you look at {{NameNodeRpcServer}}, you see that all the
methods there treat path as a {{String}}. The NN doesn't need scheme or
authority, since we assume that if you figured out how to talk to the NN, you
know those already :) Only the {{DistributedFilesystem}} (client-facing API)
needs to handle scheme and authority.
bq. Good idea, only change I'd like is to keep the directive immutable and use
a builder instead. We create the descriptor from the input directive in the
ClientNamenode PB translator, so it could possibly race here, and I don't think
mutating a directive after create time makes much sense anyway (esp after it's
used in addPathBasedCacheDirective).
Good idea. We could have a method in the {{DistributedFileSystem}} return a
{{PathBasedCacheDirective#Builder}}, perhaps.
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788552#comment-13788552
]
Colin Patrick McCabe commented on HDFS-5224:
er, wait, I think I misunderstood Chris's proposal.
No, I am *not* saying we add {{Path}} to {{PathBasedCacheEntry}}. I'm saying
we add a method in {{DistributedFileSystem}} that takes a {{Path}} and returns
a {{PathBasedCacheEntry}} initialized with the correct {{String}} (or a builder
initialized with the correct string)
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5291:
Attachment: HDFS-5291.003.patch
The new javadoc warning should be wrong report. Update the patch to see if we
can get rid of the complain.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 2013-10-02 00:13:49,875 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recover
> {code}
> And then we get into safemode
> {code}
> Construction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,277 INFO BlockStateChange
> (BlockManager.java:logAddStoredBlock(2237)) - BLOCK* addStoredBlock: blockMap
> updated: IP:1019 is added to blk_IP157{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[IP:1019|RBW],
> ReplicaUnderConstruction[172.18.145.96:1019|RBW], ReplicaUnde
> rConstruction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50
[jira] [Updated] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5317: - Status: Patch Available (was: Open) > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Attachments: HDFS-5317.000.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5317: - Attachment: HDFS-5317.000.patch > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Attachments: HDFS-5317.000.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5314) Do not expose CachePool type in AddCachePoolOp
[
https://issues.apache.org/jira/browse/HDFS-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-5314:
---
Attachment: HDFS-5314-caching.002.patch
* add more javadoc for {{CachePool}}, {{CachePoolInfo}}
* make the {{CachePool}} constructor package-private
* rename createFromPoolInfo, etc to be less verbose
> Do not expose CachePool type in AddCachePoolOp
> --
>
> Key: HDFS-5314
> URL: https://issues.apache.org/jira/browse/HDFS-5314
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5314-caching.001.patch, HDFS-5314-caching.002.patch
>
>
> {{CachePool}} is an internal type which should not be exposed or used in the
> {{FSEditLog}}. Because {{CachePool}} is mutable, storing a reference to it
> in an edit log operation could lead to the edit log operation changing as a
> result of the {{CachePool}} object being changed. Instead, we should store
> {{CachePoolInfo}}.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[
https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788569#comment-13788569
]
Hadoop QA commented on HDFS-5317:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607234/HDFS-5317.000.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5126//console
This message is automatically generated.
> Go back to DFS Home link does not work on datanode webUI
>
>
> Key: HDFS-5317
> URL: https://issues.apache.org/jira/browse/HDFS-5317
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Suresh Srinivas
>Assignee: Haohui Mai
>Priority: Critical
> Attachments: HDFS-5317.000.patch
>
>
> Here is how to reproduce this problem:
> # Goto https://namenode:https_port/dfshealth.jsp
> # Click on "Live Nodes" link
> # On the Live Nodes page, click one of the links for the datanodes. This link
> has always hardcodes the http link instead of http/https based on the context
> of the webpage.
> On datanode webUI page, the link "Go back to DFS home" does not work.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788583#comment-13788583
]
Chris Nauroth commented on HDFS-5224:
-
bq. I'm saying we add a method in DistributedFileSystem that takes a Path and
returns a PathBasedCacheEntry initialized with the correct String (or a builder
initialized with the correct string).
I had steered away from this approach, because it leaves clients getting a
{{String}} instead of a {{Path}} when they call
{{listPathBasedCacheDescriptors}}. That would still be a bit of inconsistency
in the interface. (By analogy, {{listStatus}} returns {{FileStatus}}, which
contains {{Path}}.)
While folks chew on this a little more, I'll get started on the builder code
itself. It sounds like everyone is in agreement on that part.
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5319) Links resolving either from active/standby should be same (example clicking on datanodes from Standby)
Siqi Li created HDFS-5319: - Summary: Links resolving either from active/standby should be same (example clicking on datanodes from Standby) Key: HDFS-5319 URL: https://issues.apache.org/jira/browse/HDFS-5319 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Siqi Li Priority: Minor -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5319) Links resolving either from active/standby should be same (example clicking on datanodes from Standby)
[ https://issues.apache.org/jira/browse/HDFS-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-5319: -- Description: click live nodes from standby namenode will throw exception "Operation category READ is not supported in state standby" > Links resolving either from active/standby should be same (example clicking > on datanodes from Standby) > -- > > Key: HDFS-5319 > URL: https://issues.apache.org/jira/browse/HDFS-5319 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Siqi Li >Priority: Minor > > click live nodes from standby namenode will throw exception "Operation > category READ is not supported in state standby" -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5320) Add datanode caching metrics
Andrew Wang created HDFS-5320: - Summary: Add datanode caching metrics Key: HDFS-5320 URL: https://issues.apache.org/jira/browse/HDFS-5320 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: HDFS-4949 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor It'd be good to hook up datanode metrics for # (blocks/bytes) (cached/uncached/failed to cache) over different time windows (eternity/1hr/10min/1min). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5317: - Attachment: HDFS-5317.001.patch > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5319) Links resolving either from active/standby should be same (example clicking on datanodes from Standby)
[ https://issues.apache.org/jira/browse/HDFS-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-5319: -- Attachment: HDFS-5319-v1.patch > Links resolving either from active/standby should be same (example clicking > on datanodes from Standby) > -- > > Key: HDFS-5319 > URL: https://issues.apache.org/jira/browse/HDFS-5319 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: HDFS-5319-v1.patch > > > click live nodes from standby namenode will throw exception "Operation > category READ is not supported in state standby" -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5319) Links resolving either from active/standby should be same (example clicking on datanodes from Standby)
[ https://issues.apache.org/jira/browse/HDFS-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated HDFS-5319: -- Status: Patch Available (was: Open) > Links resolving either from active/standby should be same (example clicking > on datanodes from Standby) > -- > > Key: HDFS-5319 > URL: https://issues.apache.org/jira/browse/HDFS-5319 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: HDFS-5319-v1.patch > > > click live nodes from standby namenode will throw exception "Operation > category READ is not supported in state standby" -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5316) Namenode ignores the default https port
[
https://issues.apache.org/jira/browse/HDFS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-5316:
-
Status: Patch Available (was: Open)
> Namenode ignores the default https port
> ---
>
> Key: HDFS-5316
> URL: https://issues.apache.org/jira/browse/HDFS-5316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha
>Reporter: Suresh Srinivas
>Assignee: Haohui Mai
>Priority: Critical
> Attachments: HDFS-5316.000.patch
>
>
> When dfs.https.enable is true and dfs.https.port is not configured, namenode
> does not pickup the default https port (50470), instead picks up random port.
> See:
> {code}
> boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
> InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
> ":" + conf.get(
> DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
> Configuration sslConf = new Configuration(false);
> if (certSSL) {
>
> sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
> "ssl-server.xml"));
> }
> {code}
> Unless https port is specifically configured as 0, we should not be picking
> random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5316) Namenode ignores the default https port
[
https://issues.apache.org/jira/browse/HDFS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-5316:
-
Attachment: HDFS-5316.000.patch
> Namenode ignores the default https port
> ---
>
> Key: HDFS-5316
> URL: https://issues.apache.org/jira/browse/HDFS-5316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha
>Reporter: Suresh Srinivas
>Assignee: Haohui Mai
>Priority: Critical
> Attachments: HDFS-5316.000.patch
>
>
> When dfs.https.enable is true and dfs.https.port is not configured, namenode
> does not pickup the default https port (50470), instead picks up random port.
> See:
> {code}
> boolean needClientAuth = conf.getBoolean("dfs.https.need.client.auth", false);
> InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
> ":" + conf.get(
> DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, "0"));
> Configuration sslConf = new Configuration(false);
> if (certSSL) {
>
> sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
> "ssl-server.xml"));
> }
> {code}
> Unless https port is specifically configured as 0, we should not be picking
> random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788617#comment-13788617
]
Andrew Wang commented on HDFS-5224:
---
Is it acceptable to have the descriptor's internal representation still be a
{{String}}, but then turn it into a {{Path}} for the client in the getter? I
think this is ok since the path string is always absolute, and we could even
qualify with the correct scheme and authority if desired.
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5319) Links resolving either from active/standby should be same (example clicking on datanodes from Standby)
[
https://issues.apache.org/jira/browse/HDFS-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788618#comment-13788618
]
Hadoop QA commented on HDFS-5319:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607249/HDFS-5319-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5128//console
This message is automatically generated.
> Links resolving either from active/standby should be same (example clicking
> on datanodes from Standby)
> --
>
> Key: HDFS-5319
> URL: https://issues.apache.org/jira/browse/HDFS-5319
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Siqi Li
>Priority: Minor
> Attachments: HDFS-5319-v1.patch
>
>
> click live nodes from standby namenode will throw exception "Operation
> category READ is not supported in state standby"
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5304) Expose if a block replica is cached in getFileBlockLocations
[
https://issues.apache.org/jira/browse/HDFS-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788626#comment-13788626
]
Colin Patrick McCabe commented on HDFS-5304:
An array of booleans in {{LocatedBlock}} seems kind of clunky. You have to
check all of them to find out if any are true.And then you have to match
them up with the corresponding element in the {{locs}} array. Can we have
something like this instead?
{code}
DatanodeInfo[] cachedLocs;
{code}
Obviously, these would just be pointers to the same DatanodeInfo objects stored
in {{locs}}, not copies. This way:
* it's easy to iterate over all cached locations, or check if there are any
(just do {{location.getCachedLocations.isEmpty()}}).
* this can be mapped to a statically declared empty array most of the time,
saving memory
The RPC changes themselves are fine, I think.
> Expose if a block replica is cached in getFileBlockLocations
>
>
> Key: HDFS-5304
> URL: https://issues.apache.org/jira/browse/HDFS-5304
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-5304-1.patch
>
>
> We need to expose which replicas of a block are cached so applications can
> place their tasks for memory-locality.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HDFS-5314) Do not expose CachePool type in AddCachePoolOp
[
https://issues.apache.org/jira/browse/HDFS-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe resolved HDFS-5314.
Resolution: Fixed
Fix Version/s: HDFS-4949
> Do not expose CachePool type in AddCachePoolOp
> --
>
> Key: HDFS-5314
> URL: https://issues.apache.org/jira/browse/HDFS-5314
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: HDFS-4949
>
> Attachments: HDFS-5314-caching.001.patch, HDFS-5314-caching.002.patch
>
>
> {{CachePool}} is an internal type which should not be exposed or used in the
> {{FSEditLog}}. Because {{CachePool}} is mutable, storing a reference to it
> in an edit log operation could lead to the edit log operation changing as a
> result of the {{CachePool}} object being changed. Instead, we should store
> {{CachePoolInfo}}.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
[
https://issues.apache.org/jira/browse/HDFS-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788637#comment-13788637
]
Colin Patrick McCabe commented on HDFS-5224:
I don't have a super-strong opinion about this, but I like Andrew's proposal to
have a getter that returns a Path in the {{PathBasedCacheDescriptor}}.
bq. While folks chew on this a little more, I'll get started on the builder
code itself. It sounds like everyone is in agreement on that part.
Yeah, a factory would be good.
> Refactor PathBasedCache* methods to use a Path rather than a String
> ---
>
> Key: HDFS-5224
> URL: https://issues.apache.org/jira/browse/HDFS-5224
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Andrew Wang
>Assignee: Chris Nauroth
> Attachments: HDFS-5224.1.patch
>
>
> As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and
> related methods in DistributedFileSystem to use a Path to represent paths to
> cache, rather than a String.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5293) Symlink resolution requires unnecessary RPCs
[
https://issues.apache.org/jira/browse/HDFS-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788659#comment-13788659
]
Colin Patrick McCabe commented on HDFS-5293:
bq. That doesn't work in stacked/proxy filesystems or anything that wants to do
custom filtering. At a minimum, you'd need to make the NN "chroot aware" by
defining / to be another path, treat ".." out of the root as an unresolved
link. Then the NN has to deal with cyclic links, etc. I'm not advocating any of
that.
I'm just saying the NameNode should handle resolving the symlinks that it can
resolve, and throw {{UnresolvedLinkException}} for the ones which it can't. A
symlink which points to a different scheme/authority or to something "above
root" obviously still needs to be handled via an {{UnresolvedLinkException}}.
Essentially, I am proposing an optimization which can used on
non-cross-filesystem symlinks. This is also how {{LocalFileSystem}} currently
works.
Anyway, we can continue discussing that optimization on HADOOP-9780 if you
want, since it's a separate issue from the RPC reduction issue you're
discussing here.
> Symlink resolution requires unnecessary RPCs
>
>
> Key: HDFS-5293
> URL: https://issues.apache.org/jira/browse/HDFS-5293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Priority: Critical
>
> When the NN encounters a symlink, it throws an {{UnresolvedLinkException}}.
> This exception contains only the path that is a symlink. The client issues
> another RPC to obtain the link target, followed by another RPC with the link
> target + remainder of the original path.
> {{UnresolvedLinkException}} should be returning both the link and the target
> to avoid a costly and unnecessary intermediate RPC to obtain the link target.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5321) Clean up the HTTP-related configuration in HDFS
Haohui Mai created HDFS-5321: Summary: Clean up the HTTP-related configuration in HDFS Key: HDFS-5321 URL: https://issues.apache.org/jira/browse/HDFS-5321 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Currently there are multiple configuration keys that control the ports that the NameNode and DataNode listen to, and the default ports that the hftp/webhdfs clients are connecting to. Below is a quick summary of these configuration: || Keys || Description || | dfs.namenode.http-address | The address that the namenode http server binds to | | dfs.namenode.https-address | The address that the namenode https server binds to | | dfs.http.port | The default port that the hftp/webhdfs client use to connect to the remote server| | dfs.https.port | The default port that the hsftp client use to connect to the remote server| I propose to deprecate dfs.http.port and dfs.https.port to avoid potential confusions (e.g., HDFS-5316). Note that this removes no functionality, since the users can specify ports in hftp / webhdfs URLs when they need to connect to HDFS servers with non-default ports. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5322) HDFS delegation token not found in cache errors seen on secure HA clusters
Arpit Gupta created HDFS-5322:
-
Summary: HDFS delegation token not found in cache errors seen on
secure HA clusters
Key: HDFS-5322
URL: https://issues.apache.org/jira/browse/HDFS-5322
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Affects Versions: 2.1.1-beta
Reporter: Arpit Gupta
Assignee: Jing Zhao
While running HA tests we have seen issues were we see HDFS delegation token
not found in cache errors causing jobs running to fail.
{code}
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
|2013-10-06 20:14:51,193 INFO [main] mapreduce.Job: Task Id :
attempt_1381090351344_0001_m_07_0, Status : FAILED
Error:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 11 for hrt_qa) can't be found in cache
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5322) HDFS delegation token not found in cache errors seen on secure HA clusters
[
https://issues.apache.org/jira/browse/HDFS-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788672#comment-13788672
]
Jing Zhao commented on HDFS-5322:
-
Looks like the issue happened during the standby->active transitioning process
of the namenode. The namenode had not finished tailing the editlog, which
contained information about generating the corresponding token, thus failed to
find the token in its local cache.
A possible solution is to let the client side retry the same operation in this
scenario. Thus if the namenode could not find the token during the transition,
it can wrap the InvalidToken exception in the RetriableException and ask the
client to retry.
> HDFS delegation token not found in cache errors seen on secure HA clusters
> --
>
> Key: HDFS-5322
> URL: https://issues.apache.org/jira/browse/HDFS-5322
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>
> While running HA tests we have seen issues were we see HDFS delegation token
> not found in cache errors causing jobs running to fail.
> {code}
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> |2013-10-06 20:14:51,193 INFO [main] mapreduce.Job: Task Id :
> attempt_1381090351344_0001_m_07_0, Status : FAILED
> Error:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 11 for hrt_qa) can't be found in cache
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[
https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788683#comment-13788683
]
Brandon Li commented on HDFS-5317:
--
The latest patch HDFS-5317.001.patch looks good. Can you explain why removed
the 1 second sleep in NamenodeJspHelper.java :
{noformat}
- try {
-Thread.sleep(1000);
- } catch (InterruptedException e) {
- }
-
{noformat}
> Go back to DFS Home link does not work on datanode webUI
>
>
> Key: HDFS-5317
> URL: https://issues.apache.org/jira/browse/HDFS-5317
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Suresh Srinivas
>Assignee: Haohui Mai
>Priority: Critical
> Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch
>
>
> Here is how to reproduce this problem:
> # Goto https://namenode:https_port/dfshealth.jsp
> # Click on "Live Nodes" link
> # On the Live Nodes page, click one of the links for the datanodes. This link
> has always hardcodes the http link instead of http/https based on the context
> of the webpage.
> On datanode webUI page, the link "Go back to DFS home" does not work.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788687#comment-13788687
]
Hadoop QA commented on HDFS-5291:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607230/HDFS-5291.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5125//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5125//console
This message is automatically generated.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegme
[jira] [Assigned] (HDFS-4511) Cover package org.apache.hadoop.hdfs.tools with unit test
[ https://issues.apache.org/jira/browse/HDFS-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned HDFS-4511: - Assignee: Andrey Klochkov > Cover package org.apache.hadoop.hdfs.tools with unit test > - > > Key: HDFS-4511 > URL: https://issues.apache.org/jira/browse/HDFS-4511 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Vadim Bondarev >Assignee: Andrey Klochkov > Attachments: HADOOP-4511-branch-0.23-a.patch, > HADOOP-4511-branch-2-a.patch, HADOOP-4511-trunk-a.patch, > HDFS-4511-branch-2--N2.patch, HDFS-4511-trunk--N4.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788718#comment-13788718 ] Haohui Mai commented on HDFS-5317: -- There are no obvious reasons that the sleep is required. Because the JSP page runs in a different thread, the information is never guaranteed to be up-to-date, in which case putting a sleep() in the JSP page would not help. Alternatively, we can make the page faster, and the user always has the freedom to refresh the page when needed. > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[
https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788727#comment-13788727
]
Hadoop QA commented on HDFS-5317:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607247/HDFS-5317.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5127//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5127//console
This message is automatically generated.
> Go back to DFS Home link does not work on datanode webUI
>
>
> Key: HDFS-5317
> URL: https://issues.apache.org/jira/browse/HDFS-5317
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Suresh Srinivas
>Assignee: Haohui Mai
>Priority: Critical
> Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch
>
>
> Here is how to reproduce this problem:
> # Goto https://namenode:https_port/dfshealth.jsp
> # Click on "Live Nodes" link
> # On the Live Nodes page, click one of the links for the datanodes. This link
> has always hardcodes the http link instead of http/https based on the context
> of the webpage.
> On datanode webUI page, the link "Go back to DFS home" does not work.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788724#comment-13788724 ] Shinichi Yamashita commented on HDFS-5040: -- It seems that "allowSnapshot" and "disallowSnapshot" have been already implemented in trunk. And it is possible to let NameNode's audit log output the remaining commands except "refreshNameNodes" and "deleteBlockPool". > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Raghu C Doppalapudi > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788733#comment-13788733 ] Brandon Li commented on HDFS-5317: -- +1 > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788744#comment-13788744
]
Hudson commented on HDFS-5291:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4562 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4562/])
HDFS-5291. Standby namenode after transition to active goes into safemode.
Contributed by Jing Zhao. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1530112)
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RetriableException.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SafeModeException.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Re
[jira] [Updated] (HDFS-5291) Standby namenode after transition to active goes into safemode
[
https://issues.apache.org/jira/browse/HDFS-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5291:
Resolution: Fixed
Fix Version/s: 2.2.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks again for the review, Suresh! I've committed this to trunk, branch-2 and
branch-2.2.
> Standby namenode after transition to active goes into safemode
> --
>
> Key: HDFS-5291
> URL: https://issues.apache.org/jira/browse/HDFS-5291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.1.1-beta
>Reporter: Arpit Gupta
>Assignee: Jing Zhao
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HDFS-5291.000.patch, HDFS-5291.001.patch,
> HDFS-5291.002.patch, HDFS-5291.003.patch, HDFS-5291.003.patch, nn.log
>
>
> Some log snippets
> standby state to active transition
> {code}
> 2013-10-02 00:13:49,482 INFO ipc.Server (Server.java:run(2068)) - IPC Server
> handler 69 on 8020, call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from IP:33911
> Call#1483 Retry#1: error: org.apache.hadoop.ipc.StandbyException: Operation
> category WRITE is not supported in state standby
> 2013-10-02 00:13:49,689 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for nn/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,696 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for nn/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.ha.HAServiceProtocol
> 2013-10-02 00:13:49,700 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopStandbyServices(1013)) - Stopping services started for
> standby state
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> 2013-10-02 00:13:49,704 INFO namenode.FSNamesystem
> (FSNamesystem.java:startActiveServices(885)) - Starting services required for
> active state
> 2013-10-02 00:13:49,719 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(419)) - Starting
> recovery process for unclosed journal segments...
> 2013-10-02 00:13:49,755 INFO ipc.Server (Server.java:saslProcess(1342)) -
> Auth successful for hbase/hostn...@example.com (auth:SIMPLE)
> 2013-10-02 00:13:49,761 INFO authorize.ServiceAuthorizationManager
> (ServiceAuthorizationManager.java:authorize(111)) - Authorization successful
> for hbase/hostn...@example.com (auth:KERBEROS) for protocol=interface
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnfinalizedSegments(421)) - Successfully
> started new epoch 85
> 2013-10-02 00:13:49,839 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(249)) - Beginning recovery
> of unclosed segment starting at txid 887112
> 2013-10-02 00:13:49,874 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recoverUnclosedSegment(258)) - Recovery prepare
> phase complete. Responses:
> IP:8485: segmentState { startTxId: 887112 endTxId: 887531 isInProgress: true
> } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 172.18.145.97:8485: segmentState { startTxId: 887112 endTxId: 887531
> isInProgress: true } lastWriterEpoch: 84 lastCommittedTxId: 887530
> 2013-10-02 00:13:49,875 INFO client.QuorumJournalManager
> (QuorumJournalManager.java:recover
> {code}
> And then we get into safemode
> {code}
> Construction[IP:1019|RBW]]} size 0
> 2013-10-02 00:13:50,277 INFO BlockStateChange
> (BlockManager.java:logAddStoredBlock(2237)) - BLOCK* addStoredBlock: blockMap
> updated: IP:1019 is added to blk_IP157{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[IP:1019|RBW],
> ReplicaUnde
[jira] [Commented] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788754#comment-13788754 ] Hudson commented on HDFS-5317: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4563 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4563/]) HDFS-5317. Go back to DFS Home link does not work on datanode webUI. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1530114) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeJsp.java > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated HDFS-5040: - Attachment: HDFS-5040.patch I attach a patch file related to previous comment. > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Raghu C Doppalapudi > Attachments: HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated HDFS-5040: - Target Version/s: 3.0.0 Affects Version/s: 3.0.0 Status: Patch Available (was: Open) > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0 >Reporter: Raghu C Doppalapudi > Attachments: HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5321) Clean up the HTTP-related configuration in HDFS
[ https://issues.apache.org/jira/browse/HDFS-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788763#comment-13788763 ] Brandon Li commented on HDFS-5321: -- dfs.https.port and dfs.http.port are already deprecated (referring HdfsConfiguration#addDeprecatedKeys). We can use this JIRA to improve the hdfs-site.xml document to clearly describe the relationship among these configurations. For example, what happens when both dfs.namenode.https-address and dfs.https.port are configured. > Clean up the HTTP-related configuration in HDFS > --- > > Key: HDFS-5321 > URL: https://issues.apache.org/jira/browse/HDFS-5321 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > > Currently there are multiple configuration keys that control the ports that > the NameNode and DataNode listen to, and the default ports that the > hftp/webhdfs clients are connecting to. > Below is a quick summary of these configuration: > || Keys || Description || > | dfs.namenode.http-address | The address that the namenode http server binds > to | > | dfs.namenode.https-address | The address that the namenode https server > binds to | > | dfs.http.port | The default port that the hftp/webhdfs client use to > connect to the remote server| > | dfs.https.port | The default port that the hsftp client use to connect to > the remote server| > I propose to deprecate dfs.http.port and dfs.https.port to avoid potential > confusions (e.g., HDFS-5316). Note that this removes no functionality, since > the users can specify ports in hftp / webhdfs URLs when they need to connect > to HDFS servers with non-default ports. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5321) Clean up the HTTP-related configuration in HDFS
[ https://issues.apache.org/jira/browse/HDFS-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788775#comment-13788775 ] Brandon Li commented on HDFS-5321: -- Sorry dfs.https.port and dfs.http.port are not deprecated, dfs.https.address and dfs.http.address are. > Clean up the HTTP-related configuration in HDFS > --- > > Key: HDFS-5321 > URL: https://issues.apache.org/jira/browse/HDFS-5321 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > > Currently there are multiple configuration keys that control the ports that > the NameNode and DataNode listen to, and the default ports that the > hftp/webhdfs clients are connecting to. > Below is a quick summary of these configuration: > || Keys || Description || > | dfs.namenode.http-address | The address that the namenode http server binds > to | > | dfs.namenode.https-address | The address that the namenode https server > binds to | > | dfs.http.port | The default port that the hftp/webhdfs client use to > connect to the remote server| > | dfs.https.port | The default port that the hsftp client use to connect to > the remote server| > I propose to deprecate dfs.http.port and dfs.https.port to avoid potential > confusions (e.g., HDFS-5316). Note that this removes no functionality, since > the users can specify ports in hftp / webhdfs URLs when they need to connect > to HDFS servers with non-default ports. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788776#comment-13788776
]
Hadoop QA commented on HDFS-5040:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607273/HDFS-5040.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5130//console
This message is automatically generated.
> Audit log for admin commands/ logging output of all DFS admin commands
> --
>
> Key: HDFS-5040
> URL: https://issues.apache.org/jira/browse/HDFS-5040
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Raghu C Doppalapudi
> Attachments: HDFS-5040.patch
>
>
> enable audit log for all the admin commands/also provide ability to log all
> the admin commands in separate log file, at this point all the logging is
> displayed on the console.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788777#comment-13788777 ] Brandon Li commented on HDFS-5317: -- I've committed the patch. Thank you, Haohui, for the contribution! > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Fix For: 2.2.0 > > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5317: - Fix Version/s: 2.2.0 > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Fix For: 2.2.0 > > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
[ https://issues.apache.org/jira/browse/HDFS-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5317: - Resolution: Fixed Status: Resolved (was: Patch Available) > Go back to DFS Home link does not work on datanode webUI > > > Key: HDFS-5317 > URL: https://issues.apache.org/jira/browse/HDFS-5317 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Haohui Mai >Priority: Critical > Fix For: 2.2.0 > > Attachments: HDFS-5317.000.patch, HDFS-5317.001.patch > > > Here is how to reproduce this problem: > # Goto https://namenode:https_port/dfshealth.jsp > # Click on "Live Nodes" link > # On the Live Nodes page, click one of the links for the datanodes. This link > has always hardcodes the http link instead of http/https based on the context > of the webpage. > On datanode webUI page, the link "Go back to DFS home" does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-4.patch Here's a lot more rigorous testing. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, Zipkin Trace a06e941b0172ec73.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788785#comment-13788785
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607280/HDFS-5274-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5131//console
This message is automatically generated.
> Add Tracing to HDFS
> ---
>
> Key: HDFS-5274
> URL: https://issues.apache.org/jira/browse/HDFS-5274
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
> HDFS-5274-3.patch, HDFS-5274-4.patch, Zipkin Trace a06e941b0172ec73.png
>
>
> Since Google's Dapper paper has shown the benefits of tracing for a large
> distributed system, it seems like a good time to add tracing to HDFS. HBase
> has added tracing using HTrace. I propose that the same can be done within
> HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
