date:20111127


[ 
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158019#comment-13158019
 ] 

Eli Collins commented on HDFS-2582:
---

lgtm, +1 with the follow-on jiras.

The code that uses the new map (Mapns id, Mapnn id, addr) is a little 
unwieldy since we don't have types for the ns and nn ids. Given that this is a 
tiny map the readability might be worth the overhead of a class (eg in the same 
vain of ConfiguredNNAddress). Don't feel strongly so feel free to punt if you 
disagree.

 Scope dfs.ha.namenodes config by nameservice
 

 Key: HDFS-2582
 URL: https://issues.apache.org/jira/browse/HDFS-2582
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: data-node, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: HA branch (HDFS-1623)

 Attachments: hdfs-2582-v2.txt, hdfs-2582-v3.txt, hdfs-2582.txt


 HDFS-2231 started the process of adding configuration for HA, but one piece 
 is missing. The current state of the configuration is, I believe:
 {{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
 {{dfs.federation.nameservices}} - a list of federated nameservices
 {{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific 
 config for the given namenode. If HA or federation is disabled, the extra 
 components can be elided for backwards compatibility.
 The issue here is that there is no easy way to discern which NN is paired 
 with which other NN. Additionally, adding a new federated nameservice to a 
 config will require changes to {{dfs.ha.namenodes}} which makes templating 
 harder. It would be simpler to change {{dfs.ha.namenodes}} to be 
 nameservice-scoped: {{dfs.ha.namenodes.nameservice-id}}.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2591) HA: MiniDFSCluster support to mix and match federation with HA


[ 
https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158024#comment-13158024
 ] 

Eli Collins commented on HDFS-2591:
---

What you have so far looks good.  If we end up creating a class for the map in 
HDFS-2582 it could probably share the code here since it's mostly the same map 
essentially in class form.

 HA: MiniDFSCluster support to mix and match federation with HA
 --

 Key: HDFS-2591
 URL: https://issues.apache.org/jira/browse/HDFS-2591
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha, test
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: hdfs-2591.txt


 Right now the MiniDFS builder is somewhat inflexible - it just takes a 
 numNameNodes parameter, which is used to specify federated nameservices. In 
 order to add HA support, we need to be able to be more specific when 
 configuring the NNs -- eg to test the case where there is one nameservice 
 that is HA and another which is not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice


 [ 
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-2582:
--

Attachment: hdfs-2582-v4.txt

Here's v4, which I built a few more patches on top. One more slight fix to the 
nnId detection stuff, nothing major changed.

I agree the mapmap.. is kind of clunky, but I'd like to punt to a follow-on 
to clean this up, since I have some more patches built on top.

I'll commit this soon based on Eli's prior +1 unless there are any other 
comments.

 Scope dfs.ha.namenodes config by nameservice
 

 Key: HDFS-2582
 URL: https://issues.apache.org/jira/browse/HDFS-2582
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: data-node, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: HA branch (HDFS-1623)

 Attachments: hdfs-2582-v2.txt, hdfs-2582-v3.txt, hdfs-2582-v4.txt, 
 hdfs-2582.txt


 HDFS-2231 started the process of adding configuration for HA, but one piece 
 is missing. The current state of the configuration is, I believe:
 {{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
 {{dfs.federation.nameservices}} - a list of federated nameservices
 {{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific 
 config for the given namenode. If HA or federation is disabled, the extra 
 components can be elided for backwards compatibility.
 The issue here is that there is no easy way to discern which NN is paired 
 with which other NN. Additionally, adding a new federated nameservice to a 
 config will require changes to {{dfs.ha.namenodes}} which makes templating 
 harder. It would be simpler to change {{dfs.ha.namenodes}} to be 
 nameservice-scoped: {{dfs.ha.namenodes.nameservice-id}}.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2291) HA: Checkpointing in an HA setup


[ 
https://issues.apache.org/jira/browse/HDFS-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158060#comment-13158060
 ] 

Eli Collins commented on HDFS-2291:
---

Agree that the SBN should be able to do checkpoints - someone running a typical 
20x configuration with two hosts (NN and 2NN) should be able to keep the same 
hardware config (NN and SBN).

 HA: Checkpointing in an HA setup
 

 Key: HDFS-2291
 URL: https://issues.apache.org/jira/browse/HDFS-2291
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Fix For: HA branch (HDFS-1623)


 We obviously need to create checkpoints when HA is enabled. One thought is to 
 use a third, dedicated checkpointing node in addition to the active and 
 standby nodes. Another option would be to make the standby capable of also 
 performing the function of checkpointing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice


[ 
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158061#comment-13158061
 ] 

Eli Collins commented on HDFS-2582:
---

+1 to v4, the fix in the delta lgtm

 Scope dfs.ha.namenodes config by nameservice
 

 Key: HDFS-2582
 URL: https://issues.apache.org/jira/browse/HDFS-2582
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: data-node, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: HA branch (HDFS-1623)

 Attachments: hdfs-2582-v2.txt, hdfs-2582-v3.txt, hdfs-2582-v4.txt, 
 hdfs-2582.txt


 HDFS-2231 started the process of adding configuration for HA, but one piece 
 is missing. The current state of the configuration is, I believe:
 {{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
 {{dfs.federation.nameservices}} - a list of federated nameservices
 {{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific 
 config for the given namenode. If HA or federation is disabled, the extra 
 components can be elided for backwards compatibility.
 The issue here is that there is no easy way to discern which NN is paired 
 with which other NN. Additionally, adding a new federated nameservice to a 
 config will require changes to {{dfs.ha.namenodes}} which makes templating 
 harder. It would be simpler to change {{dfs.ha.namenodes}} to be 
 nameservice-scoped: {{dfs.ha.namenodes.nameservice-id}}.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.

[
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158067#comment-13158067
]

Todd Lipcon commented on HDFS-1975:
---

bq. How about update the generation stamp after logging the new block

The issue is that we need the new block to have the new genstamp, so we have to
call {{nextGenerationStamp}} before we create the new block. Alternatively, we
could split up {{nextGenerationStamp}} into two parts -- one which increments,
and another which logs. But then we may have an issue if there is a crash so
the edit log includes the block allocation but not the SET_GENERATION_STAMP.

I've temporarily worked around this by changing the FSEditLogLoader code to
only call {{notifyGenStampUpdate}} in {{OP_ADD}}, not in {{OP_SET_GENSTAMP}}.
So, the new block is added to the block manager before it's notified as OK to
handle the DN messages.

*BUT* -- there's another issue with using genstamps as our gating mechanism
for the DN messages - comment to follow.

HA: Support for sharing the namenode state from active to standby.
--

Key: HDFS-1975
URL: https://issues.apache.org/jira/browse/HDFS-1975
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Reporter: Suresh Srinivas
Assignee: Jitendra Nath Pandey
Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch,
HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, hdfs-1975.txt,
hdfs-1975.txt

To enable hot standby namenode, the standby node must have current
information for - namenode state (image + edits) and block location
information. This jira addresses keeping the namenode state current in the
standby node. To do this, the proposed solution in this jira is to use a
shared storage to store the namenode state.
Note one could also build an alternative solution by augmenting the backup
node. A seperate jira could explore this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.

[
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158069#comment-13158069
]

Todd Lipcon commented on HDFS-1975:
---

I got this patch in conjunction with HDFS-1108 and HDFS-1971 to properly
replicate the creation of a new file, but then moved on to working on
{{setReplication}} and ran into issues there. The issue I'm seeing is this:

1) Active NN receives setReplication to drop some file's replication from 3 to 1
2) It writes OP_SET_REPLICATION to its log, invalidates two replicas, and
returns
3) The DNs report BLOCK_INVALIDATED back to both the ActiveNN and SBNN.
4) The SBNN hasn't received the OP_SET_REPLICATION yet, so it marks the block
as under-replicated.

In the case of raising replication (eg from 1 to 3) we get the opposite
problem: the SBNN marks the block as over-replicated and adds two of the
replicas to its invalidation list.

Generation stamps don't help here, because changing replication level of a
block doesn't change its gen-stamp (and it shouldn't).

The solution I'm thinking of is that we have to track the transaction ID when
we send comments to DNs. So, if a setReplication command at txid=123 causes
invalidation of two blocks, we'd send the INVALIDATE command with txid=123.
Then, when the DN does delete these blocks, it would ack back with that txid to
both NNs. The SBNN wouldn't process this message until it had loaded that txid.

A bit of a simplification from this would be that any command being processed
from an NN will include the NN's txid, which the DN records in BPOfferService
as latestCommandTxId. Then, any calls to the NN would include this txid. This
is a bit more conservative than tracking it with each block command, but
probably less prone to bugs.

I plan to take a pass at implementing this latter approach.

HA: Support for sharing the namenode state from active to standby.
--

[jira] [Updated] (HDFS-2591) HA: MiniDFSCluster support to mix and match federation with HA


 [ 
https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-2591:
--

Attachment: hdfs-2591-v2.txt

v2 patch fixes a number of issues discovered trying to build HA test cases on 
top of this. In particular, it has code to set up shared edits directories (to 
be enabled by HDFS-1971), as well as initialize the SBNN by copying one of the 
formatted namedirs from the active NN. (otherwise the two NNs would have 
different nsids/bpids and not be able to failover).

 HA: MiniDFSCluster support to mix and match federation with HA
 --

 Key: HDFS-2591
 URL: https://issues.apache.org/jira/browse/HDFS-2591
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha, test
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: hdfs-2591-v2.txt, hdfs-2591.txt


 Right now the MiniDFS builder is somewhat inflexible - it just takes a 
 numNameNodes parameter, which is used to specify federated nameservices. In 
 order to add HA support, we need to be able to be more specific when 
 configuring the NNs -- eg to test the case where there is one nameservice 
 that is HA and another which is not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1971) HA: Send block report from datanode to both active and standby namenodes


 [ 
https://issues.apache.org/jira/browse/HDFS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-1971:
--

Attachment: dualbr5.txt

This patch just has a few small cleanups from dualbr4.patch.

It still leaves the refreshNamenodes bit TODO, and adds a few more TODOs where 
we have to sort out what to do in terms of HA. I think it's worth committing 
this now, even with the TODOs, so we can address the TODOs in parallel. It will 
unblock other work and allow us to start figuring out more subtle issues.

I can file follow-up JIRAs for each of the TODOs when this is committed.

I've been testing this patch on top of HDFS-2582 and HDFS-2591 in that order, 
though I think it only strictly depends on HDFS-2582

 HA: Send block report from datanode to both active and standby namenodes
 

 Key: HDFS-1971
 URL: https://issues.apache.org/jira/browse/HDFS-1971
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: data-node, name-node
Reporter: Suresh Srinivas
Assignee: Sanjay Radia
 Attachments: DualBlockReports.pdf, daulBr1.patch, dualBr2.patch, 
 dualBr3.patch, dualbr4.txt, dualbr5.txt


 To enable hot standby namenode, the standby node must have current 
 information for - namenode state (image + edits) and block location 
 information. This jira addresses keeping the block location information 
 current in the standby node. To do this, the proposed solution is to send 
 block reports from the datanodes to both the active and the standby namenode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.


 [ 
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-1975:
--

Attachment: HDFS-1975-HDFS-1623.patch

Attached patch rebased on top of latest HDFS-2582, HDFS-2591, HDFS-1971, in 
that order.

 HA: Support for sharing the namenode state from active to standby.
 --

 Key: HDFS-1975
 URL: https://issues.apache.org/jira/browse/HDFS-1975
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: name-node
Reporter: Suresh Srinivas
Assignee: Jitendra Nath Pandey
 Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch, 
 HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, 
 HDFS-1975-HDFS-1623.patch, hdfs-1975.txt, hdfs-1975.txt


 To enable hot standby namenode, the standby node must have current 
 information for - namenode state (image + edits) and block location 
 information. This jira addresses keeping the namenode state current in the 
 standby node. To do this, the proposed solution in this jira is to use a 
 shared storage to store the namenode state. 
 Note one could also build an alternative solution by augmenting the backup 
 node. A seperate jira could explore this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1108) Log newly allocated blocks


 [ 
https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-1108:
--

Attachment: hdfs-1108-habranch.txt

Updated patch on top of HDFS-2582 - same patch except for trivial change that 
we need to pass the namespace id to HAUtil.isHaEnabled now. After HDFS-2582 
goes in, I'll commit this based on the earlier +1s from folks

 Log newly allocated blocks
 --

 Key: HDFS-1108
 URL: https://issues.apache.org/jira/browse/HDFS-1108
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: name-node
Reporter: dhruba borthakur
Assignee: Todd Lipcon
 Fix For: HA branch (HDFS-1623)

 Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, 
 hdfs-1108-habranch.txt, hdfs-1108-habranch.txt, hdfs-1108.txt


 The current HDFS design says that newly allocated blocks for a file are not 
 persisted in the NN transaction log when the block is allocated. Instead, a 
 hflush() or a close() on the file persists the blocks into the transaction 
 log. It would be nice if we can immediately persist newly allocated blocks 
 (as soon as they are allocated) for specific files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.

[
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158153#comment-13158153
]

Todd Lipcon commented on HDFS-1975:
---

another possible answer is that we need to modify
FSNamesystem.isPopulatingReplQueues to return false on the standby, and then
when it switches from standby to active, initialize the replication queues only
after reading the latest edits... I think that will solve the SET_REPLICATION
issue, but not certain if it will solve all the issues in this general class.

HA: Support for sharing the namenode state from active to standby.
--

[jira] [Commented] (HDFS-1108) Log newly allocated blocks

2011-11-27 Thread Hadoop QA (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158154#comment-13158154
]

Hadoop QA commented on HDFS-1108:
-

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12505289/hdfs-1108-habranch.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1610//console

This message is automatically generated.

Log newly allocated blocks
--

Key: HDFS-1108
URL: https://issues.apache.org/jira/browse/HDFS-1108
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Reporter: dhruba borthakur
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623)

Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt,
hdfs-1108-habranch.txt, hdfs-1108-habranch.txt, hdfs-1108.txt

The current HDFS design says that newly allocated blocks for a file are not
persisted in the NN transaction log when the block is allocated. Instead, a
hflush() or a close() on the file persists the blocks into the transaction
log. It would be nice if we can immediately persist newly allocated blocks
(as soon as they are allocated) for specific files.

[jira] [Commented] (HDFS-1108) Log newly allocated blocks


[ 
https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158156#comment-13158156
 ] 

Eli Collins commented on HDFS-1108:
---

Doesn't the test need to enable HA and shared edits dir or is it only covering 
the syncs triggered by supportAppends?

 Log newly allocated blocks
 --

 Key: HDFS-1108
 URL: https://issues.apache.org/jira/browse/HDFS-1108
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: name-node
Reporter: dhruba borthakur
Assignee: Todd Lipcon
 Fix For: HA branch (HDFS-1623)

 Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, 
 hdfs-1108-habranch.txt, hdfs-1108-habranch.txt, hdfs-1108.txt


 The current HDFS design says that newly allocated blocks for a file are not 
 persisted in the NN transaction log when the block is allocated. Instead, a 
 hflush() or a close() on the file persists the blocks into the transaction 
 log. It would be nice if we can immediately persist newly allocated blocks 
 (as soon as they are allocated) for specific files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.

[
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158168#comment-13158168
]

Todd Lipcon commented on HDFS-1975:
---

Another issue that we have to tackle before this can provide hot standby is
this:

When we close a file, or add another block to a file, we write OP_CLOSE or
OP_ADD in the txn log. FSEditLogLoader, when it sees these types of
transactions, creates new BlockInfo objects for all of the blocks listed in the
transaction. These new BlockInfos have no block locations associated. So, when
we close a file, the SBNN loses its block locations info for that file and is
no longer hot.

I have an ugly hack which copies over the old BlockInfos from the existing
INode, but I'm not convinced it's the right way. It might be cleaner to add new
opcode types like OP_ADD_ADDITIONAL_BLOCK, and actually treat OP_CLOSE as just
a finalization of INodeFileUnderConstruction to INodeFile, rather than
replacing block info at all.

Thoughts?

HA: Support for sharing the namenode state from active to standby.
--

[jira] [Updated] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly


 [ 
https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HDFS-2246:
-

Target Version/s: 0.23.1, 1.0.0  (was: 0.23.1, 0.24.0, 1.0.0)
   Fix Version/s: (was: 0.24.0)
  1.0.0
  0.23.1

corrected Fix Versions field

 Shortcut a local client reads to a Datanodes files directly
 ---

 Key: HDFS-2246
 URL: https://issues.apache.org/jira/browse/HDFS-2246
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Sanjay Radia
Assignee: Jitendra Nath Pandey
 Fix For: 0.23.1, 1.0.0

 Attachments: 0001-HDFS-347.-Local-reads.patch, 
 HDFS-2246-branch-0.20-security-205.1.patch, 
 HDFS-2246-branch-0.20-security-205.2.patch, 
 HDFS-2246-branch-0.20-security-205.patch, 
 HDFS-2246-branch-0.20-security-205.patch, 
 HDFS-2246-branch-0.20-security-205.patch, 
 HDFS-2246-branch-0.20-security.3.patch, 
 HDFS-2246-branch-0.20-security.no-softref.patch, 
 HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, 
 HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, 
 HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, 
 HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, 
 HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, 
 HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, 
 localReadShortcut20-security.2patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2588) hdfs jsp pages missing DOCTYPE [post-split branches]


 [ 
https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HDFS-2588:
-

Target Version/s: 0.23.1
   Fix Version/s: (was: 1.0.0)
 Summary: hdfs jsp pages missing DOCTYPE [post-split branches]  
(was: hdfs jsp pages missing DOCTYPE)

Corrected this bug to only relate to post-split branches/trunk.
The parent bug only will be used for pre-split 1.0.

 hdfs jsp pages missing DOCTYPE [post-split branches]
 

 Key: HDFS-2588
 URL: https://issues.apache.org/jira/browse/HDFS-2588
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: scripts
Reporter: Dave Vronay
Assignee: Dave Vronay
Priority: Trivial
 Fix For: 0.23.1


 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the 
 pages to render incorrectly on some browsers, such as IE9.  Please see parent 
 bug HADOOP-7827 for details and patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP

2011-11-27 Thread Matt Foley (Reopened) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reopened HDFS-2316:
--


re-opening to accomodate request to fix also in 0.22.0.

 webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
 --

 Key: HDFS-2316
 URL: https://issues.apache.org/jira/browse/HDFS-2316
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
  Labels: critical-0.22.0
 Fix For: 0.23.1, 1.0.0

 Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, 
 WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s


 We current have hftp for accessing HDFS over HTTP.  However, hftp is a 
 read-only FileSystem and does not provide write accesses.
 In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem 
 implementation for accessing HDFS over HTTP.  The is the umbrella JIRA for 
 the tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP


 [ 
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HDFS-2316:
-

Target Version/s: 0.23.0, 0.22.0, 1.0.0  (was: 0.22.0, 1.0.0, 0.23.0)
Release Note: 
Provide webhdfs as a complete FileSystem implementation for accessing HDFS over 
HTTP.
Previous hftp feature was a read-only FileSystem and does not provide write 
accesses.

 webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
 --

 Key: HDFS-2316
 URL: https://issues.apache.org/jira/browse/HDFS-2316
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
  Labels: critical-0.22.0
 Fix For: 0.23.1, 1.0.0

 Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, 
 WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s


 We current have hftp for accessing HDFS over HTTP.  However, hftp is a 
 read-only FileSystem and does not provide write accesses.
 In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem 
 implementation for accessing HDFS over HTTP.  The is the umbrella JIRA for 
 the tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP


 [ 
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HDFS-2316:
-

Target Version/s: 0.23.0, 0.22.0, 1.0.0  (was: 1.0.0, 0.22.0, 0.23.0)
 Summary: [umbrella] webhdfs: a complete FileSystem implementation 
for accessing HDFS over HTTP  (was: webhdfs: a complete FileSystem 
implementation for accessing HDFS over HTTP)

 [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS 
 over HTTP
 -

 Key: HDFS-2316
 URL: https://issues.apache.org/jira/browse/HDFS-2316
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
  Labels: critical-0.22.0
 Fix For: 0.23.1, 1.0.0

 Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, 
 WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s


 We current have hftp for accessing HDFS over HTTP.  However, hftp is a 
 read-only FileSystem and does not provide write accesses.
 In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem 
 implementation for accessing HDFS over HTTP.  The is the umbrella JIRA for 
 the tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2596) TestDirectoryScanner doesn't test parallel scans

2011-11-27 Thread Eli Collins (Created) (JIRA)

TestDirectoryScanner doesn't test parallel scans


 Key: HDFS-2596
 URL: https://issues.apache.org/jira/browse/HDFS-2596
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node, test
Affects Versions: 0.23.0, 0.22.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2596-1.patch

The code from HDFS-854 below doesn't run the test with parallel scanning. They 
probably intended parallelism  3.

{code}
+  public void testDirectoryScanner() throws Exception {
+// Run the test with and without parallel scanning
+for (int parallelism = 1; parallelism  2; parallelism++) {
+  runTest(parallelism);
+}
+  }
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2596) TestDirectoryScanner doesn't test parallel scans

2011-11-27 Thread Eli Collins (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins updated HDFS-2596:
--

Attachment: hdfs-2596-1.patch

Patch attached.

 TestDirectoryScanner doesn't test parallel scans
 

 Key: HDFS-2596
 URL: https://issues.apache.org/jira/browse/HDFS-2596
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node, test
Affects Versions: 0.22.0, 0.23.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2596-1.patch


 The code from HDFS-854 below doesn't run the test with parallel scanning. 
 They probably intended parallelism  3.
 {code}
 +  public void testDirectoryScanner() throws Exception {
 +// Run the test with and without parallel scanning
 +for (int parallelism = 1; parallelism  2; parallelism++) {
 +  runTest(parallelism);
 +}
 +  }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP


[ 
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158219#comment-13158219
 ] 

Hudson commented on HDFS-2316:
--

Integrated in Hadoop-Hdfs-0.23-Commit #217 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/217/])
HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo 
(Nicholas), Sze.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1206989
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS 
 over HTTP
 -

 Key: HDFS-2316
 URL: https://issues.apache.org/jira/browse/HDFS-2316
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
  Labels: critical-0.22.0
 Fix For: 0.23.1, 1.0.0

 Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, 
 WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s


 We current have hftp for accessing HDFS over HTTP.  However, hftp is a 
 read-only FileSystem and does not provide write accesses.
 In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem 
 implementation for accessing HDFS over HTTP.  The is the umbrella JIRA for 
 the tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP


[ 
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158222#comment-13158222
 ] 

Hudson commented on HDFS-2316:
--

Integrated in Hadoop-Common-0.23-Commit #220 (See 
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/220/])
HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo 
(Nicholas), Sze.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1206989
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS 
 over HTTP
 -

 Key: HDFS-2316
 URL: https://issues.apache.org/jira/browse/HDFS-2316
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
  Labels: critical-0.22.0
 Fix For: 0.23.1, 1.0.0

 Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, 
 WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s


 We current have hftp for accessing HDFS over HTTP.  However, hftp is a 
 read-only FileSystem and does not provide write accesses.
 In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem 
 implementation for accessing HDFS over HTTP.  The is the umbrella JIRA for 
 the tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP


[ 
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158231#comment-13158231
 ] 

Hudson commented on HDFS-2316:
--

Integrated in Hadoop-Hdfs-trunk-Commit #1403 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1403/])
HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo 
(Nicholas), Sze.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1206990
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS 
 over HTTP
 -

 Key: HDFS-2316
 URL: https://issues.apache.org/jira/browse/HDFS-2316
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
  Labels: critical-0.22.0
 Fix For: 0.23.1, 1.0.0

 Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, 
 WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s


 We current have hftp for accessing HDFS over HTTP.  However, hftp is a 
 read-only FileSystem and does not provide write accesses.
 In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem 
 implementation for accessing HDFS over HTTP.  The is the umbrella JIRA for 
 the tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP


[ 
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158233#comment-13158233
 ] 

Hudson commented on HDFS-2316:
--

Integrated in Hadoop-Common-trunk-Commit #1329 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1329/])
HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo 
(Nicholas), Sze.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1206990
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS 
 over HTTP
 -

 Key: HDFS-2316
 URL: https://issues.apache.org/jira/browse/HDFS-2316
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
  Labels: critical-0.22.0
 Fix For: 0.23.1, 1.0.0

 Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, 
 WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s


 We current have hftp for accessing HDFS over HTTP.  However, hftp is a 
 read-only FileSystem and does not provide write accesses.
 In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem 
 implementation for accessing HDFS over HTTP.  The is the umbrella JIRA for 
 the tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP