[jira] [Commented] (HDFS-4145) Merge hdfs cmd line scripts from branch-1-win
[ https://issues.apache.org/jira/browse/HDFS-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490145#comment-13490145 ] Chris Nauroth commented on HDFS-4145: - Ivan, I think there is an issue with handling of the --config option to override the configuration directory. hdfs.cmd has code to parse --config and then shift away the arguments. Later, it calls hadoop-config.cmd. Inside hadoop-config.cmd, it resets HADOOP_CONF_DIR to a default of %HADOOP_HOME%\etc\hadoop. hadoop-config.cmd then repeats the check for --config, but by now hdfs.cmd has already shifted the arguments out, so the config directory override isn't found. I temporarily hacked around this on my machine by commenting out the line in hadoop-config.cmd that sets it to %HADOOP_HOME%\etc\hadoop. After that, it worked for me. I successfully formatted and started a namenode. Merge hdfs cmd line scripts from branch-1-win - Key: HDFS-4145 URL: https://issues.apache.org/jira/browse/HDFS-4145 Project: Hadoop HDFS Issue Type: Bug Affects Versions: trunk-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HDFS-4145.branch-trunk-win.scripts.2.patch, HDFS-4145.branch-trunk-win.scripts.patch Tracking Jira for merging hdfs cmd line scripts from branch-1-win to trunk. Scripts also have to be updated to reflect their unix equivalents. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4147) Deletion of snapshottable dir with snapshots should fail
Jing Zhao created HDFS-4147: --- Summary: Deletion of snapshottable dir with snapshots should fail Key: HDFS-4147 URL: https://issues.apache.org/jira/browse/HDFS-4147 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao Deletion of snapshottable dir with snapshots should fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4148) disallow modification on RO snapshots
Brandon Li created HDFS-4148: Summary: disallow modification on RO snapshots Key: HDFS-4148 URL: https://issues.apache.org/jira/browse/HDFS-4148 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Snapshot (HDFS-2802) Reporter: Brandon Li Assignee: Brandon Li disallow modification on RO snapshots, including create, append, setReplication/Permission/Owner, rename, delete, makedir, setQuota/Time, createSymlink. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3979) Fix hsync and hflush semantics.
[ https://issues.apache.org/jira/browse/HDFS-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490268#comment-13490268 ] Tsz Wo (Nicholas), SZE commented on HDFS-3979: -- Sure, will check the patch. Fix hsync and hflush semantics. --- Key: HDFS-3979 URL: https://issues.apache.org/jira/browse/HDFS-3979 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client Affects Versions: 0.22.0, 0.23.0, 2.0.0-alpha Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: hdfs-3979-sketch.txt, hdfs-3979-v2.txt, hdfs-3979-v3.txt See discussion in HDFS-744. The actual sync/flush operation in BlockReceiver is not on a synchronous path from the DFSClient, hence it is possible that a DN loses data that it has already acknowledged as persisted to a client. Edit: Spelling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4146) Further work on changing INodeFile.blocks to private in Snapshot branch
[ https://issues.apache.org/jira/browse/HDFS-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4146: - Attachment: h4146_20121104.patch h4146_20121104.patch: use getter and setter in INodeFileWithLink to access blocks. Also initialize root directory as snapshottable since it cannot be replaced. Further work on changing INodeFile.blocks to private in Snapshot branch --- Key: HDFS-4146 URL: https://issues.apache.org/jira/browse/HDFS-4146 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4146_20121104.patch There was a conflict in INodeFile when merging HDFS-4143 so that I keep INodeFile.blocks as protected. Otherwise, some code cannot be compiled. It needs some more works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3979) Fix hsync and hflush semantics.
[ https://issues.apache.org/jira/browse/HDFS-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490285#comment-13490285 ] Tsz Wo (Nicholas), SZE commented on HDFS-3979: -- The patch moves ack to the end in order to fix the sync semantics. I think it will decrease the performance for non-sync write. How about keeping enqueue early when syncBlock == false? Fix hsync and hflush semantics. --- Key: HDFS-3979 URL: https://issues.apache.org/jira/browse/HDFS-3979 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client Affects Versions: 0.22.0, 0.23.0, 2.0.0-alpha Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: hdfs-3979-sketch.txt, hdfs-3979-v2.txt, hdfs-3979-v3.txt See discussion in HDFS-744. The actual sync/flush operation in BlockReceiver is not on a synchronous path from the DFSClient, hence it is possible that a DN loses data that it has already acknowledged as persisted to a client. Edit: Spelling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4146) Further work on changing INodeFile.blocks to private in Snapshot branch
[ https://issues.apache.org/jira/browse/HDFS-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490304#comment-13490304 ] Suresh Srinivas commented on HDFS-4146: --- +1 for the change. Further work on changing INodeFile.blocks to private in Snapshot branch --- Key: HDFS-4146 URL: https://issues.apache.org/jira/browse/HDFS-4146 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4146_20121104.patch There was a conflict in INodeFile when merging HDFS-4143 so that I keep INodeFile.blocks as protected. Otherwise, some code cannot be compiled. It needs some more works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4146) Further work on changing INodeFile.blocks to private in Snapshot branch
[ https://issues.apache.org/jira/browse/HDFS-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-4146. -- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed Suresh, thanks for the review. I have committed this. Further work on changing INodeFile.blocks to private in Snapshot branch --- Key: HDFS-4146 URL: https://issues.apache.org/jira/browse/HDFS-4146 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Fix For: Snapshot (HDFS-2802) Attachments: h4146_20121104.patch There was a conflict in INodeFile when merging HDFS-4143 so that I keep INodeFile.blocks as protected. Otherwise, some code cannot be compiled. It needs some more works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4149) Complete the code for disallowSnapshot
Tsz Wo (Nicholas), SZE created HDFS-4149: Summary: Complete the code for disallowSnapshot Key: HDFS-4149 URL: https://issues.apache.org/jira/browse/HDFS-4149 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Implement the disallowSnapshot(..) in FSNamesystem and add a resetSnapshottable(..) to SnapshotManager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4149) Complete the code for disallowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4149: - Attachment: h4149_20121104.patch h4149_20121104.patch: 1st patch. Complete the code for disallowSnapshot -- Key: HDFS-4149 URL: https://issues.apache.org/jira/browse/HDFS-4149 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h4149_20121104.patch Implement the disallowSnapshot(..) in FSNamesystem and add a resetSnapshottable(..) to SnapshotManager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4147) Deletion of snapshottable dir with snapshots should fail
[ https://issues.apache.org/jira/browse/HDFS-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4147: Attachment: HDFS-4147.001.patch Initial patch for this. Deletion of snapshottable dir with snapshots should fail Key: HDFS-4147 URL: https://issues.apache.org/jira/browse/HDFS-4147 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4147.001.patch Deletion of snapshottable dir with snapshots should fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4149) Complete the code for disallowSnapshot
[
https://issues.apache.org/jira/browse/HDFS-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490329#comment-13490329
]
Suresh Srinivas commented on HDFS-4149:
---
Comments:
# DistributedFileSystem.java - change the param name to createSnapshot from
snapshotRoot to root. I suggest making similar change in the FileSystem.java as
well.
# SnapshotManager#resetSnapshottable - Along with The directory has
snapshot(s)..., please also add, Please redo the operation after removing all
the snapshots.
# InodeDirectorySnapshottable
#* Member variable #snapshots - if there is a specific order in which snapshots
are stored (that decreasing order creation time), please add it in the javadoc
for this member variable.
#* Number of snapshots is allowed. should be Number of snapshots allowed.
#* It may be good to add javadoc to the InodeDirectorySnapshottable class to
say it is synchronized external by {@link SnapshotManager}
#* Make getNumSnapshots() package private so that only SnapshotManager can use
it with proper synchronization?
#* Javadoc for #getSnapshotRoot() to describe what is snapshot root would help.
Also change variable name name to snapshotName.
Complete the code for disallowSnapshot
--
Key: HDFS-4149
URL: https://issues.apache.org/jira/browse/HDFS-4149
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h4149_20121104.patch
Implement the disallowSnapshot(..) in FSNamesystem and add a
resetSnapshottable(..) to SnapshotManager.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4150) Update inode in blocksMap when deleting original/snapshot file
Jing Zhao created HDFS-4150: --- Summary: Update inode in blocksMap when deleting original/snapshot file Key: HDFS-4150 URL: https://issues.apache.org/jira/browse/HDFS-4150 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao When deleting a file/directory, instead of directly removing all the corresponding blocks, we should update inodes in blocksMap if there are snapshots for them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4149) Complete the code for disallowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4149: - Attachment: h4149_20121104b.patch h4149_20121104b.patch: incorporates the review comments and fixes some bug in dumpTreeRecursively(..). Complete the code for disallowSnapshot -- Key: HDFS-4149 URL: https://issues.apache.org/jira/browse/HDFS-4149 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h4149_20121104b.patch, h4149_20121104.patch Implement the disallowSnapshot(..) in FSNamesystem and add a resetSnapshottable(..) to SnapshotManager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4147) Deletion of snapshottable dir with snapshots should fail
[ https://issues.apache.org/jira/browse/HDFS-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490337#comment-13490337 ] Tsz Wo (Nicholas), SZE commented on HDFS-4147: -- We cannot throw IOException in unprotectedDelete since it is also used by edit log loading. The check should be done before it. Deletion of snapshottable dir with snapshots should fail Key: HDFS-4147 URL: https://issues.apache.org/jira/browse/HDFS-4147 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4147.001.patch Deletion of snapshottable dir with snapshots should fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4149) Complete the code for disallowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490344#comment-13490344 ] Suresh Srinivas commented on HDFS-4149: --- +1 for the patch. Complete the code for disallowSnapshot -- Key: HDFS-4149 URL: https://issues.apache.org/jira/browse/HDFS-4149 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h4149_20121104b.patch, h4149_20121104.patch Implement the disallowSnapshot(..) in FSNamesystem and add a resetSnapshottable(..) to SnapshotManager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4149) Complete the code for disallowSnapshot
[ https://issues.apache.org/jira/browse/HDFS-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-4149. -- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed I have committed this. Complete the code for disallowSnapshot -- Key: HDFS-4149 URL: https://issues.apache.org/jira/browse/HDFS-4149 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: Snapshot (HDFS-2802) Attachments: h4149_20121104b.patch, h4149_20121104.patch Implement the disallowSnapshot(..) in FSNamesystem and add a resetSnapshottable(..) to SnapshotManager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4147) Deletion of snapshottable dir with snapshots should fail
[ https://issues.apache.org/jira/browse/HDFS-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4147: Attachment: HDFS-4147.002.patch Thanks for the comments Nicholas! New patch uploaded. Deletion of snapshottable dir with snapshots should fail Key: HDFS-4147 URL: https://issues.apache.org/jira/browse/HDFS-4147 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4147.001.patch, HDFS-4147.002.patch Deletion of snapshottable dir with snapshots should fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4147) Deletion of snapshottable dir with snapshots should fail
[ https://issues.apache.org/jira/browse/HDFS-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4147: - Component/s: (was: data-node) Hadoop Flags: Reviewed +1 patch looks good. Deletion of snapshottable dir with snapshots should fail Key: HDFS-4147 URL: https://issues.apache.org/jira/browse/HDFS-4147 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Jing Zhao Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4147.001.patch, HDFS-4147.002.patch Deletion of snapshottable dir with snapshots should fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4147) Deletion of snapshottable dir with snapshots should fail
[ https://issues.apache.org/jira/browse/HDFS-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-4147. -- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) I have committed this. Thanks, Jing! Deletion of snapshottable dir with snapshots should fail Key: HDFS-4147 URL: https://issues.apache.org/jira/browse/HDFS-4147 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Jing Zhao Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4147.001.patch, HDFS-4147.002.patch Deletion of snapshottable dir with snapshots should fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4114) Deprecate the BackupNode and CheckpointNode in 2.0
[ https://issues.apache.org/jira/browse/HDFS-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490406#comment-13490406 ] Konstantin Shvachko commented on HDFS-4114: --- Your argument about efficiency isn't valid given HDFS-1458. It is. SNN still needs to upload edits. BN does not need to upload anything. The BlockScanner argument is a silly one: Todd, would you please try to keep this more civilized. It is not an argument, it is an example. it has had some bugs That is my point. BN has some bugs as well, introduced while the code evolved. It is not a valid reason to remove it. How about start deprecating it in 2.0? I disagree. It is just twisting the same idea. You are deprecating it in the past to remove it in the present. There is no alternative to BackupNode as an HA solution without shared storage, besides checkpointing. I'll look at the test cases listed here. I will re-purpose HDFS-2064 for 0.23 and submit a patch soon. Starting from Hadoop 0.23 there is no need for load replicator, as DNs can talk to multiple NameNodes internally. Deprecate the BackupNode and CheckpointNode in 2.0 -- Key: HDFS-4114 URL: https://issues.apache.org/jira/browse/HDFS-4114 Project: Hadoop HDFS Issue Type: Bug Reporter: Eli Collins Assignee: Eli Collins Per the thread on hdfs-dev@ (http://s.apache.org/tMT) let's remove the BackupNode and CheckpointNode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4151) Passing INodesInPath instead of INode[] in FSDirectory
Tsz Wo (Nicholas), SZE created HDFS-4151: Summary: Passing INodesInPath instead of INode[] in FSDirectory Key: HDFS-4151 URL: https://issues.apache.org/jira/browse/HDFS-4151 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Currently, many methods in FSDirectory pass INode[] as a parameter. It is better to pass INodesInPath so that we can add more path information later on. This is especially useful in Snapshot implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4151) Passing INodesInPath instead of INode[] in FSDirectory
[ https://issues.apache.org/jira/browse/HDFS-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4151: - Status: Patch Available (was: Open) Passing INodesInPath instead of INode[] in FSDirectory -- Key: HDFS-4151 URL: https://issues.apache.org/jira/browse/HDFS-4151 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4151_20121104.patch Currently, many methods in FSDirectory pass INode[] as a parameter. It is better to pass INodesInPath so that we can add more path information later on. This is especially useful in Snapshot implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4151) Passing INodesInPath instead of INode[] in FSDirectory
[ https://issues.apache.org/jira/browse/HDFS-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4151: - Attachment: h4151_20121104.patch h4151_20121104.patch: replaces INode[] with INodesInPath except for the verifyQuota/verifyFsLimits methods Passing INodesInPath instead of INode[] in FSDirectory -- Key: HDFS-4151 URL: https://issues.apache.org/jira/browse/HDFS-4151 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4151_20121104.patch Currently, many methods in FSDirectory pass INode[] as a parameter. It is better to pass INodesInPath so that we can add more path information later on. This is especially useful in Snapshot implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3979) Fix hsync and hflush semantics.
[ https://issues.apache.org/jira/browse/HDFS-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490435#comment-13490435 ] Kan Zhang commented on HDFS-3979: - bq. I think it will decrease the performance for non-sync write. I'd welcome some clarity on whether writing to OS buffers is a real concern here. bq. How about keeping enqueue early when syncBlock == false? To be on the conservative side, I'm OK with this. Fix hsync and hflush semantics. --- Key: HDFS-3979 URL: https://issues.apache.org/jira/browse/HDFS-3979 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client Affects Versions: 0.22.0, 0.23.0, 2.0.0-alpha Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: hdfs-3979-sketch.txt, hdfs-3979-v2.txt, hdfs-3979-v3.txt See discussion in HDFS-744. The actual sync/flush operation in BlockReceiver is not on a synchronous path from the DFSClient, hence it is possible that a DN loses data that it has already acknowledged as persisted to a client. Edit: Spelling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3979) Fix hsync and hflush semantics.
[ https://issues.apache.org/jira/browse/HDFS-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490445#comment-13490445 ] Lars Hofhansl commented on HDFS-3979: - I'll make that change. Fix hsync and hflush semantics. --- Key: HDFS-3979 URL: https://issues.apache.org/jira/browse/HDFS-3979 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client Affects Versions: 0.22.0, 0.23.0, 2.0.0-alpha Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: hdfs-3979-sketch.txt, hdfs-3979-v2.txt, hdfs-3979-v3.txt See discussion in HDFS-744. The actual sync/flush operation in BlockReceiver is not on a synchronous path from the DFSClient, hence it is possible that a DN loses data that it has already acknowledged as persisted to a client. Edit: Spelling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3979) Fix hsync and hflush semantics.
[ https://issues.apache.org/jira/browse/HDFS-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-3979: Attachment: hdfs-3979-v4.txt Updated patch with Nicholas' suggestion. I agree that the previous patch would have slowed all writes that reach the DN. We can't distinguish between an hflush from the client and normal packet from the client. On the other hand this no longer deals with Luke's kill -9 scenario (where a cluster management tool would kill -9 datanodes in parallel), but in the end no tool really should do that. Fix hsync and hflush semantics. --- Key: HDFS-3979 URL: https://issues.apache.org/jira/browse/HDFS-3979 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client Affects Versions: 0.22.0, 0.23.0, 2.0.0-alpha Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: hdfs-3979-sketch.txt, hdfs-3979-v2.txt, hdfs-3979-v3.txt, hdfs-3979-v4.txt See discussion in HDFS-744. The actual sync/flush operation in BlockReceiver is not on a synchronous path from the DFSClient, hence it is possible that a DN loses data that it has already acknowledged as persisted to a client. Edit: Spelling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4151) Passing INodesInPath instead of INode[] in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490457#comment-13490457
]
Hadoop QA commented on HDFS-4151:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12552059/h4151_20121104.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.TestFcHdfsSymlink
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3441//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3441//console
This message is automatically generated.
Passing INodesInPath instead of INode[] in FSDirectory
--
Key: HDFS-4151
URL: https://issues.apache.org/jira/browse/HDFS-4151
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
Attachments: h4151_20121104.patch
Currently, many methods in FSDirectory pass INode[] as a parameter. It is
better to pass INodesInPath so that we can add more path information later
on. This is especially useful in Snapshot implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
