date:20140108

Uma Maheswara Rao G created HDFS-5730:
-

 Summary: Inconsistent Audit logging for HDFS APIs
 Key: HDFS-5730
 URL: https://issues.apache.org/jira/browse/HDFS-5730
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0, 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G


When looking at the audit loggs in HDFS, I am seeing some inconsistencies what 
was logged with audit and what is added recently.

For more details please check the comments.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5730) Inconsistent Audit logging for HDFS APIs


[ 
https://issues.apache.org/jira/browse/HDFS-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865273#comment-13865273
 ] 

Uma Maheswara Rao G commented on HDFS-5730:
---

HDFS audit logging interface:
{code}
 /**
   * Same as
   * {@link #logAuditEvent(boolean, String, InetAddress, String, String, 
String, FileStatus)}
   * with additional parameters related to logging delegation token tracking
   * IDs.
   * 
   * @param succeeded Whether authorization succeeded.
   * @param userName Name of the user executing the request.
   * @param addr Remote address of the request.
   * @param cmd The requested command.
   * @param src Path of affected source file.
   * @param dst Path of affected destination file (if any).
   * @param stat File information for operations that change the file's metadata
   *  (permissions, owner, times, etc).
   * @param ugi UserGroupInformation of the current user, or null if not logging
   *  token tracking information
   * @param dtSecretManager The token secret manager, or null if not logging
   *  token tracking information
   */
  public abstract void logAuditEvent(boolean succeeded, String userName,
  InetAddress addr, String cmd, String src, String dst,
  FileStatus stat, UserGroupInformation ugi,
  DelegationTokenSecretManager dtSecretManager);
{code}

Here succeeded parameter indicates whether Authorization check succeeded.

Recent APIs like addCacheDirective, modifyCacheDirective, 
removeCacheDirective..etc are used that parameter to indicate whether op 
succeeded or not.

{code}
 boolean success = false;
 ...
 writeLock();
try {
  checkOperation(OperationCategory.WRITE);
  if (isInSafeMode()) {
throw new SafeModeException(
Cannot add cache directive, safeMode);
  }
  cacheManager.modifyDirective(directive, pc, flags);
  getEditLog().logModifyCacheDirectiveInfo(directive,
  cacheEntry != null);
  success = true;
} finally {
  writeUnlock();
  if (success) {
getEditLog().logSync();
  }
  if (isAuditEnabled()  isExternalInvocation()) {
logAuditEvent(success, modifyCacheDirective, null, null, null);
  }
  RetryCache.setState(cacheEntry, success);
}

{code}

But all the older APIs like startFile..etc handled the AccessControlException 
explicitly and passed the first parameter value as false if failure. No log for 
other IOE.


Also snapShot related APIs followed other pattern. Here we just logged only on 
success.

{code}
String createSnapshot(String snapshotRoot, String snapshotName)
  throws SafeModeException, IOException {
  ..
  .
getEditLog().logSync();

if (auditLog.isInfoEnabled()  isExternalInvocation()) {
  logAuditEvent(true, createSnapshot, snapshotRoot, snapshotPath, null);
}
return snapshotPath;
  }
{code}

So, we have to unify the audit logging here in all APIs.

 Inconsistent Audit logging for HDFS APIs
 

 Key: HDFS-5730
 URL: https://issues.apache.org/jira/browse/HDFS-5730
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 3.0.0, 2.2.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G

 When looking at the audit loggs in HDFS, I am seeing some inconsistencies 
 what was logged with audit and what is added recently.
 For more details please check the comments.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5721) sharedEditsImage in Namenode#initializeSharedEdits() should be closed before method returns


[ 
https://issues.apache.org/jira/browse/HDFS-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865281#comment-13865281
 ] 

Uma Maheswara Rao G commented on HDFS-5721:
---

{quote}
There are also other places with the similar issues that not get close in 
finally block. i.e. Namenode#Format(), FSNamesystem# loadFromDisk(), etc. I 
think we should fix all these similar issues in one JIRA
{quote}
I agree to close the streams.  Actually in most of this cases, JVM will 
terminate immediately after the command execution (ex: format ..etc). It will 
not run system for log with the leaked streams. But if we face any issue due to 
streams because of not closing them, closing would be fine now. Am i missed 
something here?


 sharedEditsImage in Namenode#initializeSharedEdits() should be closed before 
 method returns
 ---

 Key: HDFS-5721
 URL: https://issues.apache.org/jira/browse/HDFS-5721
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Attachments: hdfs-5721-v1.txt, hdfs-5721-v2.txt


 At line 901:
 {code}
   FSImage sharedEditsImage = new FSImage(conf,
   Lists.URInewArrayList(),
   sharedEditsDirs);
 {code}
 sharedEditsImage is not closed before the method returns.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5729) Lower chance to hit NPE in allocateNodeLocal


[ 
https://issues.apache.org/jira/browse/HDFS-5729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865288#comment-13865288
 ] 

Uma Maheswara Rao G commented on HDFS-5729:
---

Should we move this to YARN?

 Lower chance to hit NPE in allocateNodeLocal 
 -

 Key: HDFS-5729
 URL: https://issues.apache.org/jira/browse/HDFS-5729
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: wenwupeng
 Attachments: conf.tar.gz, log.tar.gz


 we have lower chance to hit NPE in allocateNodeLocal  when run benchmark(hit 
 4 in 20 times).
 Steps:
 1. setup hadoop 2.2.0 environment
 2. Run for i in {1..10}; do /hadoop/hadoop-smoke/bin/hadoop jar 
 /hadoop/hadoop-smoke/share/hadoop/mapreduce/hadoop-mapreduce-client-common-*.jar
  org.apache.hadoop.fs.TestDFSIO -write -nrFiles 30 -fileSize 64MB; sleep 
 10;done
 2014-01-08 03:56:14,082 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateNodeLocal(AppSchedulingInfo.java:291)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:252)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:294)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.assignContainer(FifoScheduler.java:614)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.assignNodeLocalContainers(FifoScheduler.java:524)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.assignContainersOnNode(FifoScheduler.java:482)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.assignContainers(FifoScheduler.java:419)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.nodeUpdate(FifoScheduler.java:658)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:687)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:95)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
 at java.lang.Thread.run(Thread.java:662)
 will attach log and configure files later
 Note: 
 My topology file:
 10.111.89.230   /QE1/sin2-pekaurora-bdcqe046.eng.vmware.com
 10.111.89.231   /QE1/sin2-pekaurora-bdcqe046.eng.vmware.com
 10.111.89.232   /QE1/sin2-pekaurora-bdcqe046.eng.vmware.com
 10.111.89.239   /QE1/sin2-pekaurora-bdcqe046.eng.vmware.com
 10.111.89.233   /QE1/sin2-pekaurora-bdcqe017.eng.vmware.com
 10.111.89.234   /QE1/sin2-pekaurora-bdcqe017.eng.vmware.com
 10.111.89.240   /QE1/sin2-pekaurora-bdcqe017.eng.vmware.com
 10.111.89.236   /QE2/sin2-pekaurora-bdcqe047.eng.vmware.com
 10.111.89.241   /QE2/sin2-pekaurora-bdcqe047.eng.vmware.com
 10.111.89.238   /QE2/sin2-pekaurora-bdcqe048.eng.vmware.com
 10.111.89.242   /QE2/sin2-pekaurora-bdcqe048.eng.vmware.com



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5726) Fix compilation error in AbstractINodeDiff for JDK7


[ 
https://issues.apache.org/jira/browse/HDFS-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865317#comment-13865317
 ] 

Hudson commented on HDFS-5726:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #446 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/446/])
HDFS-5726. Fix compilation error in AbstractINodeDiff for JDK7. Contributed by 
Jing Zhao. (jing9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556433)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java


 Fix compilation error in AbstractINodeDiff for JDK7
 ---

 Key: HDFS-5726
 URL: https://issues.apache.org/jira/browse/HDFS-5726
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Fix For: 3.0.0

 Attachments: HDFS-5726.000.patch


 HDFS-5715 breaks JDK7 build for the following error:
 {code}
 [ERROR] 
 /home/kasha/code/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java:[134,53]
  error: snapshotId has private access in AbstractINodeDiff
 {code}
 This jira will fix the issue.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5715) Use Snapshot ID to indicate the corresponding Snapshot for a FileDiff/DirectoryDiff


[ 
https://issues.apache.org/jira/browse/HDFS-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865313#comment-13865313
 ] 

Hudson commented on HDFS-5715:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #446 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/446/])
HDFS-5715. Use Snapshot ID to indicate the corresponding Snapshot for a 
FileDiff/DirectoryDiff. Contributed by Jing Zhao. (jing9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556353)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiff.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
*

[jira] [Commented] (HDFS-5649) Unregister NFS and Mount service when NFS gateway is shutting down


[ 
https://issues.apache.org/jira/browse/HDFS-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865316#comment-13865316
 ] 

Hudson commented on HDFS-5649:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #446 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/446/])
HDFS-5649. Unregister NFS and Mount service when NFS gateway is shutting down. 
Contributed by Brandon Li (brandonli: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556405)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapRequest.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 Unregister NFS and Mount service when NFS gateway is shutting down
 --

 Key: HDFS-5649
 URL: https://issues.apache.org/jira/browse/HDFS-5649
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 3.0.0
Reporter: Brandon Li
Assignee: Brandon Li
 Fix For: 2.3.0

 Attachments: HDFS-5649.001.patch, HDFS-5649.002.patch


 The services should be unregistered if the gateway is asked to shutdown 
 gracefully.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5724) modifyCacheDirective logging audit log command wrongly as addCacheDirective


[ 
https://issues.apache.org/jira/browse/HDFS-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865314#comment-13865314
 ] 

Hudson commented on HDFS-5724:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #446 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/446/])
HDFS-5724. modifyCacheDirective logging audit log command wrongly as 
addCacheDirective (Uma Maheswara Rao G via Colin Patrick McCabe) (cmccabe: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556386)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


 modifyCacheDirective logging audit log command wrongly as addCacheDirective
 ---

 Key: HDFS-5724
 URL: https://issues.apache.org/jira/browse/HDFS-5724
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Minor
  Labels: caching
 Attachments: HDFS-5724.patch


 modifyCacheDirective:
 {code}
  if (isAuditEnabled()  isExternalInvocation()) {
 logAuditEvent(success, addCacheDirective, null, null, null);
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction

2014-01-08 Thread Vinay (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinay updated HDFS-5723:


Assignee: Vinay
  Status: Patch Available  (was: Open)

 Append failed FINALIZED replica should not be accepted as valid when that 
 block is underconstruction
 

 Key: HDFS-5723
 URL: https://issues.apache.org/jira/browse/HDFS-5723
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Vinay
Assignee: Vinay
 Attachments: HDFS-5723.patch


 Scenario:
 1. 3 node cluster with 
 dfs.client.block.write.replace-datanode-on-failure.enable set to false.
 2. One file is written with 3 replicas, blk_id_gs1
 3. One of the datanode DN1 is down.
 4. File was opened with append and some more data is added to the file and 
 synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2
 5. Now  DN1 restarted
 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should 
 be marked corrupted.
 but since NN having appended block state as UnderConstruction, at this time 
 its not detecting this block as corrupt and adding to valid block locations.
 As long as the namenode is alive, this datanode also will be considered as 
 valid replica and read/append will fail in that datanode.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-3752) BOOTSTRAPSTANDBY for new Standby node will not work just after saveNameSpace at ANN in case of BKJM

2014-01-08 Thread Rakesh R (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh R updated HDFS-3752:
---

Attachment: HDFS-3752-testcase.patch

 BOOTSTRAPSTANDBY for new Standby node will not work just after saveNameSpace 
 at ANN in case of BKJM
 ---

 Key: HDFS-3752
 URL: https://issues.apache.org/jira/browse/HDFS-3752
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha
Affects Versions: 2.0.0-alpha
Reporter: Vinay
 Attachments: HDFS-3752-testcase.patch


 1. do {{saveNameSpace}} in ANN node by entering into safemode
 2. in another new node, install standby NN and do BOOTSTRAPSTANDBY
 3. Now StandBy NN will not able to copy the fsimage_txid from ANN
 This is because, SNN not able to find the next txid (txid+1) in shared 
 storage.
 Just after {{saveNameSpace}} shared storage will have the new logsegment with 
 only START_LOG_SEGEMENT edits op.
 and BookKeeper will not be able to read last entry from inprogress ledger.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction

2014-01-08 Thread Vinay (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinay updated HDFS-5723:


Attachment: HDFS-5723.patch

Attached the patch, Please review

 Append failed FINALIZED replica should not be accepted as valid when that 
 block is underconstruction
 

 Key: HDFS-5723
 URL: https://issues.apache.org/jira/browse/HDFS-5723
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Vinay
 Attachments: HDFS-5723.patch


 Scenario:
 1. 3 node cluster with 
 dfs.client.block.write.replace-datanode-on-failure.enable set to false.
 2. One file is written with 3 replicas, blk_id_gs1
 3. One of the datanode DN1 is down.
 4. File was opened with append and some more data is added to the file and 
 synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2
 5. Now  DN1 restarted
 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should 
 be marked corrupted.
 but since NN having appended block state as UnderConstruction, at this time 
 its not detecting this block as corrupt and adding to valid block locations.
 As long as the namenode is alive, this datanode also will be considered as 
 valid replica and read/append will fail in that datanode.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-3752) BOOTSTRAPSTANDBY for new Standby node will not work just after saveNameSpace at ANN in case of BKJM

2014-01-08 Thread Rakesh R (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865367#comment-13865367
]

Rakesh R commented on HDFS-3752:

Hi, As I understood from the discussion, when bootstrapping the standby it is
not very much required to see the transactions present in the 'in_progress'
node and skipping of 'in_progress' will not cause any inconsistencies. Anyway
StandbyToActive transition will always ensure, the edit delta transactions will
be read from the shared edit dirs and able to reliably start as Active.

bq.we could add an easy workaround flag, like bootstrapStandby
-skipSharedEditsCheck, since the check here is just to help out the user and
not actually necessary for correct operation.
I also agree in skipping the shared edits check during bootstrapstandby, in
that case there is no special fix required for this JIRA.
Presently, there is no test cases for bootstrap with BKJM shared edits and I've
tried few. Could you please review the attached test case patch. If everyone
agrees, push this in and and can close this JIRA once HDFS-4120 is in. Any
thoughts?

BOOTSTRAPSTANDBY for new Standby node will not work just after saveNameSpace
at ANN in case of BKJM
---

Key: HDFS-3752
URL: https://issues.apache.org/jira/browse/HDFS-3752
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha
Affects Versions: 2.0.0-alpha
Reporter: Vinay
Attachments: HDFS-3752-testcase.patch

1. do {{saveNameSpace}} in ANN node by entering into safemode
2. in another new node, install standby NN and do BOOTSTRAPSTANDBY
3. Now StandBy NN will not able to copy the fsimage_txid from ANN
This is because, SNN not able to find the next txid (txid+1) in shared
storage.
Just after {{saveNameSpace}} shared storage will have the new logsegment with
only START_LOG_SEGEMENT edits op.
and BookKeeper will not be able to read last entry from inprogress ledger.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5715) Use Snapshot ID to indicate the corresponding Snapshot for a FileDiff/DirectoryDiff


[ 
https://issues.apache.org/jira/browse/HDFS-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865432#comment-13865432
 ] 

Hudson commented on HDFS-5715:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1638 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1638/])
HDFS-5715. Use Snapshot ID to indicate the corresponding Snapshot for a 
FileDiff/DirectoryDiff. Contributed by Jing Zhao. (jing9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556353)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiff.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
*

[jira] [Commented] (HDFS-5649) Unregister NFS and Mount service when NFS gateway is shutting down


[ 
https://issues.apache.org/jira/browse/HDFS-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865435#comment-13865435
 ] 

Hudson commented on HDFS-5649:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1638 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1638/])
HDFS-5649. Unregister NFS and Mount service when NFS gateway is shutting down. 
Contributed by Brandon Li (brandonli: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556405)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapRequest.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 Unregister NFS and Mount service when NFS gateway is shutting down
 --

 Key: HDFS-5649
 URL: https://issues.apache.org/jira/browse/HDFS-5649
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 3.0.0
Reporter: Brandon Li
Assignee: Brandon Li
 Fix For: 2.3.0

 Attachments: HDFS-5649.001.patch, HDFS-5649.002.patch


 The services should be unregistered if the gateway is asked to shutdown 
 gracefully.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5726) Fix compilation error in AbstractINodeDiff for JDK7


[ 
https://issues.apache.org/jira/browse/HDFS-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865436#comment-13865436
 ] 

Hudson commented on HDFS-5726:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1638 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1638/])
HDFS-5726. Fix compilation error in AbstractINodeDiff for JDK7. Contributed by 
Jing Zhao. (jing9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556433)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java


 Fix compilation error in AbstractINodeDiff for JDK7
 ---

 Key: HDFS-5726
 URL: https://issues.apache.org/jira/browse/HDFS-5726
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Fix For: 3.0.0

 Attachments: HDFS-5726.000.patch


 HDFS-5715 breaks JDK7 build for the following error:
 {code}
 [ERROR] 
 /home/kasha/code/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java:[134,53]
  error: snapshotId has private access in AbstractINodeDiff
 {code}
 This jira will fix the issue.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5727) introduce a self-maintaining io queue handling mechanism

2014-01-08 Thread Richard Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Chen updated HDFS-5727:
---

Summary: introduce a self-maintaining io queue handling mechanism  (was: 
introduce a self-maintain io queue handling mechanism)

 introduce a self-maintaining io queue handling mechanism
 

 Key: HDFS-5727
 URL: https://issues.apache.org/jira/browse/HDFS-5727
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: datanode
Affects Versions: 3.0.0
Reporter: Liang Xie
Assignee: Liang Xie

 Currently the datanode read/write SLA is difficult to be guaranteed for HBase 
 online requirement. One of major reasons is we don't support io priority or 
 io request reorder inside datanode.
 I propose introducing a self-maintain io queue mechanism to handle io request 
 priority. Imaging there're lots of concurrent read/write requests from HBase 
 side, and a background datanode block scanner is running(default is every 21 
 days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile 
 latency would be vulnerable despite we have a bg thread throttling...
 the reorder stuff i have not thought clearly enough, but definitely the 
 reorder in the queue in the app side would beat the currently relying OS's io 
 queue merge.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5727) introduce a self-maintain io queue handling mechanism

2014-01-08 Thread Richard Chen (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Richard Chen updated HDFS-5727:
---

Description:
Currently the datanode read/write SLA is difficult to be guaranteed for HBase
online requirement. One of major reasons is we don't support io priority or io
request reorder inside datanode.
I propose introducing a self-maintain io queue mechanism to handle io request
priority. Imaging there're lots of concurrent read/write requests from HBase
side, and a background datanode block scanner is running(default is every 21
days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile
latency would be vulnerable despite we have a bg thread throttling...
the reorder stuff i have not thought clearly enough, but definitely the reorder
in the queue in the app side would beat the currently relying OS's io queue
merge.

was:
Currently the datanode read/write SLA is dfficult to be ganranteed for HBase
online requirement. One of major reasons is we don't support io priority or io
reqeust reorder inside datanode.
I proposal introducing a self-maintain io queue mechanism to handle io request
priority. Image there're lots of concurrent read/write reqeust from HBase side,
and a background datanode block scanner is running(default is every 21 days,
IIRC) just in time, then the HBase read/write 99% or 99.9% percentile latency
would be vulnerable despite we have a bg thread throttling...
the reorder stuf i have not thought clearly enough, but definitely the reorder
in the queue in the app side would beat the currently relying OS's io queue
merge.

introduce a self-maintain io queue handling mechanism
-

Key: HDFS-5727
URL: https://issues.apache.org/jira/browse/HDFS-5727
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode
Affects Versions: 3.0.0
Reporter: Liang Xie
Assignee: Liang Xie

Currently the datanode read/write SLA is difficult to be guaranteed for HBase
online requirement. One of major reasons is we don't support io priority or
io request reorder inside datanode.
I propose introducing a self-maintain io queue mechanism to handle io request
priority. Imaging there're lots of concurrent read/write requests from HBase
side, and a background datanode block scanner is running(default is every 21
days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile
latency would be vulnerable despite we have a bg thread throttling...
the reorder stuff i have not thought clearly enough, but definitely the
reorder in the queue in the app side would beat the currently relying OS's io
queue merge.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5731) Refactoring to define interfaces between BM and NN and simplify the flow between them

Amir Langer created HDFS-5731:
-

 Summary: Refactoring to define interfaces between BM and NN and 
simplify the flow between them
 Key: HDFS-5731
 URL: https://issues.apache.org/jira/browse/HDFS-5731
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Amir Langer


Start the separation of BlockManager (BM) from NameNode (NN) by simplifying the 
flow between the two components and defining API interfaces between them. 
The two components still exist in the same VM and use the same memory space 
(using the same instances).
Logic to calls from Datanodes should be in the BM.
NN should interact with BM using few calls and BM should use the return types 
as much as possible to pass information to the NN.
APIs between them should be defined as interfaces so later it can be improved 
to not use the same object instances and turned into a real protocol.
This still assumes a one to one relation between NN and BM, same VM and does 
not handle lifecycle of the service.




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5731) Refactoring to define interfaces between BM and NN and simplify the flow between them


 [ 
https://issues.apache.org/jira/browse/HDFS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amir Langer updated HDFS-5731:
--

Description: 
Start the separation of BlockManager (BM) from NameNode (NN) by simplifying the 
flow between the two components and defining API interfaces between them. 
The two components still exist in the same VM and use the same memory space 
(using the same instances).
Logic to calls from Datanodes should be in the BM.
NN should interact with BM using few calls and BM should use the return types 
as much as possible to pass information to the NN.
APIs between them should be defined as interfaces so later it can be improved 
to not use the same object instances and turned into a real protocol.
This still assumes a one to one relation between NN and BM, same VM and does 
not handle lifecycle of the service.

This task should maintain backward compatibility


  was:
Start the separation of BlockManager (BM) from NameNode (NN) by simplifying the 
flow between the two components and defining API interfaces between them. 
The two components still exist in the same VM and use the same memory space 
(using the same instances).
Logic to calls from Datanodes should be in the BM.
NN should interact with BM using few calls and BM should use the return types 
as much as possible to pass information to the NN.
APIs between them should be defined as interfaces so later it can be improved 
to not use the same object instances and turned into a real protocol.
This still assumes a one to one relation between NN and BM, same VM and does 
not handle lifecycle of the service.



 Refactoring to define interfaces between BM and NN and simplify the flow 
 between them
 -

 Key: HDFS-5731
 URL: https://issues.apache.org/jira/browse/HDFS-5731
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Amir Langer

 Start the separation of BlockManager (BM) from NameNode (NN) by simplifying 
 the flow between the two components and defining API interfaces between them. 
 The two components still exist in the same VM and use the same memory space 
 (using the same instances).
 Logic to calls from Datanodes should be in the BM.
 NN should interact with BM using few calls and BM should use the return types 
 as much as possible to pass information to the NN.
 APIs between them should be defined as interfaces so later it can be improved 
 to not use the same object instances and turned into a real protocol.
 This still assumes a one to one relation between NN and BM, same VM and does 
 not handle lifecycle of the service.
 This task should maintain backward compatibility



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5732) Separate memory space between BM and NN

Amir Langer created HDFS-5732:
-

 Summary: Separate memory space between BM and NN
 Key: HDFS-5732
 URL: https://issues.apache.org/jira/browse/HDFS-5732
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Amir Langer


Change created APIs to not rely on the same instance being shared in both BM 
and NN. Use immutable objects / keep state in sync.
BM and NN will still exist in the same VM work on a new BM service as an 
independent process is deferred to later tasks.
Also, a one to one relation between BM and NN is assumed. 
This task should maintain backward compatibility.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5733) Separate concurrency control between BM and NN

Amir Langer created HDFS-5733:
-

 Summary: Separate concurrency control between BM and NN
 Key: HDFS-5733
 URL: https://issues.apache.org/jira/browse/HDFS-5733
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Amir Langer


Replace usage of the namesystem locking mechanism by the BM with its own 
concurrency control to control its own internal state.

Both NN and BM will still run from the same VM.
This task should maintain backward compatibility.

 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5727) introduce a self-maintaining io queue handling mechanism

2014-01-08 Thread Richard Chen (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865467#comment-13865467
]

Richard Chen commented on HDFS-5727:

interesting but if you can improve your language further, you will help the
audience to better understand what you intend to do. My team is working on
something similar to that. I am thinking of adding your problem into our design
scope. We can certainly collaborate on this. Let me know your thoughts.

introduce a self-maintaining io queue handling mechanism

Key: HDFS-5727
URL: https://issues.apache.org/jira/browse/HDFS-5727
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode
Affects Versions: 3.0.0
Reporter: Liang Xie
Assignee: Liang Xie

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5734) A NN-internal RPC BM service

Amir Langer created HDFS-5734:
-

 Summary: A NN-internal RPC BM service
 Key: HDFS-5734
 URL: https://issues.apache.org/jira/browse/HDFS-5734
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Amir Langer


Separate the BM from NN by running it with with its own thread-pool and RPC 
protocol but still in the same process as NN.
NN and BM will in interact through some loopback call that will simulate a 
separate service.
This sprint still assumes a one to one relation between NN and BM and does not 
split the BM to a separate process, only simulates such a split inside the same 
VM. This allows us to defer any configuration issue / Testing support / scripts 
changes to later tasks. 
This task will therefore also not handle any HA issue to the BM itself. It 
will, however, deal with having BM code actually running in a different thread 
to the NN code and will handle building the initialisation / lifecycle code to 
an independent BM.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5735) Testing support for BM as a service

Amir Langer created HDFS-5735:
-

 Summary: Testing support for BM as a service
 Key: HDFS-5735
 URL: https://issues.apache.org/jira/browse/HDFS-5735
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Amir Langer


Testing support for an independent BM service. Modify tests to start it / use 
MiniDFSCluster if they require a BM. Verify that all tests still pass with an 
independent BM (running off MiniDFSCluster).




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5736) BM service as a separate process

Amir Langer created HDFS-5736:
-

 Summary: BM service as a separate process
 Key: HDFS-5736
 URL: https://issues.apache.org/jira/browse/HDFS-5736
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Amir Langer


Add scripts / config. to allow running BM as a separate service.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5731) Refactoring to define interfaces between BM and NN and simplify the flow between them

[
https://issues.apache.org/jira/browse/HDFS-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Amir Langer updated HDFS-5731:
--

Attachment: 0001-Separation-of-BM-from-NN-Step1-introduce-APIs-as-int.patch

patch are changes done on top of trunk and were last rebased to start from
commit:

HADOOP-10175. Har files system authority should preserve userinfo. Contributed
by Chuan Liu.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1553169
13f79535-47bb-0310-9956-ffa450edef68

Refactoring to define interfaces between BM and NN and simplify the flow
between them
-

Key: HDFS-5731
URL: https://issues.apache.org/jira/browse/HDFS-5731
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Amir Langer
Attachments:
0001-Separation-of-BM-from-NN-Step1-introduce-APIs-as-int.patch

Start the separation of BlockManager (BM) from NameNode (NN) by simplifying
the flow between the two components and defining API interfaces between them.
The two components still exist in the same VM and use the same memory space
(using the same instances).
Logic to calls from Datanodes should be in the BM.
NN should interact with BM using few calls and BM should use the return types
as much as possible to pass information to the NN.
APIs between them should be defined as interfaces so later it can be improved
to not use the same object instances and turned into a real protocol.
This still assumes a one to one relation between NN and BM, same VM and does
not handle lifecycle of the service.
This task should maintain backward compatibility

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction

2014-01-08 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865487#comment-13865487
 ] 

Hadoop QA commented on HDFS-5723:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12621965/HDFS-5723.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/5845//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5845//console

This message is automatically generated.

 Append failed FINALIZED replica should not be accepted as valid when that 
 block is underconstruction
 

 Key: HDFS-5723
 URL: https://issues.apache.org/jira/browse/HDFS-5723
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.2.0
Reporter: Vinay
Assignee: Vinay
 Attachments: HDFS-5723.patch


 Scenario:
 1. 3 node cluster with 
 dfs.client.block.write.replace-datanode-on-failure.enable set to false.
 2. One file is written with 3 replicas, blk_id_gs1
 3. One of the datanode DN1 is down.
 4. File was opened with append and some more data is added to the file and 
 synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2
 5. Now  DN1 restarted
 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should 
 be marked corrupted.
 but since NN having appended block state as UnderConstruction, at this time 
 its not detecting this block as corrupt and adding to valid block locations.
 As long as the namenode is alive, this datanode also will be considered as 
 valid replica and read/append will fail in that datanode.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5724) modifyCacheDirective logging audit log command wrongly as addCacheDirective


[ 
https://issues.apache.org/jira/browse/HDFS-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865503#comment-13865503
 ] 

Hudson commented on HDFS-5724:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1663 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1663/])
HDFS-5724. modifyCacheDirective logging audit log command wrongly as 
addCacheDirective (Uma Maheswara Rao G via Colin Patrick McCabe) (cmccabe: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556386)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


 modifyCacheDirective logging audit log command wrongly as addCacheDirective
 ---

 Key: HDFS-5724
 URL: https://issues.apache.org/jira/browse/HDFS-5724
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Minor
  Labels: caching
 Attachments: HDFS-5724.patch


 modifyCacheDirective:
 {code}
  if (isAuditEnabled()  isExternalInvocation()) {
 logAuditEvent(success, addCacheDirective, null, null, null);
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5715) Use Snapshot ID to indicate the corresponding Snapshot for a FileDiff/DirectoryDiff


[ 
https://issues.apache.org/jira/browse/HDFS-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865502#comment-13865502
 ] 

Hudson commented on HDFS-5715:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1663 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1663/])
HDFS-5715. Use Snapshot ID to indicate the corresponding Snapshot for a 
FileDiff/DirectoryDiff. Contributed by Jing Zhao. (jing9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556353)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiff.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
*

[jira] [Commented] (HDFS-5726) Fix compilation error in AbstractINodeDiff for JDK7


[ 
https://issues.apache.org/jira/browse/HDFS-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865506#comment-13865506
 ] 

Hudson commented on HDFS-5726:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1663 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1663/])
HDFS-5726. Fix compilation error in AbstractINodeDiff for JDK7. Contributed by 
Jing Zhao. (jing9: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556433)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java


 Fix compilation error in AbstractINodeDiff for JDK7
 ---

 Key: HDFS-5726
 URL: https://issues.apache.org/jira/browse/HDFS-5726
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Fix For: 3.0.0

 Attachments: HDFS-5726.000.patch


 HDFS-5715 breaks JDK7 build for the following error:
 {code}
 [ERROR] 
 /home/kasha/code/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiff.java:[134,53]
  error: snapshotId has private access in AbstractINodeDiff
 {code}
 This jira will fix the issue.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5649) Unregister NFS and Mount service when NFS gateway is shutting down


[ 
https://issues.apache.org/jira/browse/HDFS-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865505#comment-13865505
 ] 

Hudson commented on HDFS-5649:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1663 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1663/])
HDFS-5649. Unregister NFS and Mount service when NFS gateway is shutting down. 
Contributed by Brandon Li (brandonli: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1556405)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapRequest.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 Unregister NFS and Mount service when NFS gateway is shutting down
 --

 Key: HDFS-5649
 URL: https://issues.apache.org/jira/browse/HDFS-5649
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: nfs
Affects Versions: 3.0.0
Reporter: Brandon Li
Assignee: Brandon Li
 Fix For: 2.3.0

 Attachments: HDFS-5649.001.patch, HDFS-5649.002.patch


 The services should be unregistered if the gateway is asked to shutdown 
 gracefully.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5721) sharedEditsImage in Namenode#initializeSharedEdits() should be closed before method returns

2014-01-08 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HDFS-5721:
-

Attachment: hdfs-5721-v3.txt

Patch v3 addresses Junping's comment.

 sharedEditsImage in Namenode#initializeSharedEdits() should be closed before 
 method returns
 ---

 Key: HDFS-5721
 URL: https://issues.apache.org/jira/browse/HDFS-5721
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Attachments: hdfs-5721-v1.txt, hdfs-5721-v2.txt, hdfs-5721-v3.txt


 At line 901:
 {code}
   FSImage sharedEditsImage = new FSImage(conf,
   Lists.URInewArrayList(),
   sharedEditsDirs);
 {code}
 sharedEditsImage is not closed before the method returns.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5734) A NN-internal RPC BM service

2014-01-08 Thread jay vyas (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865628#comment-13865628
 ] 

jay vyas commented on HDFS-5734:


Sorry to ask, but... whats BM ?  Is that the BackupNameNode?

 A NN-internal RPC BM service
 

 Key: HDFS-5734
 URL: https://issues.apache.org/jira/browse/HDFS-5734
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Amir Langer

 Separate the BM from NN by running it with with its own thread-pool and RPC 
 protocol but still in the same process as NN.
 NN and BM will in interact through some loopback call that will simulate a 
 separate service.
 This sprint still assumes a one to one relation between NN and BM and does 
 not split the BM to a separate process, only simulates such a split inside 
 the same VM. This allows us to defer any configuration issue / Testing 
 support / scripts changes to later tasks. 
 This task will therefore also not handle any HA issue to the BM itself. It 
 will, however, deal with having BM code actually running in a different 
 thread to the NN code and will handle building the initialisation / lifecycle 
 code to an independent BM.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5721) sharedEditsImage in Namenode#initializeSharedEdits() should be closed before method returns

2014-01-08 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865669#comment-13865669
 ] 

Hadoop QA commented on HDFS-5721:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12621987/hdfs-5721-v3.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/5846//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5846//console

This message is automatically generated.

 sharedEditsImage in Namenode#initializeSharedEdits() should be closed before 
 method returns
 ---

 Key: HDFS-5721
 URL: https://issues.apache.org/jira/browse/HDFS-5721
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Attachments: hdfs-5721-v1.txt, hdfs-5721-v2.txt, hdfs-5721-v3.txt


 At line 901:
 {code}
   FSImage sharedEditsImage = new FSImage(conf,
   Lists.URInewArrayList(),
   sharedEditsDirs);
 {code}
 sharedEditsImage is not closed before the method returns.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (HDFS-2261) AOP unit tests are not getting compiled or run

2014-01-08 Thread Karthik Kambatla (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla reassigned HDFS-2261:
--

Assignee: (was: Karthik Kambatla)

 AOP unit tests are not getting compiled or run 
 ---

 Key: HDFS-2261
 URL: https://issues.apache.org/jira/browse/HDFS-2261
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 2.0.0-alpha, 2.0.4-alpha
 Environment: 
 https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/834/console
 -compile-fault-inject ant target 
Reporter: Giridharan Kesavan
Priority: Minor
 Attachments: hdfs-2261.patch


 The tests in src/test/aop are not getting compiled or run.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5579) Under construction files make DataNode decommission take very long hours

2014-01-08 Thread Jing Zhao (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865705#comment-13865705
 ] 

Jing Zhao commented on HDFS-5579:
-

{code}
+if (bc.isUnderConstruction()) {
+  if (block.equals(bc.getLastBlock())  curReplicas  minReplication) 
{
+continue;
+  }
+  underReplicatedInOpenFiles++;
+}
{code}

Here if {{block}} is not the last block, and {{block}} is not under replicated, 
underReplicatedInOpenFiles will still increase?

 Under construction files make DataNode decommission take very long hours
 

 Key: HDFS-5579
 URL: https://issues.apache.org/jira/browse/HDFS-5579
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 1.2.0, 2.2.0
Reporter: zhaoyunjiong
Assignee: zhaoyunjiong
 Attachments: HDFS-5579-branch-1.2.patch, HDFS-5579.patch


 We noticed that some times decommission DataNodes takes very long time, even 
 exceeds 100 hours.
 After check the code, I found that in 
 BlockManager:computeReplicationWorkForBlocks(ListListBlock 
 blocksToReplicate) it won't replicate blocks which belongs to under 
 construction files, however in 
 BlockManager:isReplicationInProgress(DatanodeDescriptor srcNode), if there  
 is block need replicate no matter whether it belongs to under construction or 
 not, the decommission progress will continue running.
 That's the reason some time the decommission takes very long time.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5737) Replacing only the default ACL can fail to copy unspecified base entries from the access ACL.

Chris Nauroth created HDFS-5737:
---

 Summary: Replacing only the default ACL can fail to copy 
unspecified base entries from the access ACL.
 Key: HDFS-5737
 URL: https://issues.apache.org/jira/browse/HDFS-5737
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth


The final round of changes in HDFS-5673 switched to a search approach instead 
of a scan approach for finding base access entries that need to be copied to 
the default ACL.  However, in the case of doing full replacement on the default 
ACL, the list may not be sorted properly at this point in the code, causing the 
searches to miss the access entries.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Work started] (HDFS-5737) Replacing only the default ACL can fail to copy unspecified base entries from the access ACL.


 [ 
https://issues.apache.org/jira/browse/HDFS-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-5737 started by Chris Nauroth.

 Replacing only the default ACL can fail to copy unspecified base entries from 
 the access ACL.
 -

 Key: HDFS-5737
 URL: https://issues.apache.org/jira/browse/HDFS-5737
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth

 The final round of changes in HDFS-5673 switched to a search approach instead 
 of a scan approach for finding base access entries that need to be copied to 
 the default ACL.  However, in the case of doing full replacement on the 
 default ACL, the list may not be sorted properly at this point in the code, 
 causing the searches to miss the access entries.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5737) Replacing only the default ACL can fail to copy unspecified base entries from the access ACL.


 [ 
https://issues.apache.org/jira/browse/HDFS-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-5737:


Attachment: (was: HDFS-5673.1.patch)

 Replacing only the default ACL can fail to copy unspecified base entries from 
 the access ACL.
 -

 Key: HDFS-5737
 URL: https://issues.apache.org/jira/browse/HDFS-5737
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HDFS-5737.1.patch


 The final round of changes in HDFS-5673 switched to a search approach instead 
 of a scan approach for finding base access entries that need to be copied to 
 the default ACL.  However, in the case of doing full replacement on the 
 default ACL, the list may not be sorted properly at this point in the code, 
 causing the searches to miss the access entries.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5737) Replacing only the default ACL can fail to copy unspecified base entries from the access ACL.


 [ 
https://issues.apache.org/jira/browse/HDFS-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-5737:


Attachment: HDFS-5737.1.patch

 Replacing only the default ACL can fail to copy unspecified base entries from 
 the access ACL.
 -

 Key: HDFS-5737
 URL: https://issues.apache.org/jira/browse/HDFS-5737
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HDFS-5737.1.patch


 The final round of changes in HDFS-5673 switched to a search approach instead 
 of a scan approach for finding base access entries that need to be copied to 
 the default ACL.  However, in the case of doing full replacement on the 
 default ACL, the list may not be sorted properly at this point in the code, 
 causing the searches to miss the access entries.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5737) Replacing only the default ACL can fail to copy unspecified base entries from the access ACL.