[jira] [Created] (HDFS-9298) remove replica and not add replica with wrong genStamp

2015-10-23 Thread Chang Li (JIRA) 
Chang Li created HDFS-9298:
--

 Summary: remove replica and not add replica with wrong genStamp
 Key: HDFS-9298
 URL: https://issues.apache.org/jira/browse/HDFS-9298
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li


currently, in setGenerationStampAndVerifyReplicas, replica with wrong gen stamp 
is not really removed, only StorageLocation of that replica is removed. 
Moreover, we should check genStamp before addReplicaIfNotPresent



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9294) DFSClient deadlock when close file and failed to renew lease

2015-10-23 Thread JIRA 
邓飞 created HDFS-9294:


 Summary: DFSClient  deadlock when close file and failed to renew 
lease
 Key: HDFS-9294
 URL: https://issues.apache.org/jira/browse/HDFS-9294
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: HDFS, hdfs-client
Affects Versions: 2.7.1, 2.2.0
 Environment: Hadoop 2.2.0
Reporter: 邓飞


We found a deadlock at our HBase(0.98) cluster(and the Hadoop Version is 
2.2.0),and it should be HDFS BUG,at the time our network is not stable.
 below is the stack:

*
Found one Java-level deadlock:
=
"MemStoreFlusher.1":
  waiting to lock monitor 0x7ff27cfa5218 (object 0x0002fae5ebe0, a 
org.apache.hadoop.hdfs.LeaseRenewer),
  which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
  waiting to lock monitor 0x7ff2e67e16a8 (object 0x000486ce6620, a 
org.apache.hadoop.hdfs.DFSOutputStream),
  which is held by "MemStoreFlusher.0"
"MemStoreFlusher.0":
  waiting to lock monitor 0x7ff27cfa5218 (object 0x0002fae5ebe0, a 
org.apache.hadoop.hdfs.LeaseRenewer),
  which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"

Java stack information for the threads listed above:
===
"MemStoreFlusher.1":
at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
- waiting to lock <0x0002fae5ebe0> (a 
org.apache.hadoop.hdfs.LeaseRenewer)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
at 
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
- locked <0x00055b606cb0> (a org.apache.hadoop.hdfs.DFSOutputStream)
at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
at 
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)
at 
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
at 
org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)
at 
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
- locked <0x00059869eed8> (a java.lang.Object)
at 
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
at 
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)
at 
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)
at 
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)
at 
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:211)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:66)
at 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:238)
at java.lang.Thread.run(Thread.java:744)
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
at 
org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:1822)
- waiting to lock <0x000486ce6620> (a 
org.apache.hadoop.hdfs.DFSOutputStream)
at 
org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:780)
at org.apache.hadoop.hdfs.DFSClient.abort(DFSClient.java:753)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:453)
- locked <0x0002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:744)
"MemStoreFlusher.0":
at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
- waiting to lock <0x0002fae5ebe0> (a 
org.apache.hadoop.hdfs.LeaseRenewer)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
at

[jira] [Created] (HDFS-9296) ShellBasedUnixGroupMapping should support group names with space

2015-10-23 Thread Wei-Chiu Chuang (JIRA) 
Wei-Chiu Chuang created HDFS-9296:
-

 Summary: ShellBasedUnixGroupMapping should support group names 
with space
 Key: HDFS-9296
 URL: https://issues.apache.org/jira/browse/HDFS-9296
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


In a typical configuration, group name is obtained from AD through SSSD/LDAP. 
AD permits group names with space (e.g. "Domain Users").

Unfortunately, the present implementation of ShellBasedUnixGroupMapping parses 
the output of shell command "id -Gn", and assumes group names are separated by 
space. 

This could be achieved by using a combination of shell scripts, for example, 

bash -c 'id -G weichiu | tr " " "\n" | xargs -I % getent group "%" | cut -d":" 
-f1'

But I am still looking for a more compact form, and potentially more efficient 
one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9296) ShellBasedUnixGroupMapping should support group names with space

2015-10-23 Thread Wei-Chiu Chuang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-9296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-9296.
---
Resolution: Duplicate

I filed in the wrong category. A new one is filed as HADOOP-12505

> ShellBasedUnixGroupMapping should support group names with space
> 
>
> Key: HDFS-9296
> URL: https://issues.apache.org/jira/browse/HDFS-9296
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> In a typical configuration, group name is obtained from AD through SSSD/LDAP. 
> AD permits group names with space (e.g. "Domain Users").
> Unfortunately, the present implementation of ShellBasedUnixGroupMapping 
> parses the output of shell command "id -Gn", and assumes group names are 
> separated by space. 
> This could be achieved by using a combination of shell scripts, for example, 
> bash -c 'id -G weichiu | tr " " "\n" | xargs -I % getent group "%" | cut 
> -d":" -f1'
> But I am still looking for a more compact form, and potentially more 
> efficient one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9295) Add a thorough test of the full KMS code path

2015-10-23 Thread Daniel Templeton (JIRA) 
Daniel Templeton created HDFS-9295:
--

 Summary: Add a thorough test of the full KMS code path
 Key: HDFS-9295
 URL: https://issues.apache.org/jira/browse/HDFS-9295
 Project: Hadoop HDFS
  Issue Type: Test
  Components: security, test
Affects Versions: 2.6.1
Reporter: Daniel Templeton
Assignee: Daniel Templeton
Priority: Critical


TestKMS does a good job of testing the ACLs directly, but they are tested out 
of context.  Additional tests are needed that test how the ACL impact key 
creation, EZ creation, file creation in an EZ, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9293) FSEditLog's 'OpInstanceCache' instance of threadLocal cache exists dirty 'rpcId',which may cause standby NN too busy to communicate

2015-10-23 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/HDFS-9293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

邓飞 resolved HDFS-9293.
--
   Resolution: Fixed
Fix Version/s: 2.7.1

> FSEditLog's  'OpInstanceCache' instance of threadLocal cache exists dirty 
> 'rpcId',which may cause standby NN too busy  to communicate 
> --
>
> Key: HDFS-9293
> URL: https://issues.apache.org/jira/browse/HDFS-9293
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.2.0, 2.7.1
>Reporter: 邓飞
>Assignee: 邓飞
> Fix For: 2.7.1
>
>
>   In our cluster (hadoop 2.2.0-HA,700+ DN),we found standby NN tail editlog 
> slowly,and hold the fsnamesystem writelock during the work and the DN's 
> heartbeart/blockreport IPC request blocked.Lead to Active NN remove stale DN 
> which can't send heartbeat  because blocking at process Standby NN Regiest 
> common(FIXED at 2.7.1).
>   Below is the standby NN  stack:
> "Edit log tailer" prio=10 tid=0x7f28fcf35800 nid=0x1a7d runnable 
> [0x7f0dd1d76000]
>java.lang.Thread.State: RUNNABLE
>   at java.util.PriorityQueue.remove(PriorityQueue.java:360)
>   at 
> org.apache.hadoop.util.LightWeightCache.put(LightWeightCache.java:217)
>   at org.apache.hadoop.ipc.RetryCache.addCacheEntry(RetryCache.java:270)
>   - locked <0x7f12817714b8> (a org.apache.hadoop.ipc.RetryCache)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntry(FSNamesystem.java:724)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:406)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:199)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:112)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:733)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
>   at 
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
>
> When apply editLogOp,if the IPC retryCache is found,need  to remove the 
> previous from priorityQueue(O(N)), The updateblock is don't  need record 
> rpcId on editlog except  'client request updatePipeline',but we found many 
> 'UpdateBlocksOp' has repeat ipcId.
>  
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9297) Update TestBlockMissingException to use corruptBlockOnDataNodesByDeletingBlockFile()

2015-10-23 Thread Tony Wu (JIRA) 
Tony Wu created HDFS-9297:
-

 Summary: Update TestBlockMissingException to use 
corruptBlockOnDataNodesByDeletingBlockFile()
 Key: HDFS-9297
 URL: https://issues.apache.org/jira/browse/HDFS-9297
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: HDFS, test
Affects Versions: 2.7.1
Reporter: Tony Wu
Assignee: Tony Wu
Priority: Trivial


TestBlockMissingException uses its own function to corrupt a block by deleting 
all its block files. HDFS-7235 introduced a helper function 
{{corruptBlockOnDataNodesByDeletingBlockFile()}} that does exactly the same 
thing. We can update this test to use the helper function.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9300) DirectoryScanner.testThrottle() is still a little flakey

2015-10-23 Thread Daniel Templeton (JIRA) 
Daniel Templeton created HDFS-9300:
--

 Summary: DirectoryScanner.testThrottle() is still a little flakey
 Key: HDFS-9300
 URL: https://issues.apache.org/jira/browse/HDFS-9300
 Project: Hadoop HDFS
  Issue Type: Test
  Components: balancer & mover, test
Affects Versions: 2.7.1
Reporter: Daniel Templeton
Assignee: Daniel Templeton


It failed in:

https://builds.apache.org/job/PreCommit-HDFS-Build/13160/testReport/org.apache.hadoop.hdfs.server.datanode/TestDirectoryScanner/testThrottling/

by narrowly missing the performance boundaries.  The only solution I have is to 
relax the boundaries a little.  The throttle's just a hard thing to test in an 
unpredictable environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Hadoop-Hdfs-trunk #2466

2015-10-23 Thread Apache Jenkins Server 
See 

Changes:

[ozawa] HADOOP-7266. Deprecate metrics v1. Contributed by Akira AJISAKA.

[jeagles] YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev

[mingma] YARN-2913. Fair scheduler should have ability to set MaxResourceDefault

--
[...truncated 6686 lines...]
Running 
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.095 sec - in 
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
Running 
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.202 sec - in 
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
Running org.apache.hadoop.hdfs.tools.TestDFSHAAdmin
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.33 sec - in 
org.apache.hadoop.hdfs.tools.TestDFSHAAdmin
Running org.apache.hadoop.hdfs.tools.TestGetConf
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.52 sec - in 
org.apache.hadoop.hdfs.tools.TestGetConf
Running org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.729 sec - in 
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.257 sec - in 
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Running org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.593 sec - in 
org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
Running org.apache.hadoop.hdfs.tools.TestGetGroups
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.722 sec - in 
org.apache.hadoop.hdfs.tools.TestGetGroups
Running org.apache.hadoop.hdfs.tools.TestDebugAdmin
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.797 sec - in 
org.apache.hadoop.hdfs.tools.TestDebugAdmin
Running org.apache.hadoop.hdfs.tools.TestStoragePolicyCommands
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.14 sec - in 
org.apache.hadoop.hdfs.tools.TestStoragePolicyCommands
Running org.apache.hadoop.hdfs.tools.TestDFSAdmin
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.228 sec - in 
org.apache.hadoop.hdfs.tools.TestDFSAdmin
Running org.apache.hadoop.hdfs.tools.TestDelegationTokenFetcher
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.52 sec - in 
org.apache.hadoop.hdfs.tools.TestDelegationTokenFetcher
Running org.apache.hadoop.hdfs.TestBlockStoragePolicy
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.25 sec - in 
org.apache.hadoop.hdfs.TestBlockStoragePolicy
Running org.apache.hadoop.hdfs.TestCrcCorruption
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.378 sec - in 
org.apache.hadoop.hdfs.TestCrcCorruption
Running org.apache.hadoop.hdfs.TestDFSRename
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.232 sec - in 
org.apache.hadoop.hdfs.TestDFSRename
Running org.apache.hadoop.hdfs.TestLargeBlock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.929 sec - in 
org.apache.hadoop.hdfs.TestLargeBlock
Running org.apache.hadoop.hdfs.TestDatanodeConfig
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.823 sec - in 
org.apache.hadoop.hdfs.TestDatanodeConfig
Running org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.909 sec - in 
org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
Running org.apache.hadoop.hdfs.TestFileAppend2
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.502 sec - in 
org.apache.hadoop.hdfs.TestFileAppend2
Running org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.767 sec - in 
org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Running org.apache.hadoop.hdfs.TestSetrepIncreasing
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.411 sec - in 
org.apache.hadoop.hdfs.TestSetrepIncreasing
Running org.apache.hadoop.hdfs.TestDFSClientRetries
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.452 sec - 
in org.apache.hadoop.hdfs.TestDFSClientRetries
Running org.apache.hadoop.hdfs.TestBlockReaderLocal
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.735 sec - 
in org.apache.hadoop.hdfs.TestBlockReaderLocal
Running org.apache.hadoop.hdfs.TestHdfsAdmin
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.912 sec - in 
org.apache.hadoop.hdfs.TestHdfsAdmin
Running org.apache.hadoop.hdfs.TestDataTransferKeepalive
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time

Hadoop-Hdfs-trunk - Build # 2466 - Still Failing

2015-10-23 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2466/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 6879 lines...]
main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Skipping javadoc generation
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS Client . SUCCESS [04:22 min]
[INFO] Apache Hadoop HDFS  FAILURE [  03:20 h]
[INFO] Apache Hadoop HDFS Native Client .. SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  0.066 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 03:24 h
[INFO] Finished at: 2015-10-23T19:45:09+00:00
[INFO] Final Memory: 55M/696M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating YARN-4009
Updating YARN-2913
Updating HADOOP-7266
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.testNodeCount

Error Message:
Timeout: excess replica count not equal to 2 for block blk_1073741825_1001 
after 2 msec.  Last counts: live = 2, excess = 0, corrupt = 0

Stack Trace:
java.util.concurrent.TimeoutException: Timeout: excess replica count not equal 
to 2 for block blk_1073741825_1001 after 2 msec.  Last counts: live = 2, 
excess = 0, corrupt = 0
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.checkTimeout(TestNodeCount.java:152)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.checkTimeout(TestNodeCount.java:146)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount.testNodeCount(TestNodeCount.java:130)

Hadoop-Hdfs-trunk - Build # 2467 - Still Failing

2015-10-23 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2467/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 7057 lines...]
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Skipping javadoc generation
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS Client . SUCCESS [03:52 min]
[INFO] Apache Hadoop HDFS  FAILURE [  03:57 h]
[INFO] Apache Hadoop HDFS Native Client .. SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  0.067 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 04:01 h
[INFO] Finished at: 2015-10-24T01:26:53+00:00
[INFO] Final Memory: 54M/673M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-hdfs: ExecutionException: java.lang.RuntimeException: The forked 
VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs
 && /home/jenkins/tools/java/jdk1.7.0_55/jre/bin/java -Xmx2048m 
-XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefirebooter1020770693056756747.jar
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefire2287146431550555631tmp
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire/surefire_5462365585445472498401tmp
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating HDFS-8808
Updating YARN-4041
Updating HDFS-9264
Updating HDFS-9184
Updating MAPREDUCE-6508
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
8 tests failed.
FAILED:  
org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryAfterNameNodeRestart2

Error Message:
org.apache.hadoop.util.ExitUtil$ExitException: Could not sync enough journals 
to persistent storage due to No journals available to flush. Unsynced 
transactions: 1
 at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
 at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:637)
 at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1316)
 at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:362)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1214)
 at 
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1740)
 at

Re: hadoop-hdfs-client splitoff is going to break code

2015-10-23 Thread Kihwal Lee 
I think a lot of "client-side" tests use MiniDFSCluster. I know mechanical 
division is possible, but what about test coverage?
Kihwal

  From: Haohui Mai 
 To: hdfs-dev@hadoop.apache.org; Kihwal Lee  
Cc: "common-...@hadoop.apache.org"  
 Sent: Friday, October 23, 2015 4:43 PM
 Subject: Re: hadoop-hdfs-client splitoff is going to break code
   
All tests that need to spin up a MiniDFSCluster will need to stay in
hadoop-hdfs. Other client only tests are being moved to the
hadoop-hdfs-client module, which is tracked in HDFS-9168.

~Haohui



On Fri, Oct 23, 2015 at 2:14 PM, Kihwal Lee
 wrote:
> I am not sure whether it was mentioned by anyone before, butI noticed that 
> client only changes do not trigger running anytest in hdfs-precommit. This is 
> because hadoop-hdfs-client does nothave any test.
> Kihwal
>
>      From: Colin P. McCabe 
>  To: "hdfs-dev@hadoop.apache.org" 
> Cc: "common-...@hadoop.apache.org" 
>  Sent: Monday, October 19, 2015 4:01 PM
>  Subject: Re: hadoop-hdfs-client splitoff is going to break code
>
> Thanks for being proactive here, Steve.  I think this is a good example of
> why this change should have been done in a branch rather than having been
> done directly in trunk.
>
> regards,
> Colin
>
>
>
>
> On Wed, Oct 14, 2015 at 10:36 AM, Steve Loughran 
> wrote:
>
>> just an FYI, the split off of hadoop hdfs into client and server is going
>> to break things.
>>
>> I know that, as my code is broken; DFSConfigKeys off the path,
>> HdfsConfiguration, the class I've been loading to force pickup of
>> hdfs-site.xml -all missing.
>>
>> This is because hadoop-client  POM now depends on hadoop-hdfs-client, not
>> hadoop-hdfs, so the things I'm referencing are gone. I'm particularly sad
>> about DfsConfigKeys, as everybody uses it as the one hard-coded resource of
>> HDFS constants, HDFS-6566 covering the issue of making this public,
>> something that's been sitting around for a year.
>>
>> I'm fixing my build by explicitly adding a hadoop-hdfs dependency.
>>
>> Any application which used stuff which has now been declared server-side
>> isn't going to compile any more, which does appear to break the
>> compatibility guidelines we've adopted, specifically "The hadoop-client
>> artifact (maven groupId:artifactId) stays compatible within a major release"
>>
>>
>> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html#Build_artifacts
>>
>>
>> We need to do one of
>>
>> 1. agree that this change, is considered acceptable according to policy,
>> and mark it as incompatible in hdfs/CHANGES.TXT
>> 2. Change the POMs to add both hdfs-client and -hdfs server in
>> hadoop-client -with downstream users free to exclude the server code
>>
>> We unintentionally caused similar grief with the move of the s3n clients
>> to hadoop-aws , HADOOP-11074 -something we should have picked up and -1'd.
>> This time we know the problems going to arise, so lets explicitly make a
>> decision this time, and share it with our users.
>>
>> -steve
>>
>
>
>

[jira] [Created] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

2015-10-23 Thread Karthik Palaniappan (JIRA) 
Karthik Palaniappan created HDFS-9302:
-

 Summary: WebHDFS throws NullPointerException if newLength is not 
provided
 Key: HDFS-9302
 URL: https://issues.apache.org/jira/browse/HDFS-9302
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: HDFS
Affects Versions: 2.7.1
 Environment: Centos6
Reporter: Karthik Palaniappan
Priority: Minor


$ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
{"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}

We should change newLength to be a required parameter in the webhdfs 
documentation 
(https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
 and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #531

2015-10-23 Thread Apache Jenkins Server 
See 

Changes:

[junping_du] MAPREDUCE-6508. TestNetworkedJob fails consistently due to 
delegation

[jitendra] HDFS-9184. Logging HDFS operation's caller context into audit logs.

[lei] HDFS-9264. Minor cleanup of operations on FsVolumeList#volumes.  (Walter

[jlowe] YARN-4041. Slow delegation token renewal can severely prolong RM

[zhz] HDFS-8808. dfs.image.transfer.bandwidthPerSec should not apply to

--
[...truncated 7405 lines...]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:894)
at 
org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery(TestLeaseRecovery2.java:276)
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at 
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1105)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1000)

org.apache.hadoop.hdfs.TestLeaseRecovery2  Time elapsed: 0.02 sec  <<< FAILURE!
java.lang.AssertionError: Test resulted in an unexpected exit
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1895)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1882)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1875)
at 
org.apache.hadoop.hdfs.TestLeaseRecovery2.tearDown(TestLeaseRecovery2.java:105)

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.TestLayoutVersion
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.257 sec - in 
org.apache.hadoop.hdfs.protocol.TestLayoutVersion
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.datatransfer.TestPacketReceiver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.448 sec - in 
org.apache.hadoop.hdfs.protocol.datatransfer.TestPacketReceiver
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.64 sec - in 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.TestAnnotations
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.134 sec - in 
org.apache.hadoop.hdfs.protocol.TestAnnotations
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.TestLocatedBlock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.261 sec - in 
org.apache.hadoop.hdfs.protocol.TestLocatedBlock
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.015 sec - in 
org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.protocol.TestExtendedBlock
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec - in 
org.apache.hadoop.hdfs.protocol.TestExtendedBlock
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestAbandonBlock
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.956 sec - in 
org.apache.hadoop.hdfs.TestAbandonBlock
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.822 sec - 
in org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestFileAppendRestart
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.952 sec - in 
org.apache.hadoop.hdfs.TestFileAppendRestart
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestFetchImage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.121 sec - in 
org.apache.hadoop.hdfs.TestFetchImage
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestSafeModeWithStripedFile
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.198 sec - in

Hadoop-Hdfs-trunk-Java8 - Build # 531 - Still Failing

2015-10-23 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/531/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 7598 lines...]
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS Client . SUCCESS [03:31 min]
[INFO] Apache Hadoop HDFS  FAILURE [  03:16 h]
[INFO] Apache Hadoop HDFS Native Client .. SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  0.493 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 03:20 h
[INFO] Finished at: 2015-10-24T00:37:03+00:00
[INFO] Final Memory: 55M/433M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating HDFS-8808
Updating YARN-4041
Updating HDFS-9264
Updating HDFS-9184
Updating MAPREDUCE-6508
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
6 tests failed.
FAILED:  
org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithNoXmlDefaults

Error Message:
null

Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at 
org.apache.hadoop.hdfs.TestDistributedFileSystem.testDFSClient(TestDistributedFileSystem.java:418)
at 
org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithNoXmlDefaults(TestDistributedFileSystem.java:812)


FAILED:  
org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart

Error Message:
org.apache.hadoop.util.ExitUtil$ExitException: Could not sync enough journals 
to persistent storage due to No journals available to flush. Unsynced 
transactions: 1
 at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
 at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:637)
 at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1316)
 at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:362)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1214)
 at 
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1740)
 at

Re: hadoop-hdfs-client splitoff is going to break code

2015-10-23 Thread Haohui Mai 
All tests that need to spin up a MiniDFSCluster will need to stay in
hadoop-hdfs. Other client only tests are being moved to the
hadoop-hdfs-client module, which is tracked in HDFS-9168.

~Haohui

On Fri, Oct 23, 2015 at 2:14 PM, Kihwal Lee
 wrote:
> I am not sure whether it was mentioned by anyone before, butI noticed that 
> client only changes do not trigger running anytest in hdfs-precommit. This is 
> because hadoop-hdfs-client does nothave any test.
> Kihwal
>
>   From: Colin P. McCabe 
>  To: "hdfs-dev@hadoop.apache.org" 
> Cc: "common-...@hadoop.apache.org" 
>  Sent: Monday, October 19, 2015 4:01 PM
>  Subject: Re: hadoop-hdfs-client splitoff is going to break code
>
> Thanks for being proactive here, Steve.  I think this is a good example of
> why this change should have been done in a branch rather than having been
> done directly in trunk.
>
> regards,
> Colin
>
>
>
>
> On Wed, Oct 14, 2015 at 10:36 AM, Steve Loughran 
> wrote:
>
>> just an FYI, the split off of hadoop hdfs into client and server is going
>> to break things.
>>
>> I know that, as my code is broken; DFSConfigKeys off the path,
>> HdfsConfiguration, the class I've been loading to force pickup of
>> hdfs-site.xml -all missing.
>>
>> This is because hadoop-client  POM now depends on hadoop-hdfs-client, not
>> hadoop-hdfs, so the things I'm referencing are gone. I'm particularly sad
>> about DfsConfigKeys, as everybody uses it as the one hard-coded resource of
>> HDFS constants, HDFS-6566 covering the issue of making this public,
>> something that's been sitting around for a year.
>>
>> I'm fixing my build by explicitly adding a hadoop-hdfs dependency.
>>
>> Any application which used stuff which has now been declared server-side
>> isn't going to compile any more, which does appear to break the
>> compatibility guidelines we've adopted, specifically "The hadoop-client
>> artifact (maven groupId:artifactId) stays compatible within a major release"
>>
>>
>> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html#Build_artifacts
>>
>>
>> We need to do one of
>>
>> 1. agree that this change, is considered acceptable according to policy,
>> and mark it as incompatible in hdfs/CHANGES.TXT
>> 2. Change the POMs to add both hdfs-client and -hdfs server in
>> hadoop-client -with downstream users free to exclude the server code
>>
>> We unintentionally caused similar grief with the move of the s3n clients
>> to hadoop-aws , HADOOP-11074 -something we should have picked up and -1'd.
>> This time we know the problems going to arise, so lets explicitly make a
>> decision this time, and share it with our users.
>>
>> -steve
>>
>
>
>

Re: hadoop-hdfs-client splitoff is going to break code

2015-10-23 Thread Kihwal Lee 
I am not sure whether it was mentioned by anyone before, butI noticed that 
client only changes do not trigger running anytest in hdfs-precommit. This is 
because hadoop-hdfs-client does nothave any test.
Kihwal

  From: Colin P. McCabe 
 To: "hdfs-dev@hadoop.apache.org"  
Cc: "common-...@hadoop.apache.org"  
 Sent: Monday, October 19, 2015 4:01 PM
 Subject: Re: hadoop-hdfs-client splitoff is going to break code
   
Thanks for being proactive here, Steve.  I think this is a good example of
why this change should have been done in a branch rather than having been
done directly in trunk.

regards,
Colin




On Wed, Oct 14, 2015 at 10:36 AM, Steve Loughran 
wrote:

> just an FYI, the split off of hadoop hdfs into client and server is going
> to break things.
>
> I know that, as my code is broken; DFSConfigKeys off the path,
> HdfsConfiguration, the class I've been loading to force pickup of
> hdfs-site.xml -all missing.
>
> This is because hadoop-client  POM now depends on hadoop-hdfs-client, not
> hadoop-hdfs, so the things I'm referencing are gone. I'm particularly sad
> about DfsConfigKeys, as everybody uses it as the one hard-coded resource of
> HDFS constants, HDFS-6566 covering the issue of making this public,
> something that's been sitting around for a year.
>
> I'm fixing my build by explicitly adding a hadoop-hdfs dependency.
>
> Any application which used stuff which has now been declared server-side
> isn't going to compile any more, which does appear to break the
> compatibility guidelines we've adopted, specifically "The hadoop-client
> artifact (maven groupId:artifactId) stays compatible within a major release"
>
>
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html#Build_artifacts
>
>
> We need to do one of
>
> 1. agree that this change, is considered acceptable according to policy,
> and mark it as incompatible in hdfs/CHANGES.TXT
> 2. Change the POMs to add both hdfs-client and -hdfs server in
> hadoop-client -with downstream users free to exclude the server code
>
> We unintentionally caused similar grief with the move of the s3n clients
> to hadoop-aws , HADOOP-11074 -something we should have picked up and -1'd.
> This time we know the problems going to arise, so lets explicitly make a
> decision this time, and share it with our users.
>
> -steve
>

[jira] [Resolved] (HDFS-9287) Block placement completely fails if too many nodes are decommissioning

2015-10-23 Thread Kuhu Shukla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kuhu Shukla resolved HDFS-9287.
---
   Resolution: Duplicate
Fix Version/s: 2.8.0

HDFS-7725 fixes this issue. Verified through a unit test.

> Block placement completely fails if too many nodes are decommissioning
> --
>
> Key: HDFS-9287
> URL: https://issues.apache.org/jira/browse/HDFS-9287
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.6.0
>Reporter: Daryn Sharp
>Assignee: Kuhu Shukla
>Priority: Critical
> Fix For: 2.8.0
>
>
> The DatanodeManager coordinates with the HeartbeatManager to update 
> HeartbeatManager.Stats to track capacity and load.   This is crucial for 
> block placement to consider space and load.  It's completely broken for 
> decomm nodes.
> The heartbeat manager substracts the prior values before it adds new values.  
> During registration of a decomm node, it substracts before seeding the 
> initial values.  This decrements nodesInService, flips state to decomm, add 
> will not increment nodesInService (correct).  There are other math bugs 
> (double adding) that accidentally work due to 0 values.
> The result is every decomm node decrements the node count used for block 
> placement.  When enough nodes are decomm, the replication monitor will 
> silently stop working.  No logging.  It searches all nodes and just gives up. 
>  Eventually, all block allocation will also completely fail.  No files can be 
> created.  No jobs can be submitted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9293) FSEditLog's 'OpInstanceCache' instance of threadLocal cache exists dirty 'rpcId',which may cause standby NN too busy to communicate

2015-10-23 Thread JIRA 
邓飞 created HDFS-9293:


 Summary: FSEditLog's  'OpInstanceCache' instance of threadLocal 
cache exists dirty 'rpcId',which may cause standby NN too busy  to communicate 
 Key: HDFS-9293
 URL: https://issues.apache.org/jira/browse/HDFS-9293
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.7.1, 2.2.0
Reporter: 邓飞
Assignee: 邓飞






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)