date:20151125

[jira] [Commented] (HDFS-9426) Rollingupgrade finalization is not backward compatible

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028314#comment-15028314
 ] 

Hudson commented on HDFS-9426:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #1458 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1458/])
HDFS-9426. Rollingupgrade finalization is not backward compatible 
(vinayakumarb: rev 9f256d1d716a7e17606245fcfc619901a8fa299a)
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java


> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in 
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade 
> because it's not null.
> NN might have to check the DN version and return different 
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but 
> unfortunately did not address the semantics of the heartbeat response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9294) DFSClient deadlock when close file and failed to renew lease

2015-11-25 Thread JIRA


[ 
https://issues.apache.org/jira/browse/HDFS-9294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028304#comment-15028304
 ] 

邓飞 commented on HDFS-9294:
--

dfsClient.endFileLease seem idempotent,and not have side-effect.

> DFSClient  deadlock when close file and failed to renew lease
> -
>
> Key: HDFS-9294
> URL: https://issues.apache.org/jira/browse/HDFS-9294
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Affects Versions: 2.2.0, 2.7.1
> Environment: Hadoop 2.2.0
>Reporter: 邓飞
>Assignee: 邓飞
>
> We found a deadlock at our HBase(0.98) cluster(and the Hadoop Version is 
> 2.2.0),and it should be HDFS BUG,at the time our network is not stable.
>  below is the stack:
> *
> Found one Java-level deadlock:
> =
> "MemStoreFlusher.1":
>   waiting to lock monitor 0x7ff27cfa5218 (object 0x0002fae5ebe0, a 
> org.apache.hadoop.hdfs.LeaseRenewer),
>   which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"
> "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
>   waiting to lock monitor 0x7ff2e67e16a8 (object 0x000486ce6620, a 
> org.apache.hadoop.hdfs.DFSOutputStream),
>   which is held by "MemStoreFlusher.0"
> "MemStoreFlusher.0":
>   waiting to lock monitor 0x7ff27cfa5218 (object 0x0002fae5ebe0, a 
> org.apache.hadoop.hdfs.LeaseRenewer),
>   which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"
> Java stack information for the threads listed above:
> ===
> "MemStoreFlusher.1":
>   at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
>   - waiting to lock <0x0002fae5ebe0> (a 
> org.apache.hadoop.hdfs.LeaseRenewer)
>   at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
>   at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
>   at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
>   - locked <0x00055b606cb0> (a org.apache.hadoop.hdfs.DFSOutputStream)
>   at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
>   at 
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
>   at 
> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
>   - locked <0x00059869eed8> (a java.lang.Object)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:211)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:66)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:238)
>   at java.lang.Thread.run(Thread.java:744)
> "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:1822)
>   - waiting to lock <0x000486ce6620> (a 
> org.apache.hadoop.hdfs.DFSOutputStream)
>   at 
> org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:780)
>   at org.apache.hadoop.hdfs.DFSClient.abort(DFSClient.java:753)
>   at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:453)
>   - locked <0x0002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
>   at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>   at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>   at java.lang.Thread.run(Thread.java:744)
> "MemStoreFlusher.0":
>   at org.apache.hadoop.hdfs.Leas

[jira] [Commented] (HDFS-9294) DFSClient deadlock when close file and failed to renew lease

2015-11-25 Thread JIRA


[ 
https://issues.apache.org/jira/browse/HDFS-9294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028303#comment-15028303
 ] 

邓飞 commented on HDFS-9294:
--

Sorry,It's too late finish it.
This is our solution,we just move "dfsClient.endFileLease"  out of synchronized 
block which try to acquire LeaseRenewer lock while DFSOutputStream is closing 
or aborting, rather then set 'closing' or 'aborting' flag on 
DFSOutputStream,and let the LeaseRenewer  thread detect and  skip.
but is't not graceful,do you have any better idea?

> DFSClient  deadlock when close file and failed to renew lease
> -
>
> Key: HDFS-9294
> URL: https://issues.apache.org/jira/browse/HDFS-9294
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Affects Versions: 2.2.0, 2.7.1
> Environment: Hadoop 2.2.0
>Reporter: 邓飞
>Assignee: 邓飞
>
> We found a deadlock at our HBase(0.98) cluster(and the Hadoop Version is 
> 2.2.0),and it should be HDFS BUG,at the time our network is not stable.
>  below is the stack:
> *
> Found one Java-level deadlock:
> =
> "MemStoreFlusher.1":
>   waiting to lock monitor 0x7ff27cfa5218 (object 0x0002fae5ebe0, a 
> org.apache.hadoop.hdfs.LeaseRenewer),
>   which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"
> "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
>   waiting to lock monitor 0x7ff2e67e16a8 (object 0x000486ce6620, a 
> org.apache.hadoop.hdfs.DFSOutputStream),
>   which is held by "MemStoreFlusher.0"
> "MemStoreFlusher.0":
>   waiting to lock monitor 0x7ff27cfa5218 (object 0x0002fae5ebe0, a 
> org.apache.hadoop.hdfs.LeaseRenewer),
>   which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"
> Java stack information for the threads listed above:
> ===
> "MemStoreFlusher.1":
>   at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
>   - waiting to lock <0x0002fae5ebe0> (a 
> org.apache.hadoop.hdfs.LeaseRenewer)
>   at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
>   at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
>   at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
>   - locked <0x00055b606cb0> (a org.apache.hadoop.hdfs.DFSOutputStream)
>   at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
>   at 
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
>   at 
> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
>   - locked <0x00059869eed8> (a java.lang.Object)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:211)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:66)
>   at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:238)
>   at java.lang.Thread.run(Thread.java:744)
> "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:1822)
>   - waiting to lock <0x000486ce6620> (a 
> org.apache.hadoop.hdfs.DFSOutputStream)
>   at 
> org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:780)
>   at org.apache.hadoop.hdfs.DFSClient.abort(DFSClient.java:753)
>   at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:453)
>   - locked <0x0002fae5ebe0>

[jira] [Commented] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028289#comment-15028289
 ] 

Hadoop QA commented on HDFS-9457:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s 
{color} | {color:red} Patch generated 28 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 126, now 148). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 16s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 8s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 24s 
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 212m 41s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
|   | hadoop.hdfs.TestPread |
|   | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
|   | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | hadoop.hdfs.tools.TestDFSAdminWithHA |
|   | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
|   | hadoop.hdfs.server.datanode.TestDirectoryScanner |
|   | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|

[jira] [Commented] (HDFS-8415) Erasure coding: rotated parity placement (RAID5/6)

2015-11-25 Thread Zhe Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028276#comment-15028276
 ] 

Zhe Zhang commented on HDFS-8415:
-

[~szetszwo] I think the upsides are similar to the upsides of RAID5 (rotating 
parity) compared to RAID3. Quoting from Wikipedia:
{quote}
In comparison to RAID 4, RAID 5's distributed parity evens out the stress of a 
dedicated parity disk among all RAID members. Additionally, read performance is 
increased since all RAID members participate in serving of the read requests.
{quote}
If we rotate the role of data / parity blocks for each stripe of cells, then 
we'll be reading from {{DN0, DN1, DN2, DN3, DN4, DN5}} for the 1st stripe, and 
{{DN1, DN2, DN3, DN4, DN5, DN6}} for the 2nd stripe. In theory this could 
leverage 9 disk spindles in parallel, instead of 6 without rotation.

The downside is the added complexity.

> Erasure coding: rotated parity placement (RAID5/6)
> --
>
> Key: HDFS-8415
> URL: https://issues.apache.org/jira/browse/HDFS-8415
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> Our current implementation uses fixed internal blocks for parity data, 
> similar to RAID3. Rotated parity placement like RAID5/6 has interesting 
> tradeoffs that we should investigate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Xiao Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9470:

Attachment: HDFS-9470.003.patch

Patch 3 to fix the style warning.
Failed tests are not related.

> Encryption zone on root not loaded from fsimage after NN restart
> 
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch, HDFS-9470.002.patch, 
> HDFS-9470.003.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8031) Follow-on work for erasure coding phase I (striping layout)

2015-11-25 Thread Zhe Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028269#comment-15028269
 ] 

Zhe Zhang commented on HDFS-8031:
-

Thanks for the report Rui! Great to know that EC works on a 20-node cluster. 
Two minor questions:
# The link to the test plan is broken
# Is it possible to add the hardware configuration of the nodes?

> Follow-on work for erasure coding phase I (striping layout)
> ---
>
> Key: HDFS-8031
> URL: https://issues.apache.org/jira/browse/HDFS-8031
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFSErasureCodingSystemTestReport-20151106.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028261#comment-15028261
 ] 

Hadoop QA commented on HDFS-9457:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 57s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s 
{color} | {color:red} Patch generated 28 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 126, now 148). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 36s 
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 18s 
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. 
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 31s 
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 255m 26s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
|   | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
|   | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
|   | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.TestRollingUpgrade |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | hadoop.hdfs.server.namenode.ha.TestSever

[jira] [Commented] (HDFS-8648) Revisit FsDirectory#resolvePath() function usage to check the call is made under proper lock

2015-11-25 Thread Rakesh R (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028248#comment-15028248
 ] 

Rakesh R commented on HDFS-8648:


Attached initial patch, here I've tried moving the {{fsd#resolvePath()}} 
function call under fsd#lock. I haven't included the {{FsDirWriteFileOp.java}} 
changes, as this requires more refactoring. Will incorporate this changes later 
in this jira or a separate sub-task.

> Revisit FsDirectory#resolvePath() function usage to check the call is made 
> under proper lock
> 
>
> Key: HDFS-8648
> URL: https://issues.apache.org/jira/browse/HDFS-8648
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8648-00.patch
>
>
> As per the 
> [discussion|https://issues.apache.org/jira/browse/HDFS-8493?focusedCommentId=14595735&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14595735]
>  in HDFS-8493 the function {{FsDirectory#resolvePath}} usage needs to be 
> reviewed. It seems there are many places it has done the resolution 
> {{fsd.resolvePath(pc, src, pathComponents);}} by acquiring only fsn lock and 
> not fsd lock. As per the initial analysis following are such cases, probably 
> it needs to filter out and fix wrong usage.
> # FsDirAclOp.java
> -> getAclStatus()
> -> modifyAclEntries()
> -> removeAcl()
> -> removeDefaultAcl()
> -> setAcl()
> -> getAclStatus()
> # FsDirDeleteOp.java
> -> delete(fsn, src, recursive, logRetryCache)
> # FsDirRenameOp.java
> -> renameToInt(fsd, srcArg, dstArg, logRetryCache)
> -> renameToInt(fsd, srcArg, dstArg, logRetryCache, options)
> # FsDirStatAndListingOp.java
> -> getContentSummary(fsd, src)
> -> getFileInfo(fsd, srcArg, resolveLink)
> -> isFileClosed(fsd, src)
> -> getListingInt(fsd, srcArg, startAfter, needLocation)
> # FsDirWriteFileOp.java
> -> abandonBlock()
> -> completeFile(fsn, pc, srcArg, holder, last, fileId)
> -> getEncryptionKeyInfo(fsn, pc, src, supportedVersions)
> -> startFile()
> -> validateAddBlock()
> # FsDirXAttrOp.java
> -> getXAttrs(fsd, srcArg, xAttrs)
> -> listXAttrs(fsd, src)
> -> setXAttr(fsd, src, xAttr, flag, logRetryCache)
> # FSNamesystem.java
> -> createEncryptionZoneInt()
> -> getEZForPath()
> Thanks [~wheat9], [~vinayrpet] for the advice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8648) Revisit FsDirectory#resolvePath() function usage to check the call is made under proper lock

2015-11-25 Thread Rakesh R (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh R updated HDFS-8648:
---
Status: Patch Available  (was: Open)

> Revisit FsDirectory#resolvePath() function usage to check the call is made 
> under proper lock
> 
>
> Key: HDFS-8648
> URL: https://issues.apache.org/jira/browse/HDFS-8648
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8648-00.patch
>
>
> As per the 
> [discussion|https://issues.apache.org/jira/browse/HDFS-8493?focusedCommentId=14595735&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14595735]
>  in HDFS-8493 the function {{FsDirectory#resolvePath}} usage needs to be 
> reviewed. It seems there are many places it has done the resolution 
> {{fsd.resolvePath(pc, src, pathComponents);}} by acquiring only fsn lock and 
> not fsd lock. As per the initial analysis following are such cases, probably 
> it needs to filter out and fix wrong usage.
> # FsDirAclOp.java
> -> getAclStatus()
> -> modifyAclEntries()
> -> removeAcl()
> -> removeDefaultAcl()
> -> setAcl()
> -> getAclStatus()
> # FsDirDeleteOp.java
> -> delete(fsn, src, recursive, logRetryCache)
> # FsDirRenameOp.java
> -> renameToInt(fsd, srcArg, dstArg, logRetryCache)
> -> renameToInt(fsd, srcArg, dstArg, logRetryCache, options)
> # FsDirStatAndListingOp.java
> -> getContentSummary(fsd, src)
> -> getFileInfo(fsd, srcArg, resolveLink)
> -> isFileClosed(fsd, src)
> -> getListingInt(fsd, srcArg, startAfter, needLocation)
> # FsDirWriteFileOp.java
> -> abandonBlock()
> -> completeFile(fsn, pc, srcArg, holder, last, fileId)
> -> getEncryptionKeyInfo(fsn, pc, src, supportedVersions)
> -> startFile()
> -> validateAddBlock()
> # FsDirXAttrOp.java
> -> getXAttrs(fsd, srcArg, xAttrs)
> -> listXAttrs(fsd, src)
> -> setXAttr(fsd, src, xAttr, flag, logRetryCache)
> # FSNamesystem.java
> -> createEncryptionZoneInt()
> -> getEZForPath()
> Thanks [~wheat9], [~vinayrpet] for the advice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9426) Rollingupgrade finalization is not backward compatible

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028247#comment-15028247
 ] 

Hudson commented on HDFS-9426:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #727 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/727/])
HDFS-9426. Rollingupgrade finalization is not backward compatible 
(vinayakumarb: rev 9f256d1d716a7e17606245fcfc619901a8fa299a)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java


> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in 
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade 
> because it's not null.
> NN might have to check the DN version and return different 
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but 
> unfortunately did not address the semantics of the heartbeat response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9445) Deadlock in datanode

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028237#comment-15028237
 ] 

Hadoop QA commented on HDFS-9445:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s 
{color} | {color:red} Patch generated 1 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 123, now 123). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 28s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 15s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s 
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 30s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
|   | hadoop.hdfs.TestWriteReadStripedFile |
|   | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
|   | hadoop.hdfs.TestHFlush |
|   | hadoop.hdfs.server.datanode.TestBlockScanner |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
|   | hadoop.hdfs.web.TestWebHdfsTimeouts |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| JDK v1.7.0_85 Failed junit tests

[jira] [Updated] (HDFS-8648) Revisit FsDirectory#resolvePath() function usage to check the call is made under proper lock

2015-11-25 Thread Rakesh R (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh R updated HDFS-8648:
---
Attachment: HDFS-8648-00.patch

> Revisit FsDirectory#resolvePath() function usage to check the call is made 
> under proper lock
> 
>
> Key: HDFS-8648
> URL: https://issues.apache.org/jira/browse/HDFS-8648
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8648-00.patch
>
>
> As per the 
> [discussion|https://issues.apache.org/jira/browse/HDFS-8493?focusedCommentId=14595735&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14595735]
>  in HDFS-8493 the function {{FsDirectory#resolvePath}} usage needs to be 
> reviewed. It seems there are many places it has done the resolution 
> {{fsd.resolvePath(pc, src, pathComponents);}} by acquiring only fsn lock and 
> not fsd lock. As per the initial analysis following are such cases, probably 
> it needs to filter out and fix wrong usage.
> # FsDirAclOp.java
> -> getAclStatus()
> -> modifyAclEntries()
> -> removeAcl()
> -> removeDefaultAcl()
> -> setAcl()
> -> getAclStatus()
> # FsDirDeleteOp.java
> -> delete(fsn, src, recursive, logRetryCache)
> # FsDirRenameOp.java
> -> renameToInt(fsd, srcArg, dstArg, logRetryCache)
> -> renameToInt(fsd, srcArg, dstArg, logRetryCache, options)
> # FsDirStatAndListingOp.java
> -> getContentSummary(fsd, src)
> -> getFileInfo(fsd, srcArg, resolveLink)
> -> isFileClosed(fsd, src)
> -> getListingInt(fsd, srcArg, startAfter, needLocation)
> # FsDirWriteFileOp.java
> -> abandonBlock()
> -> completeFile(fsn, pc, srcArg, holder, last, fileId)
> -> getEncryptionKeyInfo(fsn, pc, src, supportedVersions)
> -> startFile()
> -> validateAddBlock()
> # FsDirXAttrOp.java
> -> getXAttrs(fsd, srcArg, xAttrs)
> -> listXAttrs(fsd, src)
> -> setXAttr(fsd, src, xAttr, flag, logRetryCache)
> # FSNamesystem.java
> -> createEncryptionZoneInt()
> -> getEZForPath()
> Thanks [~wheat9], [~vinayrpet] for the advice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028217#comment-15028217
 ] 

Brahma Reddy Battula commented on HDFS-9407:


[~shv] Thanks a lot for review and commit and thanks to others!!

> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9129) Move the safemode block count into BlockManager

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028213#comment-15028213
 ] 

Hadoop QA commented on HDFS-9129:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 17s {color} 
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 
generated 1 new issues (was 33, now 33). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 58s {color} 
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85 with JDK v1.7.0_85 
generated 1 new issues (was 35, now 35). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s 
{color} | {color:red} Patch generated 4 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 807, now 753). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 51m 8s 
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 14s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 28s 
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 35s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_85 Failed junit tests | 
hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
|   | hadoop.hdfs.TestFileCreationDelete |
|   | hadoop.hdfs.web.TestWebHDFSXAtt

[jira] [Commented] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028200#comment-15028200
 ] 

Hadoop QA commented on HDFS-9470:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s 
{color} | {color:red} Patch generated 1 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 66, now 67). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 58s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 49m 29s 
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_85. 
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s 
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 8s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
|   | hadoop.hdfs.server.datanode.TestBlockScanner |
|   | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12774450/HDFS-9470.002.patch |
| JIRA Issue | HDFS-9470 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux f25d2cde9159 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMP

[jira] [Commented] (HDFS-9426) Rollingupgrade finalization is not backward compatible

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028177#comment-15028177
 ] 

Hudson commented on HDFS-9426:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8895 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8895/])
HDFS-9426. Rollingupgrade finalization is not backward compatible 
(vinayakumarb: rev 9f256d1d716a7e17606245fcfc619901a8fa299a)
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in 
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade 
> because it's not null.
> NN might have to check the DN version and return different 
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but 
> unfortunately did not address the semantics of the heartbeat response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9467) Fix data race accessing writeLockHeldTimeStamp in FSNamesystem

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028160#comment-15028160
 ] 

Hudson commented on HDFS-9467:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/])
HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev 
e556c35b0596700f9ec9d0a51cf5027259d531b5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
> --
>
> Key: HDFS-9467
> URL: https://issues.apache.org/jira/browse/HDFS-9467
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9467.000.patch, HDFS-9467.001.patch
>
>
> This is a followup of [HDFS-9145].
> Actually the value of {{writeLockInterval}} should be captured within the 
> lock. The current code has a race on {{writeLockHeldTimeStamp}}. Thanks to 
> [~jingzhao] for reporting this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028161#comment-15028161
 ] 

Hudson commented on HDFS-9407:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/])
HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by (shv: 
rev fc799ab16cc62b04fef5a416af30e4ae845dd7a5)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java


> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028156#comment-15028156
 ] 

Hadoop QA commented on HDFS-9079:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 46s 
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 42s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s 
{color} | {color:red} Patch generated 83 new checkstyle issues in 
hadoop-hdfs-project (total was 365, now 440). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 21s 
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client introduced 4 new 
FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 33s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 9s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s 
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 45s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
|  |  Synchronization performed on java.util.concurrent.BlockingQueue in 
org.apache.hadoop.hdfs.BlockMetadataCoordinator.processedAllEvents()  At 
Blo

[jira] [Updated] (HDFS-9426) Rollingupgrade finalization is not backward compatible

2015-11-25 Thread Vinayakumar B (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B updated HDFS-9426:

   Resolution: Fixed
 Assignee: Kihwal Lee
 Hadoop Flags: Reviewed
Fix Version/s: 2.7.2
   Status: Resolved  (was: Patch Available)

Committed to trunk, branch-2, branch-2.7 and branch-2.7.2

> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in 
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade 
> because it's not null.
> NN might have to check the DN version and return different 
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but 
> unfortunately did not address the semantics of the heartbeat response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9214) Reconfigure DN concurrent move on the fly

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028131#comment-15028131
 ] 

Hadoop QA commented on HDFS-9214:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s 
{color} | {color:red} Patch generated 11 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 157, now 167). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s 
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs 
issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 33s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 14s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s 
{color} | {color:red} Patch generated 59 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 54s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Boxing/unboxing to parse a primitive 
org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(String, 
String)  At 
DataNode.java:org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(String,
 String)  At DataNode.java:[line 537] |
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestAclsEndToEnd |
|   | hadoop.hdfs.tools.TestDFSAdmin |
|   | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
|   | hadoop.hdfs.TestRollingUpgrade |
|   | hadoop.hdfs.TestDFSCli

[jira] [Commented] (HDFS-9426) Rollingupgrade finalization is not backward compatible

2015-11-25 Thread Vinayakumar B (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028130#comment-15028130
 ] 

Vinayakumar B commented on HDFS-9426:
-

+1 for the patch,

Committing

> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in 
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade 
> because it's not null.
> NN might have to check the DN version and return different 
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but 
> unfortunately did not address the semantics of the heartbeat response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9449) DiskBalancer : Add connectors

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028071#comment-15028071
 ] 

Hadoop QA commented on HDFS-9449:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
36s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s 
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s 
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
27s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s 
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
11s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s 
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s 
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 45s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 14s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 24s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 30s 
{color} | {color:red} Patch generated 57 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 227m 59s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.security.TestPermission |
|   | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | hadoop.hdfs.TestDFSUpgradeFromImage |
|   | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
|   | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
|   | hadoop.hdfs.server.datanode.TestDirectoryScanner |
|   | hadoop.hdfs.TestReplaceDatanodeOnFailure |
|   | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| JDK v1.7.0_85 Failed junit tests | hadoop.security.TestPermission |
|   | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
|   | hadoop.hdfs.TestLeaseRecovery2 |
|   | hadoop.hdf

[jira] [Commented] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Xiao Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028064#comment-15028064
 ] 

Xiao Chen commented on HDFS-9470:
-

Thanks Andrew.
Also answering the question asked in HDFS-9273: this one is also long standing, 
since the initial implementation of HDFS-6516.

> Encryption zone on root not loaded from fsimage after NN restart
> 
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch, HDFS-9470.002.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028043#comment-15028043
 ] 

Hudson commented on HDFS-9407:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #736 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/736/])
HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by (shv: 
rev fc799ab16cc62b04fef5a416af30e4ae845dd7a5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9467) Fix data race accessing writeLockHeldTimeStamp in FSNamesystem

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028042#comment-15028042
 ] 

Hudson commented on HDFS-9467:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #736 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/736/])
HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev 
e556c35b0596700f9ec9d0a51cf5027259d531b5)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


> Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
> --
>
> Key: HDFS-9467
> URL: https://issues.apache.org/jira/browse/HDFS-9467
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9467.000.patch, HDFS-9467.001.patch
>
>
> This is a followup of [HDFS-9145].
> Actually the value of {{writeLockInterval}} should be captured within the 
> lock. The current code has a race on {{writeLockHeldTimeStamp}}. Thanks to 
> [~jingzhao] for reporting this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028033#comment-15028033
 ] 

Hudson commented on HDFS-9407:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/])
HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by (shv: 
rev fc799ab16cc62b04fef5a416af30e4ae845dd7a5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9451) Clean up depreated umasks and related unit tests

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028030#comment-15028030
 ] 

Hudson commented on HDFS-9451:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/])
HDFS-9451. Clean up depreated umasks and related unit tests. Contributed 
(wheat9: rev b21dffb1fec376784810af89cfc9cc05e5f781ce)
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/permission/TestFsPermission.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java


> Clean up depreated umasks and related unit tests
> 
>
> Key: HDFS-9451
> URL: https://issues.apache.org/jira/browse/HDFS-9451
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 3.0.0
>
> Attachments: HDFS-9451.001.patch, HDFS-9451.002.patch, 
> HDFS-9451.003.patch, HDFS-9451.004.patch
>
>
> I noticed this test failed consistently since yesterday. The first failed 
> jenkins job is 
> https://builds.apache.org/job/Hadoop-common-trunk-Java8/723/changes, and from 
> the change log:
> {noformat}
> Changes:
> [wheat9] HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter 
> Su.
> [wheat9] HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
> [wheat9] HADOOP-12467. Respect user-defined JAVA_LIBRARY_PATH in Windows 
> Hadoop
> [wheat9] HDFS-8914. Document HA support in the HDFS HdfsDesign.md. 
> Contributed by
> [wheat9] HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai 
> Zheng.
> [wheat9] HDFS-7796. Include X-editable for slick contenteditable fields in the
> [wheat9] HDFS-3302. Review and improve HDFS trash documentation. Contributed 
> by
> [wheat9] HADOOP-12294. Remove the support of the deprecated dfs.umask.
> {noformat}
> HADOOP-12294 looks to be the most likely cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9467) Fix data race accessing writeLockHeldTimeStamp in FSNamesystem

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028032#comment-15028032
 ] 

Hudson commented on HDFS-9467:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/])
HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev 
e556c35b0596700f9ec9d0a51cf5027259d531b5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
> --
>
> Key: HDFS-9467
> URL: https://issues.apache.org/jira/browse/HDFS-9467
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9467.000.patch, HDFS-9467.001.patch
>
>
> This is a followup of [HDFS-9145].
> Actually the value of {{writeLockInterval}} should be captured within the 
> lock. The current code has a race on {{writeLockHeldTimeStamp}}. Thanks to 
> [~jingzhao] for reporting this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9459) hadoop-hdfs-native-client fails test build on Windows after transition to ctest.

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028031#comment-15028031
 ] 

Hudson commented on HDFS-9459:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/])
HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: 
rev 95d5227c75f430da7c77847f31734b34b36157d2)
* hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> hadoop-hdfs-native-client fails test build on Windows after transition to 
> ctest.
> 
>
> Key: HDFS-9459
> URL: https://issues.apache.org/jira/browse/HDFS-9459
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build, test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Blocker
> Fix For: 2.8.0
>
>
> HDFS-9369 transitioned to usage of {{ctest}} for running the HDFS native 
> tests.  This broke the {{mvn test}} build on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8512) WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028029#comment-15028029
 ] 

Hudson commented on HDFS-8512:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2581 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2581/])
HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with (xyao: rev 
e3d673901b396cf5bbede5ed6f607ce68301ec0a)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java


> WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info
> -
>
> Key: HDFS-8512
> URL: https://issues.apache.org/jira/browse/HDFS-8512
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Sumana Sathish
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8512.00.patch, HDFS-8512.01.patch
>
>
> Storage type inside LocatedBlock object is not fully exposed for GETFILESTATUS
> {code}
> $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GETFILESTATUS";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785853423&s=W4O5kKiYHmzzey4h7I9J9eL9EMY=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:13 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"FileStatus":{"accessTime":1432683737985,"blockSize":134217728,"childrenNum":0,"fileId":16405,"group":"hadoop","length":150318178,"modificationTime":1432683738427,"owner":"xyao","pathSuffix":"","permission":"644","replication":1,"storagePolicy":7,"type":"FILE"}}
>  $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GET_BLOCK_LOCATIONS&offset=0&length=150318178";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785895031&s=TUiaNsCrARAPKz6xrddoQ1eHOXA=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:55 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"LocatedBlocks":{"fileLength":150318178,"isLastBlockComplete":true,"isUnderConstruction":false,"lastLocatedBlock":{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":134217728},"locatedBlocks":[{"block":{"blockId":1073741846,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1022,"numBytes":134217728},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":0},{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default

[jira] [Updated] (HDFS-9445) Deadlock in datanode

2015-11-25 Thread Walter Su (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Walter Su updated HDFS-9445:

Status: Patch Available  (was: Open)

> Deadlock in datanode
> 
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Assignee: Walter Su
>Priority: Blocker
> Attachments: HDFS-9445.00.patch, HDFS-9445.01.patch
>
>
> {noformat}
> Found one Java-level deadlock:
> =
> "DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block 
> BP-x:blk_123_456]":
>   waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
>   which is held by "Thread-565"
> "Thread-565":
>   waiting for ownable synchronizer 0xd55613c8, (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "DataNode: heartbeating to my-nn:8020"
> "DataNode: heartbeating to my-nn:8020":
>   waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
>   which is held by "Thread-565"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9445) Deadlock in datanode

2015-11-25 Thread Walter Su (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Walter Su updated HDFS-9445:

Attachment: HDFS-9445.01.patch

Sorry for late update. Let's get it done.
{code}
// If a DFSClient has the replica in its cache of short-circuit file
// descriptors (and the client is using ShortCircuitShm), invalidate it.
datanode.getShortCircuitRegistry().processBlockInvalidation(
  new ExtendedBlockId(invalidBlks[i].getBlockId(), bpid));

// If the block is cached, start uncaching it.
cacheManager.uncacheBlock(bpid, invalidBlks[i].getBlockId());
{code}
The 2 lines are copied from _public void invalidate(..)_. It's a clean-up after 
remove blocks from volumeMap.  Those 2 lines are the only part same as 
{{invalidate}}.
These lines look unrelated. So it's hard to pick a proper func name for them.
Those 2 lines don't require inside the FSDatasetLock.
I just moved them to {{removeVolumes}}. Let me know if it's ok to you.

> Deadlock in datanode
> 
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Assignee: Walter Su
>Priority: Blocker
> Attachments: HDFS-9445.00.patch, HDFS-9445.01.patch
>
>
> {noformat}
> Found one Java-level deadlock:
> =
> "DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block 
> BP-x:blk_123_456]":
>   waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
>   which is held by "Thread-565"
> "Thread-565":
>   waiting for ownable synchronizer 0xd55613c8, (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "DataNode: heartbeating to my-nn:8020"
> "DataNode: heartbeating to my-nn:8020":
>   waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
>   which is held by "Thread-565"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default

2015-11-25 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027999#comment-15027999
 ] 

Mingliang Liu commented on HDFS-9436:
-

Thank you very much for the detailed explanation. This is very helpful to me to 
understand the benchmark.

# The {{numOpsRequired}} default value 10 and number of datanodes default 10 
together seem OK. So perhaps we don't need to change {{numOpsRequired}} here?
# My second concern in last comment was that {{TinyDatanode#blocks}} is an 
empty ArrayList with initial capacity. In {{TinyDatanode#addBlock()}} first 
statement, the {{if(nrBlocks == blocks.size()) {}} will always be true. I 
believe this is wrong in current {{trunk}} code. We should either fill the 
{{blocks}} with dummy report in {{TinyDatanode()}} constructor, or use initial 
capacity instead of blocks.size() in the above _if_ statement (we should 
replace {{ArrayList#set}} with {{ArrayList#add}} as well). The overall change 
is like following (not tested):
{code}
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
index 4db4da0..0ff7893 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
@@ -929,6 +929,7 @@ long executeOp(int daemonId, int inputIdx, String ignore)
 int nrBlocks; // actual number of blocks
 BlockListAsLongs blockReportList;
 final int dnIdx;
+private final int blockCapacity;

 private static int getNodePort(int num) throws IOException {
   int port = 1 + num;
@@ -939,6 +940,7 @@ private static int getNodePort(int num) throws IOException {
 TinyDatanode(int dnIdx, int blockCapacity) throws IOException {
   this.dnIdx = dnIdx;
   this.blocks = new ArrayList(blockCapacity);
+  this.blockCapacity = blockCapacity;
   this.nrBlocks = 0;
 }

@@ -996,22 +998,22 @@ void sendHeartbeat() throws IOException {
 }

 boolean addBlock(Block blk) {
-  if(nrBlocks == blocks.size()) {
+  if(nrBlocks == blockCapacity) {
 if(LOG.isDebugEnabled()) {
   LOG.debug("Cannot add block: datanode capacity = " + blocks.size());
 }
 return false;
   }
-  blocks.set(nrBlocks, new BlockReportReplica(blk));
+  blocks.add(new BlockReportReplica(blk));
   nrBlocks++;
   return true;
 }

 void formBlockReport() {
   // fill remaining slots with blocks that do not exist
-  for (int idx = blocks.size()-1; idx >= nrBlocks; idx--) {
+  for (int idx = blockCapacity - 1; idx >= nrBlocks; idx--) {
 Block block = new Block(blocks.size() - idx, 0, 0);
-blocks.set(idx, new BlockReportReplica(block));
+blocks.add(new BlockReportReplica(block));
   }
   blockReportList = BlockListAsLongs.EMPTY;
 }
{code}
Scroll down if not fully shown.

> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
> 
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good 
> to change the default for {{numThreads}} as a >=10 value and may be 
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in 
> [HDFS-9379] is tested in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-11-25 Thread Chris Trezzo (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027993#comment-15027993
 ] 

Chris Trezzo commented on HDFS-8791:


Thanks all for the comments.

[~cmccabe]
bq. how long did these upgrades take in the 0.5 million, 1.2 million, and 2.7 
million block cases?
I have attached a [new 
version|https://issues.apache.org/jira/secure/attachment/12774454/32x32DatanodeLayoutTesting-v2.pdf]
 of the testing document with more details around the upgrade testing. The 
upgrade for the above case was a setup with very low block density and the data 
node upgraded with a startup time of 1 minute. I did do an upgrade test with a 
data node that had around 2 million blocks in total. In that case the hard 
linking alone took around 9 minutes.

[~andrew.wang] Agreed. I think it would be awesome if we could get this into 
branch-2 and I am currently in the process of adding unit tests.

[~kihwal] I took a look at a node during upgrade, and it seemed like the 
{{previous.tmp}} directory did indeed have the old layout like it should. Maybe 
I am misunderstanding which directory you are looking at, so I will continue to 
investigate.

As a side note: I am still early in my search, but I can't seem to find where 
in a unit test we actually verify that the {{finalized}} directory does indeed 
have the correct layout after an upgrade. The same goes for if the 
{{previous.tmp}} directory actually has the old format during an upgrade that 
isn't finalized yet. I see 
{{TestDatanodeLayoutUpgrade#testUpgradeToIdBasedLayout}}, but a {{null}} 
verifier is passed in to the {{upgradeAndVerify}} method. Additionally, all of 
the other tests (i.e. TestDFSFinalize, TestRollingUpgradeRollback, 
TestRollingUpgrade) seem to either be layout agnostic or simply check the 
{{VERSION}} file. I will continue to investigate.

> block ID-based DN storage layout can be very slow for datanode on ext4
> --
>
> Key: HDFS-8791
> URL: https://issues.apache.org/jira/browse/HDFS-8791
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0, 2.8.0, 2.7.1
>Reporter: Nathan Roberts
>Assignee: Chris Trezzo
>Priority: Critical
> Attachments: 32x32DatanodeLayoutTesting-v1.pdf, 
> 32x32DatanodeLayoutTesting-v2.pdf, HDFS-8791-trunk-v1.patch
>
>
> We are seeing cases where the new directory layout causes the datanode to 
> basically cause the disks to seek for 10s of minutes. This can be when the 
> datanode is running du, and it can also be when it is performing a 
> checkDirs(). Both of these operations currently scan all directories in the 
> block pool and that's very expensive in the new layout.
> The new layout creates 256 subdirs, each with 256 subdirs. Essentially 64K 
> leaf directories where block files are placed.
> So, what we have on disk is:
> - 256 inodes for the first level directories
> - 256 directory blocks for the first level directories
> - 256*256 inodes for the second level directories
> - 256*256 directory blocks for the second level directories
> - Then the inodes and blocks to store the the HDFS blocks themselves.
> The main problem is the 256*256 directory blocks. 
> inodes and dentries will be cached by linux and one can configure how likely 
> the system is to prune those entries (vfs_cache_pressure). However, ext4 
> relies on the buffer cache to cache the directory blocks and I'm not aware of 
> any way to tell linux to favor buffer cache pages (even if it did I'm not 
> sure I would want it to in general).
> Also, ext4 tries hard to spread directories evenly across the entire volume, 
> this basically means the 64K directory blocks are probably randomly spread 
> across the entire disk. A du type scan will look at directories one at a 
> time, so the ioscheduler can't optimize the corresponding seeks, meaning the 
> seeks will be random and far. 
> In a system I was using to diagnose this, I had 60K blocks. A DU when things 
> are hot is less than 1 second. When things are cold, about 20 minutes.
> How do things get cold?
> - A large set of tasks run on the node. This pushes almost all of the buffer 
> cache out, causing the next DU to hit this situation. We are seeing cases 
> where a large job can cause a seek storm across the entire cluster.
> Why didn't the previous layout see this?
> - It might have but it wasn't nearly as pronounced. The previous layout would 
> be a few hundred directory blocks. Even when completely cold, these would 
> only take a few a hundred seeks which would mean single digit seconds.  
> - With only a few hundred directories, the odds of the directory blocks 
> getting modified is quite high, this keeps those blocks hot and much less 
> likely to be evicted.



--
This messa

[jira] [Commented] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027989#comment-15027989
 ] 

Andrew Wang commented on HDFS-9470:
---

+1 LGTM pending jenkins

> Encryption zone on root not loaded from fsimage after NN restart
> 
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch, HDFS-9470.002.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9467) Fix data race accessing writeLockHeldTimeStamp in FSNamesystem

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027985#comment-15027985
 ] 

Hudson commented on HDFS-9467:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #725 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/725/])
HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev 
e556c35b0596700f9ec9d0a51cf5027259d531b5)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


> Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
> --
>
> Key: HDFS-9467
> URL: https://issues.apache.org/jira/browse/HDFS-9467
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9467.000.patch, HDFS-9467.001.patch
>
>
> This is a followup of [HDFS-9145].
> Actually the value of {{writeLockInterval}} should be captured within the 
> lock. The current code has a race on {{writeLockHeldTimeStamp}}. Thanks to 
> [~jingzhao] for reporting this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027986#comment-15027986
 ] 

Hudson commented on HDFS-9407:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #725 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/725/])
HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by (shv: 
rev fc799ab16cc62b04fef5a416af30e4ae845dd7a5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-11-25 Thread Chris Trezzo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Trezzo updated HDFS-8791:
---
Attachment: 32x32DatanodeLayoutTesting-v2.pdf

> block ID-based DN storage layout can be very slow for datanode on ext4
> --
>
> Key: HDFS-8791
> URL: https://issues.apache.org/jira/browse/HDFS-8791
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0, 2.8.0, 2.7.1
>Reporter: Nathan Roberts
>Assignee: Chris Trezzo
>Priority: Critical
> Attachments: 32x32DatanodeLayoutTesting-v1.pdf, 
> 32x32DatanodeLayoutTesting-v2.pdf, HDFS-8791-trunk-v1.patch
>
>
> We are seeing cases where the new directory layout causes the datanode to 
> basically cause the disks to seek for 10s of minutes. This can be when the 
> datanode is running du, and it can also be when it is performing a 
> checkDirs(). Both of these operations currently scan all directories in the 
> block pool and that's very expensive in the new layout.
> The new layout creates 256 subdirs, each with 256 subdirs. Essentially 64K 
> leaf directories where block files are placed.
> So, what we have on disk is:
> - 256 inodes for the first level directories
> - 256 directory blocks for the first level directories
> - 256*256 inodes for the second level directories
> - 256*256 directory blocks for the second level directories
> - Then the inodes and blocks to store the the HDFS blocks themselves.
> The main problem is the 256*256 directory blocks. 
> inodes and dentries will be cached by linux and one can configure how likely 
> the system is to prune those entries (vfs_cache_pressure). However, ext4 
> relies on the buffer cache to cache the directory blocks and I'm not aware of 
> any way to tell linux to favor buffer cache pages (even if it did I'm not 
> sure I would want it to in general).
> Also, ext4 tries hard to spread directories evenly across the entire volume, 
> this basically means the 64K directory blocks are probably randomly spread 
> across the entire disk. A du type scan will look at directories one at a 
> time, so the ioscheduler can't optimize the corresponding seeks, meaning the 
> seeks will be random and far. 
> In a system I was using to diagnose this, I had 60K blocks. A DU when things 
> are hot is less than 1 second. When things are cold, about 20 minutes.
> How do things get cold?
> - A large set of tasks run on the node. This pushes almost all of the buffer 
> cache out, causing the next DU to hit this situation. We are seeing cases 
> where a large job can cause a seek storm across the entire cluster.
> Why didn't the previous layout see this?
> - It might have but it wasn't nearly as pronounced. The previous layout would 
> be a few hundred directory blocks. Even when completely cold, these would 
> only take a few a hundred seeks which would mean single digit seconds.  
> - With only a few hundred directories, the odds of the directory blocks 
> getting modified is quite high, this keeps those blocks hot and much less 
> likely to be evicted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027972#comment-15027972
 ] 

Mingliang Liu commented on HDFS-9457:
-

Instead of deleting the previous patch and upload a new one with the same file 
name, you can simply upload a new version patch with different names. For 
example, _HDFS-9457.000.patch_, _HDFS-9457.001.patch_, _HDFS-9457.002.patch_, 
_HDFS-9457.003.patch_...

I think both will work just fine. You're right to preserve what was already 
there, especially the format. My preference may be subjective, plus I don't 
have commit access as I'm a contributor (not committer) yet. Let's wait for 
more input from others.

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9459) hadoop-hdfs-native-client fails test build on Windows after transition to ctest.

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027971#comment-15027971
 ] 

Hudson commented on HDFS-9459:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #644 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/644/])
HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: 
rev 95d5227c75f430da7c77847f31734b34b36157d2)
* hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> hadoop-hdfs-native-client fails test build on Windows after transition to 
> ctest.
> 
>
> Key: HDFS-9459
> URL: https://issues.apache.org/jira/browse/HDFS-9459
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build, test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Blocker
> Fix For: 2.8.0
>
>
> HDFS-9369 transitioned to usage of {{ctest}} for running the HDFS native 
> tests.  This broke the {{mvn test}} build on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9451) Clean up depreated umasks and related unit tests

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027970#comment-15027970
 ] 

Hudson commented on HDFS-9451:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #644 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/644/])
HDFS-9451. Clean up depreated umasks and related unit tests. Contributed 
(wheat9: rev b21dffb1fec376784810af89cfc9cc05e5f781ce)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/permission/TestFsPermission.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java


> Clean up depreated umasks and related unit tests
> 
>
> Key: HDFS-9451
> URL: https://issues.apache.org/jira/browse/HDFS-9451
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 3.0.0
>
> Attachments: HDFS-9451.001.patch, HDFS-9451.002.patch, 
> HDFS-9451.003.patch, HDFS-9451.004.patch
>
>
> I noticed this test failed consistently since yesterday. The first failed 
> jenkins job is 
> https://builds.apache.org/job/Hadoop-common-trunk-Java8/723/changes, and from 
> the change log:
> {noformat}
> Changes:
> [wheat9] HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter 
> Su.
> [wheat9] HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
> [wheat9] HADOOP-12467. Respect user-defined JAVA_LIBRARY_PATH in Windows 
> Hadoop
> [wheat9] HDFS-8914. Document HA support in the HDFS HdfsDesign.md. 
> Contributed by
> [wheat9] HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai 
> Zheng.
> [wheat9] HDFS-7796. Include X-editable for slick contenteditable fields in the
> [wheat9] HDFS-3302. Review and improve HDFS trash documentation. Contributed 
> by
> [wheat9] HADOOP-12294. Remove the support of the deprecated dfs.umask.
> {noformat}
> HADOOP-12294 looks to be the most likely cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8512) WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027969#comment-15027969
 ] 

Hudson commented on HDFS-8512:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #644 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/644/])
HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with (xyao: rev 
e3d673901b396cf5bbede5ed6f607ce68301ec0a)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java


> WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info
> -
>
> Key: HDFS-8512
> URL: https://issues.apache.org/jira/browse/HDFS-8512
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Sumana Sathish
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8512.00.patch, HDFS-8512.01.patch
>
>
> Storage type inside LocatedBlock object is not fully exposed for GETFILESTATUS
> {code}
> $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GETFILESTATUS";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785853423&s=W4O5kKiYHmzzey4h7I9J9eL9EMY=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:13 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"FileStatus":{"accessTime":1432683737985,"blockSize":134217728,"childrenNum":0,"fileId":16405,"group":"hadoop","length":150318178,"modificationTime":1432683738427,"owner":"xyao","pathSuffix":"","permission":"644","replication":1,"storagePolicy":7,"type":"FILE"}}
>  $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GET_BLOCK_LOCATIONS&offset=0&length=150318178";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785895031&s=TUiaNsCrARAPKz6xrddoQ1eHOXA=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:55 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"LocatedBlocks":{"fileLength":150318178,"isLastBlockComplete":true,"isUnderConstruction":false,"lastLocatedBlock":{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":134217728},"locatedBlocks":[{"block":{"blockId":1073741846,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1022,"numBytes":134217728},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":0},{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation"

[jira] [Updated] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Xiao Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9470:

Status: Patch Available  (was: Open)

> Encryption zone on root not loaded from fsimage after NN restart
> 
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch, HDFS-9470.002.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027957#comment-15027957
 ] 

BELUGA BEHR commented on HDFS-9457:
---

New patch supplied.  Sorry.

I am aware of the feature, however, the current code does not take advantage of 
it.  The examples supplied in the previous comment are taken from the current 
code base.  I was trying to preserve what was already there.  Would you like 
the code, in general, to be modified to:

{code}
LOG.warn("Received exception in Datanode#join", ex);
{code}

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027958#comment-15027958
 ] 

BELUGA BEHR commented on HDFS-9457:
---

New patch supplied.  Sorry.

I am aware of the feature, however, the current code does not take advantage of 
it.  The examples supplied in the previous comment are taken from the current 
code base.  I was trying to preserve what was already there.  Would you like 
the code, in general, to be modified to:

{code}
LOG.warn("Received exception in Datanode#join", ex);
{code}

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default

2015-11-25 Thread Konstantin Shvachko (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko updated HDFS-9436:
--
Target Version/s: 2.8.0
   Fix Version/s: (was: 2.8.0)

> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
> 
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good 
> to change the default for {{numThreads}} as a >=10 value and may be 
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in 
> [HDFS-9379] is tested in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9129) Move the safemode block count into BlockManager

2015-11-25 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-9129:

Attachment: HDFS-9129.024.patch

Per offline discussion with [~jingzhao], the v24 patch removes the 
{{shouldIncrementallyTrackBlocks}} as it's not needed, and initializes the 
{{startTime}} in {{activate}}.

After this change, all previous comments should have been addressed.

> Move the safemode block count into BlockManager
> ---
>
> Key: HDFS-9129
> URL: https://issues.apache.org/jira/browse/HDFS-9129
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-9129.000.patch, HDFS-9129.001.patch, 
> HDFS-9129.002.patch, HDFS-9129.003.patch, HDFS-9129.004.patch, 
> HDFS-9129.005.patch, HDFS-9129.006.patch, HDFS-9129.007.patch, 
> HDFS-9129.008.patch, HDFS-9129.009.patch, HDFS-9129.010.patch, 
> HDFS-9129.011.patch, HDFS-9129.012.patch, HDFS-9129.013.patch, 
> HDFS-9129.014.patch, HDFS-9129.015.patch, HDFS-9129.016.patch, 
> HDFS-9129.017.patch, HDFS-9129.018.patch, HDFS-9129.019.patch, 
> HDFS-9129.020.patch, HDFS-9129.021.patch, HDFS-9129.022.patch, 
> HDFS-9129.023.patch, HDFS-9129.024.patch
>
>
> The {{SafeMode}} needs to track whether there are enough blocks so that the 
> NN can get out of the safemode. These fields can moved to the 
> {{BlockManager}} class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Xiao Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027949#comment-15027949
 ] 

Xiao Chen commented on HDFS-9470:
-

Thanks a lot [~andrew.wang] for the review and setting target versions.
Your comments make sense to me. Attached patch 2 to refactor as you suggested.

> Encryption zone on root not loaded from fsimage after NN restart
> 
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch, HDFS-9470.002.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HDFS-9457:
--
Attachment: (was: logging.patch)

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HDFS-9457:
--
Attachment: logging.patch

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HDFS-9457:
--
Attachment: logging.patch

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HDFS-9457:
--
Attachment: (was: logging.patch)

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Xiao Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9470:

Attachment: HDFS-9470.002.patch

> Encryption zone on root not loaded from fsimage after NN restart
> 
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch, HDFS-9470.002.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9470) Encryption zone on root not loaded from fsimage after NN restart

2015-11-25 Thread Andrew Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-9470:
--
Affects Version/s: 2.6.0
  Summary: Encryption zone on root not loaded from fsimage after NN 
restart  (was: EncryptionZone on rootDir missing after NNrestart)
 Target Version/s: 2.7.2, 2.6.3

I went ahead and set the target versions accordingly, if the ACL fix goes in 
this one should too. Thx Xiao.

> Encryption zone on root not loaded from fsimage after NN restart
> 
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9470) EncryptionZone on rootDir missing after NNrestart

2015-11-25 Thread Xiao Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027936#comment-15027936
 ] 

Xiao Chen commented on HDFS-9470:
-

Figured I should also ping [~vinodkv] and [~sjlee0] like [~andrew.wang] did in 
HDFS-9273. This seems to be also a critical one.
I'm still putting up fix #2 though.

> EncryptionZone on rootDir missing after NNrestart
> -
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9470) EncryptionZone on rootDir missing after NNrestart

2015-11-25 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027935#comment-15027935
 ] 

Andrew Wang commented on HDFS-9470:
---

Hi Xiao, thanks for picking this up, overall looks good. Only a few comments:

* Is it possible to share some code between this new function and 
addToInodeMap? We could have a more generically named helper function, and call 
it from both addToInodeMap and loadRootInode. I don't think the log print is 
that important, can keep it generic.
* Little nit, we already have the XAttrFeature in loadRootINode, so we could 
pass it instead of looking it up again inside addRootDirToEncryptionZone.

> EncryptionZone on rootDir missing after NNrestart
> -
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default

2015-11-25 Thread Konstantin Shvachko (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027931#comment-15027931
 ] 

Konstantin Shvachko commented on HDFS-9436:
---

In {{BlockReportStats}} 
- {{numOpsRequired}} means the total number of block reports processed by NN 
from all DNs
- {{numThreads}} is the number of DNs, sending the reports, each as a separate 
thread
- {{addBlock()}} appends a new block to the list of {{blocks}}, which 
represents replicas to be reported. The list is formed from "real" blocks first 
and then padded by fake ones.
- {{blockReportList}} is a sequence of longs, representing the actual report of 
all the replicas in {{blocks}}, which is sent to the NN

Hope this clarifies. A better JavaDoc is always welcome.

I think everything works as expected, except the EMPTY array introduced as you 
mentioned by HDFS-7435. We should fix it in a different jira.

> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
> 
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good 
> to change the default for {{numThreads}} as a >=10 value and may be 
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in 
> [HDFS-9379] is tested in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9467) Fix data race accessing writeLockHeldTimeStamp in FSNamesystem

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027925#comment-15027925
 ] 

Hudson commented on HDFS-9467:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2667 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2667/])
HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev 
e556c35b0596700f9ec9d0a51cf5027259d531b5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
> --
>
> Key: HDFS-9467
> URL: https://issues.apache.org/jira/browse/HDFS-9467
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9467.000.patch, HDFS-9467.001.patch
>
>
> This is a followup of [HDFS-9145].
> Actually the value of {{writeLockInterval}} should be captured within the 
> lock. The current code has a race on {{writeLockHeldTimeStamp}}. Thanks to 
> [~jingzhao] for reporting this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9470) EncryptionZone on rootDir missing after NNrestart

2015-11-25 Thread Xiao Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9470:

Priority: Critical  (was: Major)

> EncryptionZone on rootDir missing after NNrestart
> -
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9470.001.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027926#comment-15027926
 ] 

Hudson commented on HDFS-9407:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2667 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2667/])
HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by (shv: 
rev fc799ab16cc62b04fef5a416af30e4ae845dd7a5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8512) WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027923#comment-15027923
 ] 

Hudson commented on HDFS-8512:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2667 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2667/])
HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with (xyao: rev 
e3d673901b396cf5bbede5ed6f607ce68301ec0a)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java


> WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info
> -
>
> Key: HDFS-8512
> URL: https://issues.apache.org/jira/browse/HDFS-8512
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Sumana Sathish
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8512.00.patch, HDFS-8512.01.patch
>
>
> Storage type inside LocatedBlock object is not fully exposed for GETFILESTATUS
> {code}
> $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GETFILESTATUS";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785853423&s=W4O5kKiYHmzzey4h7I9J9eL9EMY=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:13 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"FileStatus":{"accessTime":1432683737985,"blockSize":134217728,"childrenNum":0,"fileId":16405,"group":"hadoop","length":150318178,"modificationTime":1432683738427,"owner":"xyao","pathSuffix":"","permission":"644","replication":1,"storagePolicy":7,"type":"FILE"}}
>  $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GET_BLOCK_LOCATIONS&offset=0&length=150318178";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785895031&s=TUiaNsCrARAPKz6xrddoQ1eHOXA=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:55 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"LocatedBlocks":{"fileLength":150318178,"isLastBlockComplete":true,"isUnderConstruction":false,"lastLocatedBlock":{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":134217728},"locatedBlocks":[{"block":{"blockId":1073741846,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1022,"numBytes":134217728},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":0},{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation"

[jira] [Commented] (HDFS-9451) Clean up depreated umasks and related unit tests

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027924#comment-15027924
 ] 

Hudson commented on HDFS-9451:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2667 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2667/])
HDFS-9451. Clean up depreated umasks and related unit tests. Contributed 
(wheat9: rev b21dffb1fec376784810af89cfc9cc05e5f781ce)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/permission/TestFsPermission.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java


> Clean up depreated umasks and related unit tests
> 
>
> Key: HDFS-9451
> URL: https://issues.apache.org/jira/browse/HDFS-9451
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 3.0.0
>
> Attachments: HDFS-9451.001.patch, HDFS-9451.002.patch, 
> HDFS-9451.003.patch, HDFS-9451.004.patch
>
>
> I noticed this test failed consistently since yesterday. The first failed 
> jenkins job is 
> https://builds.apache.org/job/Hadoop-common-trunk-Java8/723/changes, and from 
> the change log:
> {noformat}
> Changes:
> [wheat9] HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter 
> Su.
> [wheat9] HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
> [wheat9] HADOOP-12467. Respect user-defined JAVA_LIBRARY_PATH in Windows 
> Hadoop
> [wheat9] HDFS-8914. Document HA support in the HDFS HdfsDesign.md. 
> Contributed by
> [wheat9] HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai 
> Zheng.
> [wheat9] HDFS-7796. Include X-editable for slick contenteditable fields in the
> [wheat9] HDFS-3302. Review and improve HDFS trash documentation. Contributed 
> by
> [wheat9] HADOOP-12294. Remove the support of the deprecated dfs.umask.
> {noformat}
> HADOOP-12294 looks to be the most likely cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6363) Improve concurrency while checking inclusion and exclusion of datanodes

2015-11-25 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027921#comment-15027921
 ] 

Mingliang Liu commented on HDFS-6363:
-

+1 (non-binding)

> Improve concurrency while checking inclusion and exclusion of datanodes
> ---
>
> Key: HDFS-6363
> URL: https://issues.apache.org/jira/browse/HDFS-6363
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Benoy Antony
>Assignee: Benoy Antony
>  Labels: BB2015-05-TBR
> Attachments: HDFS-6363-002.patch, HDFS-6363.patch
>
>
> HostFileManager holds two effectively immutable objects - includes and 
> excludes. These two objects can be safely published together using a volatile 
> container instead of synchronizing for all mutators and accessors.
> This improves the concurrency while using HostFileManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9214) Reconfigure DN concurrent move on the fly

2015-11-25 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027919#comment-15027919
 ] 

Mingliang Liu commented on HDFS-9214:
-

Overall looks good. Minor comments:

# Any logging when re-configure properties in {{DataNode}} will be helpful
# Make {{DataXceiverServer#maxThreads}} final if possible
# Annotate {{getMaxThreads()}} with @VisiableForTesting

> Reconfigure DN concurrent move on the fly
> -
>
> Key: HDFS-9214
> URL: https://issues.apache.org/jira/browse/HDFS-9214
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9214.001.patch, HDFS-9214.002.patch, 
> HDFS-9214.003.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.balance.max.concurrent.moves
> {code} without restarting DN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9273) ACLs on root directory may be lost after NN restart

2015-11-25 Thread Xiao Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027916#comment-15027916
 ] 

Xiao Chen commented on HDFS-9273:
-

Thanks Andrew for identifying this. Also thanks Chirs and Andrew for the quick 
response.
bq.  it's a long-standing bug.
+1 (non-binding)

> ACLs on root directory may be lost after NN restart
> ---
>
> Key: HDFS-9273
> URL: https://issues.apache.org/jira/browse/HDFS-9273
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.7.1
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HDFS-9273.001.patch, HDFS-9273.002.patch
>
>
> After restarting namenode, the ACLs on the root directory ("/") may be lost 
> if it's rolled over to fsimage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9470) EncryptionZone on rootDir missing after NNrestart

2015-11-25 Thread Xiao Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9470:

Attachment: HDFS-9470.001.patch

Attached patch 1 fixes the issue. Locally verified the unit test fails without 
the fix, passes with.
I choose to add the method {{addRootDirToEncryptionZone}} so that {{ezManager}} 
is still private in {{FSDirectory}}. (Directly using {{addToInodeMap}} would be 
problematic, because the {{INodeDirectory root}} is temporary and not the same 
object as {{FSDirectory.rootDir}})

> EncryptionZone on rootDir missing after NNrestart
> -
>
> Key: HDFS-9470
> URL: https://issues.apache.org/jira/browse/HDFS-9470
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9470.001.patch
>
>
> When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
> correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7435) PB encoding of block reports is very inefficient

2015-11-25 Thread Konstantin Shvachko (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027910#comment-15027910
 ] 

Konstantin Shvachko commented on HDFS-7435:
---

I agree this broke {NNThroughputBenchmark.BlockReportStats}} operation as it 
makes DNs always send an empty block report no matter what the parameters of 
the benchmark are.
Good catch Mingliang. Let's file a jira to fix this. [~daryn] are you available 
to fix this?

> PB encoding of block reports is very inefficient
> 
>
> Key: HDFS-7435
> URL: https://issues.apache.org/jira/browse/HDFS-7435
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode, namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Fix For: 2.7.0
>
> Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch, 
> HDFS-7435.002.patch, HDFS-7435.patch, HDFS-7435.patch, HDFS-7435.patch, 
> HDFS-7435.patch, HDFS-7435.patch, HDFS-7435.patch, HDFS-7435.patch, 
> HDFS-7435.patch, HDFS-7435.patch
>
>
> Block reports are encoded as a PB repeating long.  Repeating fields use an 
> {{ArrayList}} with default capacity of 10.  A block report containing tens or 
> hundreds of thousand of longs (3 for each replica) is extremely expensive 
> since the {{ArrayList}} must realloc many times.  Also, decoding repeating 
> fields will box the primitive longs which must then be unboxed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9381) When same block came for replication for Striped mode, we can move that block to PendingReplications

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027905#comment-15027905
 ] 

Hadoop QA commented on HDFS-9381:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 17s {color} 
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 
generated 2 new issues (was 33, now 33). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 6s {color} 
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85 with JDK v1.7.0_85 
generated 2 new issues (was 35, now 35). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s 
{color} | {color:red} Patch generated 1 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 169, now 169). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 44s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 1s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s 
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 177m 30s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
|   | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | hadoop.hdfs.server

[jira] [Commented] (HDFS-9381) When same block came for replication for Striped mode, we can move that block to PendingReplications

2015-11-25 Thread Zhe Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027903#comment-15027903
 ] 

Zhe Zhang commented on HDFS-9381:
-

I agree that the scenario of _a new recovery task waiting for previous tasks 
for the same striped block_ is rare. And I think the position of this change is 
to optimize the worst case consequence of this rare situation. To determine 
whether the optimization justifies the added complexity, I think we can create 
a more concrete example.

The example I can think of is: 2 DNs failed. They happen to share a pretty 
large set of striped blocks (e.g., on the order of 1,000). Roughly how much 
locking time is reduced by this change?  [~umamaheswararao] Do you have a clue 
how to estimate?

bq. I think besides reducing locking contention, this change also speeds up the 
recovery of non-striping blocks. E.g., when a rack fails, there could be a lot 
of striped block recovery work waiting. They could block regular recovery 
tasks. 
I also listed the above as another benefit. [~jingzhao] Any thoughts?

> When same block came for replication for Striped mode, we can move that block 
> to PendingReplications
> 
>
> Key: HDFS-9381
> URL: https://issues.apache.org/jira/browse/HDFS-9381
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding, namenode
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-9381-02.patch, HDFS-9381-03.patch, 
> HDFS-9381.00.patch, HDFS-9381.01.patch
>
>
> Currently I noticed that we are just returning null if block already exists 
> in pendingReplications in replication flow for striped blocks.
> {code}
> if (block.isStriped()) {
>   if (pendingNum > 0) {
> // Wait the previous recovery to finish.
> return null;
>   }
> {code}
>  Here if we just return null and if neededReplications contains only fewer 
> blocks(basically by default if less than numliveNodes*2), then same blocks 
> can be picked again from neededReplications from next loop as we are not 
> removing element from neededReplications. Since this replication process need 
> to take fsnamesystmem lock and do, we may spend some time unnecessarily in 
> every loop. 
> So my suggestion/improvement is:
>  Instead of just returning null, how about incrementing pendingReplications 
> for this block and remove from neededReplications? and also another point to 
> consider here is, to add into pendingReplications, generally we need target 
> and it is nothing but to which node we issued replication command. Later when 
> after replication success and DN reported it, block will be removed from 
> pendingReplications from NN addBlock. 
>  So since this is newly picked block from neededReplications, we would not 
> have selected target yet. So which target to be passed to pendingReplications 
> if we add this block? One Option I am thinking is, how about just passing 
> srcNode itself as target for this special condition? So, anyway if the block 
> is really missed, srcNode will not report it. So this block will not be 
> removed from pending replications, so that when it is timed out, it will be 
> considered for replication again and that time it will find actual target to 
> replicate while processing as part of regular replication flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9470) EncryptionZone on rootDir missing after NNrestart

2015-11-25 Thread Xiao Chen (JIRA)

Xiao Chen created HDFS-9470:
---

 Summary: EncryptionZone on rootDir missing after NNrestart
 Key: HDFS-9470
 URL: https://issues.apache.org/jira/browse/HDFS-9470
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Xiao Chen
Assignee: Xiao Chen


When restarting namenode, the encryption zone for {{rootDir}} is not loaded 
correctly from fsimage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8512) WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027881#comment-15027881
 ] 

Hudson commented on HDFS-8512:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #1456 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1456/])
HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with (xyao: rev 
e3d673901b396cf5bbede5ed6f607ce68301ec0a)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java


> WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info
> -
>
> Key: HDFS-8512
> URL: https://issues.apache.org/jira/browse/HDFS-8512
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Sumana Sathish
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8512.00.patch, HDFS-8512.01.patch
>
>
> Storage type inside LocatedBlock object is not fully exposed for GETFILESTATUS
> {code}
> $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GETFILESTATUS";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785853423&s=W4O5kKiYHmzzey4h7I9J9eL9EMY=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:13 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"FileStatus":{"accessTime":1432683737985,"blockSize":134217728,"childrenNum":0,"fileId":16405,"group":"hadoop","length":150318178,"modificationTime":1432683738427,"owner":"xyao","pathSuffix":"","permission":"644","replication":1,"storagePolicy":7,"type":"FILE"}}
>  $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GET_BLOCK_LOCATIONS&offset=0&length=150318178";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785895031&s=TUiaNsCrARAPKz6xrddoQ1eHOXA=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:55 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"LocatedBlocks":{"fileLength":150318178,"isLastBlockComplete":true,"isUnderConstruction":false,"lastLocatedBlock":{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":134217728},"locatedBlocks":[{"block":{"blockId":1073741846,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1022,"numBytes":134217728},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":0},{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default

[jira] [Commented] (HDFS-9467) Fix data race accessing writeLockHeldTimeStamp in FSNamesystem

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027883#comment-15027883
 ] 

Hudson commented on HDFS-9467:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #1456 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1456/])
HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in (jing9: rev 
e556c35b0596700f9ec9d0a51cf5027259d531b5)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


> Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
> --
>
> Key: HDFS-9467
> URL: https://issues.apache.org/jira/browse/HDFS-9467
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-9467.000.patch, HDFS-9467.001.patch
>
>
> This is a followup of [HDFS-9145].
> Actually the value of {{writeLockInterval}} should be captured within the 
> lock. The current code has a race on {{writeLockHeldTimeStamp}}. Thanks to 
> [~jingzhao] for reporting this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027884#comment-15027884
 ] 

Hudson commented on HDFS-9407:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #1456 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1456/])
HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by (shv: 
rev fc799ab16cc62b04fef5a416af30e4ae845dd7a5)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9451) Clean up depreated umasks and related unit tests

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027882#comment-15027882
 ] 

Hudson commented on HDFS-9451:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #1456 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1456/])
HDFS-9451. Clean up depreated umasks and related unit tests. Contributed 
(wheat9: rev b21dffb1fec376784810af89cfc9cc05e5f781ce)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/permission/TestFsPermission.java


> Clean up depreated umasks and related unit tests
> 
>
> Key: HDFS-9451
> URL: https://issues.apache.org/jira/browse/HDFS-9451
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 3.0.0
>
> Attachments: HDFS-9451.001.patch, HDFS-9451.002.patch, 
> HDFS-9451.003.patch, HDFS-9451.004.patch
>
>
> I noticed this test failed consistently since yesterday. The first failed 
> jenkins job is 
> https://builds.apache.org/job/Hadoop-common-trunk-Java8/723/changes, and from 
> the change log:
> {noformat}
> Changes:
> [wheat9] HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter 
> Su.
> [wheat9] HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
> [wheat9] HADOOP-12467. Respect user-defined JAVA_LIBRARY_PATH in Windows 
> Hadoop
> [wheat9] HDFS-8914. Document HA support in the HDFS HdfsDesign.md. 
> Contributed by
> [wheat9] HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai 
> Zheng.
> [wheat9] HDFS-7796. Include X-editable for slick contenteditable fields in the
> [wheat9] HDFS-3302. Review and improve HDFS trash documentation. Contributed 
> by
> [wheat9] HADOOP-12294. Remove the support of the deprecated dfs.umask.
> {noformat}
> HADOOP-12294 looks to be the most likely cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers

2015-11-25 Thread Zhe Zhang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-9079:

Attachment: HDFS-9079.14.patch

Updating patch to fix a trivial NPE bug and the block token test failure.

So the {{TestDFSStripedOutputStreamWithFailurexxx}} tests sometimes time out 
during setup and tearDown. We should think of a way to reduce this kind of 
false alarms. I turned off the random checking in {{TestBase}} and ran all 
tests locally, only got timeouts in setup and tearDown.

> Erasure coding: preallocate multiple generation stamps and serialize updates 
> from data streamers
> 
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-9079-HDFS-7285.00.patch, HDFS-9079.01.patch, 
> HDFS-9079.02.patch, HDFS-9079.03.patch, HDFS-9079.04.patch, 
> HDFS-9079.05.patch, HDFS-9079.06.patch, HDFS-9079.07.patch, 
> HDFS-9079.08.patch, HDFS-9079.09.patch, HDFS-9079.10.patch, 
> HDFS-9079.11.patch, HDFS-9079.12.patch, HDFS-9079.13.patch, HDFS-9079.14.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4) 
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6) 
> Updates block on NN
> {code}
> With multiple streamer threads run in parallel, we need to correctly handle a 
> large number of possible combinations of interleaved thread events. For 
> example, {{streamer_B}} starts step 2 in between events {{streamer_A.2}} and 
> {{streamer_A.3}}.
> HDFS-9040 moves steps 1, 2, 3, 6 from streamer to {{DFSStripedOutputStream}}. 
> This JIRA proposes some further optimizations based on HDFS-9040:
> # We can preallocate GS when NN creates a new striped block group 
> ({{FSN#createNewBlock}}). For each new striped block group we can reserve 
> {{NUM_PARITY_BLOCKS}} GS's. If more than {{NUM_PARITY_BLOCKS}} errors have 
> happened we shouldn't try to further recover anyway.
> # We can use a dedicated event processor to offload the error handling logic 
> from {{DFSStripedOutputStream}}, which is not a long running daemon.
> # We can limit the lifespan of a streamer to be a single block. A streamer 
> ends either after finishing the current block or when encountering a DN 
> failure.
> With the proposed change, a {{StripedDataStreamer}}'s flow becomes:
> {code}
> 1) Finds DN error => 2) Notify coordinator (async, not waiting for response) 
> => terminates
> 1) Finds external error => 2) Applies new GS to DN (createBlockOutputStream) 
> => 3) Ack from DN => 4) Notify coordinator (async, not waiting for response)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9214) Reconfigure DN concurrent move on the fly

2015-11-25 Thread Xiaobing Zhou (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-9214:

Attachment: HDFS-9214.003.patch

Thanks for [~arpitagarwal]'s reviews. I posted v003 patch to address the 
issuess, kindly review!

> Reconfigure DN concurrent move on the fly
> -
>
> Key: HDFS-9214
> URL: https://issues.apache.org/jira/browse/HDFS-9214
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-9214.001.patch, HDFS-9214.002.patch, 
> HDFS-9214.003.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.balance.max.concurrent.moves
> {code} without restarting DN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9426) Rollingupgrade finalization is not backward compatible

2015-11-25 Thread Vinod Kumar Vavilapalli (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027859#comment-15027859
 ] 

Vinod Kumar Vavilapalli commented on HDFS-9426:
---

[~vinayrpet], can you look at [~kihwal]'s patch and get it reviewed / 
committed? I am holding off 2.7.2 for this one, please let me know if this 
needs more time. Thanks.

> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in 
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade 
> because it's not null.
> NN might have to check the DN version and return different 
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but 
> unfortunately did not address the semantics of the heartbeat response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9273) ACLs on root directory may be lost after NN restart

2015-11-25 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027857#comment-15027857
 ] 

Chris Nauroth commented on HDFS-9273:
-

Yes, it's long-standing, so good to bring it into the maintenance lines.  
Thanks for identifying it, Andrew.

> ACLs on root directory may be lost after NN restart
> ---
>
> Key: HDFS-9273
> URL: https://issues.apache.org/jira/browse/HDFS-9273
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.7.1
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HDFS-9273.001.patch, HDFS-9273.002.patch
>
>
> After restarting namenode, the ACLs on the root directory ("/") may be lost 
> if it's rolled over to fsimage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6363) Improve concurrency while checking inclusion and exclusion of datanodes

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027856#comment-15027856
 ] 

Hadoop QA commented on HDFS-6363:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 32s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s 
{color} | {color:red} Patch generated 4 new checkstyle issues in 
hadoop-hdfs-project/hadoop-hdfs (total was 5, now 9). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 31s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 47s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s 
{color} | {color:red} Patch generated 57 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 190m 5s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
|   | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
|   | hadoop.hdfs.qjournal.client.TestQJMWithFaults |
|   | hadoop.hdfs.TestReplication |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
|   | hadoop.hdfs.server.datanode.TestDirectoryScanner |
|   | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| JDK v1.8.0_66 Timed out junit tests | 
org.apache.hadoop.hdfs.

[jira] [Commented] (HDFS-9445) Deadlock in datanode

2015-11-25 Thread Vinod Kumar Vavilapalli (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027854#comment-15027854
 ] 

Vinod Kumar Vavilapalli commented on HDFS-9445:
---

[~kihwal] / [~walter.k.su], I am holding off 2.7.2 for this one, please let me 
know if this needs more time. Thanks.

> Deadlock in datanode
> 
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Assignee: Walter Su
>Priority: Blocker
> Attachments: HDFS-9445.00.patch
>
>
> {noformat}
> Found one Java-level deadlock:
> =
> "DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block 
> BP-x:blk_123_456]":
>   waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
>   which is held by "Thread-565"
> "Thread-565":
>   waiting for ownable synchronizer 0xd55613c8, (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "DataNode: heartbeating to my-nn:8020"
> "DataNode: heartbeating to my-nn:8020":
>   waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
>   which is held by "Thread-565"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9273) ACLs on root directory may be lost after NN restart

2015-11-25 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027846#comment-15027846
 ] 

Andrew Wang commented on HDFS-9273:
---

FWIW I think it's a long-standing bug.

> ACLs on root directory may be lost after NN restart
> ---
>
> Key: HDFS-9273
> URL: https://issues.apache.org/jira/browse/HDFS-9273
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.7.1
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HDFS-9273.001.patch, HDFS-9273.002.patch
>
>
> After restarting namenode, the ACLs on the root directory ("/") may be lost 
> if it's rolled over to fsimage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9273) ACLs on root directory may be lost after NN restart

2015-11-25 Thread Vinod Kumar Vavilapalli (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated HDFS-9273:
--
Priority: Critical  (was: Major)

> ACLs on root directory may be lost after NN restart
> ---
>
> Key: HDFS-9273
> URL: https://issues.apache.org/jira/browse/HDFS-9273
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.7.1
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HDFS-9273.001.patch, HDFS-9273.002.patch
>
>
> After restarting namenode, the ACLs on the root directory ("/") may be lost 
> if it's rolled over to fsimage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9273) ACLs on root directory may be lost after NN restart

2015-11-25 Thread Vinod Kumar Vavilapalli (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated HDFS-9273:
--
Target Version/s: 2.7.2, 2.6.3

bq. Should this fix be included in the 2.6 / 2.7 maintenance releases? Losing 
ACL on restart seems pretty serious.
Tentatively adding for 2.6.3 and 2.7.2.

[~xiaochen], [~cnauroth], know if this is this a long standing bug or broken 
recently?

> ACLs on root directory may be lost after NN restart
> ---
>
> Key: HDFS-9273
> URL: https://issues.apache.org/jira/browse/HDFS-9273
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.7.1
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9273.001.patch, HDFS-9273.002.patch
>
>
> After restarting namenode, the ACLs on the root directory ("/") may be lost 
> if it's rolled over to fsimage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8512) WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027840#comment-15027840
 ] 

Hudson commented on HDFS-8512:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #724 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/724/])
HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with (xyao: rev 
e3d673901b396cf5bbede5ed6f607ce68301ec0a)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java


> WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info
> -
>
> Key: HDFS-8512
> URL: https://issues.apache.org/jira/browse/HDFS-8512
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Sumana Sathish
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8512.00.patch, HDFS-8512.01.patch
>
>
> Storage type inside LocatedBlock object is not fully exposed for GETFILESTATUS
> {code}
> $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GETFILESTATUS";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785853423&s=W4O5kKiYHmzzey4h7I9J9eL9EMY=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:13 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"FileStatus":{"accessTime":1432683737985,"blockSize":134217728,"childrenNum":0,"fileId":16405,"group":"hadoop","length":150318178,"modificationTime":1432683738427,"owner":"xyao","pathSuffix":"","permission":"644","replication":1,"storagePolicy":7,"type":"FILE"}}
>  $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GET_BLOCK_LOCATIONS&offset=0&length=150318178";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785895031&s=TUiaNsCrARAPKz6xrddoQ1eHOXA=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:55 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"LocatedBlocks":{"fileLength":150318178,"isLastBlockComplete":true,"isUnderConstruction":false,"lastLocatedBlock":{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":134217728},"locatedBlocks":[{"block":{"blockId":1073741846,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1022,"numBytes":134217728},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":0},{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networ

[jira] [Commented] (HDFS-9451) Clean up depreated umasks and related unit tests

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027841#comment-15027841
 ] 

Hudson commented on HDFS-9451:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #724 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/724/])
HDFS-9451. Clean up depreated umasks and related unit tests. Contributed 
(wheat9: rev b21dffb1fec376784810af89cfc9cc05e5f781ce)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/permission/TestFsPermission.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Clean up depreated umasks and related unit tests
> 
>
> Key: HDFS-9451
> URL: https://issues.apache.org/jira/browse/HDFS-9451
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 3.0.0
>
> Attachments: HDFS-9451.001.patch, HDFS-9451.002.patch, 
> HDFS-9451.003.patch, HDFS-9451.004.patch
>
>
> I noticed this test failed consistently since yesterday. The first failed 
> jenkins job is 
> https://builds.apache.org/job/Hadoop-common-trunk-Java8/723/changes, and from 
> the change log:
> {noformat}
> Changes:
> [wheat9] HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter 
> Su.
> [wheat9] HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
> [wheat9] HADOOP-12467. Respect user-defined JAVA_LIBRARY_PATH in Windows 
> Hadoop
> [wheat9] HDFS-8914. Document HA support in the HDFS HdfsDesign.md. 
> Contributed by
> [wheat9] HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai 
> Zheng.
> [wheat9] HDFS-7796. Include X-editable for slick contenteditable fields in the
> [wheat9] HDFS-3302. Review and improve HDFS trash documentation. Contributed 
> by
> [wheat9] HADOOP-12294. Remove the support of the deprecated dfs.umask.
> {noformat}
> HADOOP-12294 looks to be the most likely cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027834#comment-15027834
 ] 

Mingliang Liu commented on HDFS-9457:
-

Thanks for updating the patch.

Please refer to page at: http://www.slf4j.org/faq.html#paramException

Please address the logging format only in the patch. I see little value in 
changes here like:
{code}
-  public void reconfigurePropertyImpl(String property, String newVal)
+  public void reconfigurePropertyImpl(final String property, final String 
newVal)
{code}

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9273) ACLs on root directory may be lost after NN restart

2015-11-25 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027833#comment-15027833
 ] 

Andrew Wang commented on HDFS-9273:
---

Should this fix be included in the 2.6 / 2.7 maintenance releases? Losing ACL 
on restart seems pretty serious. Ping [~vinodkv] and [~sjlee0] too.

> ACLs on root directory may be lost after NN restart
> ---
>
> Key: HDFS-9273
> URL: https://issues.apache.org/jira/browse/HDFS-9273
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.7.1
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9273.001.patch, HDFS-9273.002.patch
>
>
> After restarting namenode, the ACLs on the root directory ("/") may be lost 
> if it's rolled over to fsimage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8512) WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027828#comment-15027828
 ] 

Hudson commented on HDFS-8512:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #735 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/735/])
HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with (xyao: rev 
e3d673901b396cf5bbede5ed6f607ce68301ec0a)
* 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info
> -
>
> Key: HDFS-8512
> URL: https://issues.apache.org/jira/browse/HDFS-8512
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Sumana Sathish
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8512.00.patch, HDFS-8512.01.patch
>
>
> Storage type inside LocatedBlock object is not fully exposed for GETFILESTATUS
> {code}
> $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GETFILESTATUS";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785853423&s=W4O5kKiYHmzzey4h7I9J9eL9EMY=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:13 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"FileStatus":{"accessTime":1432683737985,"blockSize":134217728,"childrenNum":0,"fileId":16405,"group":"hadoop","length":150318178,"modificationTime":1432683738427,"owner":"xyao","pathSuffix":"","permission":"644","replication":1,"storagePolicy":7,"type":"FILE"}}
>  $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GET_BLOCK_LOCATIONS&offset=0&length=150318178";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785895031&s=TUiaNsCrARAPKz6xrddoQ1eHOXA=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:55 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"LocatedBlocks":{"fileLength":150318178,"isLastBlockComplete":true,"isUnderConstruction":false,"lastLocatedBlock":{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":134217728},"locatedBlocks":[{"block":{"blockId":1073741846,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1022,"numBytes":134217728},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":0},{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation"

[jira] [Commented] (HDFS-9451) Clean up depreated umasks and related unit tests

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027829#comment-15027829
 ] 

Hudson commented on HDFS-9451:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #735 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/735/])
HDFS-9451. Clean up depreated umasks and related unit tests. Contributed 
(wheat9: rev b21dffb1fec376784810af89cfc9cc05e5f781ce)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/permission/TestFsPermission.java


> Clean up depreated umasks and related unit tests
> 
>
> Key: HDFS-9451
> URL: https://issues.apache.org/jira/browse/HDFS-9451
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 3.0.0
>
> Attachments: HDFS-9451.001.patch, HDFS-9451.002.patch, 
> HDFS-9451.003.patch, HDFS-9451.004.patch
>
>
> I noticed this test failed consistently since yesterday. The first failed 
> jenkins job is 
> https://builds.apache.org/job/Hadoop-common-trunk-Java8/723/changes, and from 
> the change log:
> {noformat}
> Changes:
> [wheat9] HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter 
> Su.
> [wheat9] HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory.
> [wheat9] HADOOP-12467. Respect user-defined JAVA_LIBRARY_PATH in Windows 
> Hadoop
> [wheat9] HDFS-8914. Document HA support in the HDFS HdfsDesign.md. 
> Contributed by
> [wheat9] HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai 
> Zheng.
> [wheat9] HDFS-7796. Include X-editable for slick contenteditable fields in the
> [wheat9] HDFS-3302. Review and improve HDFS trash documentation. Contributed 
> by
> [wheat9] HADOOP-12294. Remove the support of the deprecated dfs.umask.
> {noformat}
> HADOOP-12294 looks to be the most likely cause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027805#comment-15027805
 ] 

BELUGA BEHR commented on HDFS-9457:
---

I updated to include all areas and to make use of SLF4J place holders.

One thing I noticed was inconsistency in reporting exceptions.  What do you 
think should be done? I will apply in a different ticket.

{code}
LOG.warn("Received exception in Datanode#join: {}", ex);

v.s.

LOG.error("Exception in secureMain", e);
{code}

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9210) Fix some misuse of %n in VolumeScanner#printStats

2015-11-25 Thread Daniel Templeton (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027804#comment-15027804
 ] 

Daniel Templeton commented on HDFS-9210:


They are, but {{line separator()}} doesn't require string parsing to
interpret.



> Fix some misuse of %n in VolumeScanner#printStats
> -
>
> Key: HDFS-9210
> URL: https://issues.apache.org/jira/browse/HDFS-9210
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.7.1
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9210.00.patch, HDFS-9210.01.patch, 
> HDFS-9210.02.patch
>
>
> Found 2 extra "%n" in the VolumeScanner report and lines not well formatted  
> below. This JIRA is opened to fix the format issue.
> {code}
> Block scanner information for volume DS-93fb2503-de00-4f98-a8bc-c2bc13b8f0f7 
> with base path /hadoop/hdfs/data%nBytes verified in last hour   : 
> 136882014
> Blocks scanned in current period  :   
>   5
> Blocks scanned since restart  :   
>   5
> Block pool scans since restart:   
>   0
> Block scan errors since restart   :   
>   0
> Hours until next block pool scan  :   
> 476.000
> Last block scanned: 
> BP-1792969149-192.168.70.101-1444150984999:blk_1073742088_1274
> More blocks to scan in period :   
>   false
> %n
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HDFS-9457:
--
Attachment: logging.patch

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: logging.patch
>
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9457) Datanode Logging Improvement

2015-11-25 Thread BELUGA BEHR (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HDFS-9457:
--
Attachment: (was: Logging.diff)

> Datanode Logging Improvement
> 
>
> Key: HDFS-9457
> URL: https://issues.apache.org/jira/browse/HDFS-9457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0, 2.7.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
>
> Please accept my patch for some minor clean-up of logging. Patch is against 
> 3.0.0 but applies to previous versions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9210) Fix some misuse of %n in VolumeScanner#printStats

2015-11-25 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027787#comment-15027787
 ] 

Andrew Wang commented on HDFS-9210:
---

LGTM +1, though aren't {{lineSeparator}} and {{"%n"}} the same? Formatter 
javadoc under "Line Separator" says {{n}} is same as 
{{System.getProperty("line.separator")}}: 
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/Formatter.html#syntax

> Fix some misuse of %n in VolumeScanner#printStats
> -
>
> Key: HDFS-9210
> URL: https://issues.apache.org/jira/browse/HDFS-9210
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.7.1
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9210.00.patch, HDFS-9210.01.patch, 
> HDFS-9210.02.patch
>
>
> Found 2 extra "%n" in the VolumeScanner report and lines not well formatted  
> below. This JIRA is opened to fix the format issue.
> {code}
> Block scanner information for volume DS-93fb2503-de00-4f98-a8bc-c2bc13b8f0f7 
> with base path /hadoop/hdfs/data%nBytes verified in last hour   : 
> 136882014
> Blocks scanned in current period  :   
>   5
> Blocks scanned since restart  :   
>   5
> Block pool scans since restart:   
>   0
> Block scan errors since restart   :   
>   0
> Hours until next block pool scan  :   
> 476.000
> Last block scanned: 
> BP-1792969149-192.168.70.101-1444150984999:blk_1073742088_1274
> More blocks to scan in period :   
>   false
> %n
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9459) hadoop-hdfs-native-client fails test build on Windows after transition to ctest.

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027760#comment-15027760
 ] 

Hudson commented on HDFS-9459:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2666 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2666/])
HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after (wheat9: 
rev 95d5227c75f430da7c77847f31734b34b36157d2)
* hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> hadoop-hdfs-native-client fails test build on Windows after transition to 
> ctest.
> 
>
> Key: HDFS-9459
> URL: https://issues.apache.org/jira/browse/HDFS-9459
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build, test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Blocker
> Fix For: 2.8.0
>
>
> HDFS-9369 transitioned to usage of {{ctest}} for running the HDFS native 
> tests.  This broke the {{mvn test}} build on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9210) Fix some misuse of %n in VolumeScanner#printStats

2015-11-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027736#comment-15027736
 ] 

Hadoop QA commented on HDFS-9210:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s 
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 10s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 11s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s 
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 13s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.security.TestPermission |
|   | hadoop.hdfs.server.namenode.TestCacheDirectives |
| JDK v1.7.0_85 Failed junit tests | hadoop.security.TestPermission |
|   | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| JDK v1.7.0_85 Timed out junit tests | 
org.apache.hadoop.hdfs.web.TestWebHDFSAcl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachm

[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027728#comment-15027728
 ] 

Hudson commented on HDFS-9407:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8893 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8893/])
HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by (shv: 
rev fc799ab16cc62b04fef5a416af30e4ae845dd7a5)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java


> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
>  https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8512) WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info

2015-11-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027726#comment-15027726
 ] 

Hudson commented on HDFS-8512:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8893 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8893/])
HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with (xyao: rev 
e3d673901b396cf5bbede5ed6f607ce68301ec0a)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info
> -
>
> Key: HDFS-8512
> URL: https://issues.apache.org/jira/browse/HDFS-8512
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Sumana Sathish
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HDFS-8512.00.patch, HDFS-8512.01.patch
>
>
> Storage type inside LocatedBlock object is not fully exposed for GETFILESTATUS
> {code}
> $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GETFILESTATUS";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:13 GMT
> Date: Wed, 27 May 2015 18:04:13 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785853423&s=W4O5kKiYHmzzey4h7I9J9eL9EMY=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:13 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"FileStatus":{"accessTime":1432683737985,"blockSize":134217728,"childrenNum":0,"fileId":16405,"group":"hadoop","length":150318178,"modificationTime":1432683738427,"owner":"xyao","pathSuffix":"","permission":"644","replication":1,"storagePolicy":7,"type":"FILE"}}
>  $ curl -i 
> "http://127.0.0.1:50070/webhdfs/v1/HOT/FILE1?user.name=xyao&op=GET_BLOCK_LOCATIONS&offset=0&length=150318178";
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Expires: Wed, 27 May 2015 18:04:55 GMT
> Date: Wed, 27 May 2015 18:04:55 GMT
> Pragma: no-cache
> Content-Type: application/json
> Set-Cookie: 
> hadoop.auth="u=xyao&p=xyao&t=simple&e=1432785895031&s=TUiaNsCrARAPKz6xrddoQ1eHOXA=";
>  Path=/; Expires=Thu, 28-May-2015 04:04:55 GMT; HttpOnly
> Transfer-Encoding: chunked
> Server: Jetty(6.1.26)
>  
> {"LocatedBlocks":{"fileLength":150318178,"isLastBlockComplete":true,"isUnderConstruction":false,"lastLocatedBlock":{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":134217728},"locatedBlocks":[{"block":{"blockId":1073741846,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1022,"numBytes":134217728},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/default-rack","remaining":782138327040,"storageID":"49a30d0f-99f8-4b87-b986-502fe926271a","xceiverCount":1,"xferPort":50010}],"startOffset":0},{"block":{"blockId":1073741847,"blockPoolId":"BP-474445704-192.168.70.1-1432674221011","generationStamp":1023,"numBytes":16100450},"blockToken":{"urlString":"AA"},"cachedLocations":[],"isCorrupt":false,"locations":[{"adminState":"NORMAL","blockPoolUsed":300670976,"cacheCapacity":0,"cacheUsed":0,"capacity":1996329943040,"dfsUsed":300670976,"hostName":"192.168.70.1","infoPort":50075,"infoSecurePort":0,"ipAddr":"192.168.70.1","ipcPort":50020,"lastUpdate":1432749892058,"lastUpdateMonotonic":1432749892058,"name":"192.168.70.1:50010","networkLocation":"/def

1 2 >

1 - 100 of 181 matches

Mail list logo