[jira] [Commented] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails
[
https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712581#comment-14712581
]
Li Bo commented on HDFS-8704:
-
Thanks Zhe’s review. The handle of multiple block groups for datanode failure
is complex, let me give some detailed explanation.
Suppose packets {{p1,p2,p3,p4}} belong to the first block and {{p5,p6,p7,p8}}
belong to the next one. Now streamer fails to send packet p2 and set its
{{streamerClosed}} as true. Its {{dataQueue}} still contains {{p3,p4,p5}}.
Then {{DFSStripedOutputStream}} will fail to write data of packet p6, and set
this streamer as failed. After that {{DFSStripedOutputStream}} will not write
{{p7,p8}} to that streamer any more. Maybe the streamer is allocated a good
datanode and {{p5,p6,p7,p8}} should be successfully written to that datanode.
We only see {{StripedDataStreamer#setFailed(true)}}, where and when to mark the
streamer as not failed?
Due to the asynchronization of {{StripedDataStreamer}} and
{{DFSStripedOutputStream}}, it’s more reasonable to let
{{DFSStripedOutputStream}} be unware of the status of streamers unless there’re
no enough streamers. When a streamer fails to connect datanode or write some
packet, it removes the following trivial packets of current block in its
{{dataQueue}} and waiting for next block to be allocated to it.
1. Please see above explanation
2. If just writing a file smaller than a block group, you’ll not find the
problem of {{setFailed}}. The failed status should be marked and erased by data
streamer itself, not the outputstream.
3. When streamer fails, it should do some extra work to prepare for the
next block. It’s very difficult to achieve this if not overriding {{run}}. I
will give some description in JIRA summary later. When you fail to send a
packet, the following packets belonging to the same block are trivial packets,
you just need to remove them from the {{dataQueue}}.
4. I will update the patch after HDFS-8838 committed.
5. For small issues: If streamer fails at some block, it still sends to
coordinator an end block with a negative {{numBytes}}.
> Erasure Coding: client fails to write large file when one datanode fails
>
>
> Key: HDFS-8704
> URL: https://issues.apache.org/jira/browse/HDFS-8704
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Li Bo
>Assignee: Li Bo
> Attachments: HDFS-8704-000.patch, HDFS-8704-HDFS-7285-002.patch,
> HDFS-8704-HDFS-7285-003.patch, HDFS-8704-HDFS-7285-004.patch,
> HDFS-8704-HDFS-7285-005.patch
>
>
> I test current code on a 5-node cluster using RS(3,2). When a datanode is
> corrupt, client succeeds to write a file smaller than a block group but fails
> to write a large one. {{TestDFSStripeOutputStreamWithFailure}} only tests
> files smaller than a block group, this jira will add more test situations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712563#comment-14712563
]
Walter Su commented on HDFS-8909:
-
TestWriteStripedFileWithFailure, TestFileTruncate are not related.
TestFileAppend4 is not related also, it's an issue left over by history, just
didn't show up before.
{code}
static LocatedBlock makeLocatedBlock(FSNamesystem fsn, BlockInfo blk,
//FSDirWriteFileOp.java
DatanodeStorageInfo[] locs, long offset) throws IOException {
LocatedBlock lBlk = BlockManager.newLocatedBlock(fsn.getExtendedBlock(blk),
blk, locs, offset);
{code}
These code leaks BlockInfo to LocatedBlock and causes the TestFileAppend4
fails. The test case injects namenodeRpcServer to dfsclient, so the client will
get the same BlockInfo object. It's not a problem in other cases because
ExtendBlockProto has fewer fields than BlockInfo.
LocatedBlock belongs to {{hadooop-hdfs-client}} project, so BlockInfo should
not appear in LocatedBlock.
I'll appreciate if you could fix this together. Thanks
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909-HDFS-7285.003.patch, HDFS-8909.000.patch,
> HDFS-8909.001.patch, HDFS-8909.002.patch, HDFS-8909.003.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8956) Not able to start Datanode
sreelakshmi created HDFS-8956: - Summary: Not able to start Datanode Key: HDFS-8956 URL: https://issues.apache.org/jira/browse/HDFS-8956 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.7.0 Environment: Centos Reporter: sreelakshmi Data node service is not started on one of the data nodes, "java.net.bind exception" is thrown. Verified that ports 50010,50070 and 50075 are not in use by any other application. 15/08/26 01:50:15 INFO http.HttpServer2: HttpServer.start() threw a non Bind IOException java.net.BindException: Port in use: localhost:0 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856) at org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:779) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1134) at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:434) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2404) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2291) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2338) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2515) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2539) Caused by: java.net.BindException: Cannot assign requested address -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712541#comment-14712541
]
Zhe Zhang commented on HDFS-8853:
-
Thanks for the update [~andreina]. Since there are many failures (likely to be
unrelated though), I just triggered Jenkins again.
> Erasure Coding: Provide ECSchema validation when creating ECZone
>
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Attachments: HDFS-8853-HDFS-7285-01.patch,
> HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-merge-02.patch,
> HDFS-8853-HDFS-7285-merge-03.patch, HDFS-8853-HDFS-7285-merge-04.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712520#comment-14712520
]
J.Andreina commented on HDFS-8853:
--
Testcase failures are not related to this patch. Please review.
> Erasure Coding: Provide ECSchema validation when creating ECZone
>
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Attachments: HDFS-8853-HDFS-7285-01.patch,
> HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-merge-02.patch,
> HDFS-8853-HDFS-7285-merge-03.patch, HDFS-8853-HDFS-7285-merge-04.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not balance block in RAM_DISK use lazy_persist policy on datanode local copy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Description: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! when I use other machine without datanode installed , it is worked fine all DN tmpfs is fill enough Regards was: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regards > Not balance block in RAM_DISK use lazy_persist policy on datanode local copy > - > > Key: HDFS-8954 > URL: https://issues.apache.org/jira/browse/HDFS-8954 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Xu Chen >Priority: Critical > > I was try to use Heterogeneous Storage feature and I config RAM_DISK volume > for each DN size of 20g , and set one directory policy of lazy_persist at > HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , > it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to > look tmpfs usage it only my execute command DN fill enough the tmpfs , other > DN tmpfs is empty ! > when I use other machine without datanode installed , it is worked fine all > DN tmpfs is fill enough > Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not balance block in RAM_DISK use lazy_persist policy on datanode local copy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Summary: Not balance block in RAM_DISK use lazy_persist policy on datanode local copy (was: Not all the one block in RAM_DISK use lazy_persist policy) > Not balance block in RAM_DISK use lazy_persist policy on datanode local copy > - > > Key: HDFS-8954 > URL: https://issues.apache.org/jira/browse/HDFS-8954 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Xu Chen >Priority: Critical > > I was try to use Heterogeneous Storage feature and I config RAM_DISK volume > for each DN size of 20g , and set one directory policy of lazy_persist at > HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , > it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to > look tmpfs usage it only my execute command DN fill enough the tmpfs , other > DN tmpfs is empty ! > that a bug ? or my config issue? > Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Component/s: (was: balancer & mover) datanode > Not all the one block in RAM_DISK use lazy_persist policy > - > > Key: HDFS-8954 > URL: https://issues.apache.org/jira/browse/HDFS-8954 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Xu Chen >Priority: Critical > > I was try to use Heterogeneous Storage feature and I config RAM_DISK volume > for each DN size of 20g , and set one directory policy of lazy_persist at > HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , > it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to > look tmpfs usage it only my execute command DN fill enough the tmpfs , other > DN tmpfs is empty ! > that a bug ? or my config issue? > Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8581) count cmd calculate wrong when huge files exist in one folder
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8581: - Attachment: HDFS-8581.3.patch Updated the patch. Please review. > count cmd calculate wrong when huge files exist in one folder > - > > Key: HDFS-8581 > URL: https://issues.apache.org/jira/browse/HDFS-8581 > Project: Hadoop HDFS > Issue Type: Bug > Components: HDFS >Reporter: tongshiquan >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch, HDFS-8581.3.patch > > > If one directory such as "/result" exists about 20 files, then when > execute "hdfs dfs -count /", the result will go wrong. For all directories > whose name after "/result", file num will not be included. > My cluster see as below, "/result_1433858936" is the directory exist huge > files, and files in "/sparkJobHistory", "/tmp", "/user" are not included > vm-221:/export1/BigData/current # hdfs dfs -ls / > 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. > Found 9 items > -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 > /PRE_CREATE_DIR.SUCCESS > drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume > drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase > drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt > drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history > drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 > /result_1433858936 > drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory > drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp > drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count / > 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. > 1043 171536 1756375688 / > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS > 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. >01 0 /PRE_CREATE_DIR.SUCCESS > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /flume > 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /flume > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hbase > 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. > 36 18 14807 /hbase > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hyt > 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /hyt > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /mr-history > 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. >30 0 /mr-history > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 > 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. > 1001 171517 1756360881 /result_1433858936 > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory > 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. >13 21785 /sparkJobHistory > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /tmp > 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. > 176 35958 /tmp > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /user > 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. > 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[ https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712494#comment-14712494 ] Brandon Li commented on HDFS-4750: -- Overlapped writes happen usually for the first block. For example, a file has 100 bytes. User write this file to NFS mount point. After a while (the 100 bytes still in OS buffer cache), the user appends another 100 bytes. In this case, the NFS client will rewrite the first block (0-199th byte), and thus we see an overlapped write. These kind of case should be well handled by current code (but, still might have bugs there :-( ) Based on your description, the problem seems not the above case. There is one case, we currently have not way to control it on the server side: 1. client sends write 0-99, we cache this write and do the write asynchronously 2. client sends another write 180-299, we cache it too, can't write since there is a hole 3. client sends another write 100-199, we will do the write since it's a sequential write 4. after we finish writing (0-99) and (100-199), we see an overlapped write (180-299) in the cache. This is where you see the error message since we are expecting another sequential write (200-xxx) This kind of overlapped write happens very rarely. In case it happens, we have multiple copies of the same range (180-198 in the above example). The data could be different. When they are different, it could be hard to know which one is really expected by the client. > Support NFSv3 interface to HDFS > --- > > Key: HDFS-4750 > URL: https://issues.apache.org/jira/browse/HDFS-4750 > Project: Hadoop HDFS > Issue Type: New Feature > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HADOOP-NFS-Proposal.pdf, HDFS-4750.patch, nfs-trunk.patch > > > Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless > integration with client’s file system makes it difficult for users and > impossible for some applications to access HDFS. NFS interface support is one > way for HDFS to have such easy integration. > This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS > client, webHDFS and the NFS interface, HDFS will be easier to access and be > able support more applications and use cases. > We will upload the design document and the initial implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[
https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712448#comment-14712448
]
Hadoop QA commented on HDFS-8950:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 43s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 13s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 21s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 8s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 166m 42s | Tests failed in hadoop-hdfs. |
| | | 211m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752359/HDFS-8950.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a4d9acc |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/console |
This message was automatically generated.
> NameNode refresh doesn't remove DataNodes
> -
>
> Key: HDFS-8950
> URL: https://issues.apache.org/jira/browse/HDFS-8950
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 2.8.0
>
> Attachments: HDFS-8950.001.patch, HDFS-8950.002.patch
>
>
> If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN
> refresh, it doesn't remove it actually and the NN UI keeps showing that node.
> It may try to allocate some blocks to that DN as well during an MR job. This
> issue is independent from DN decommission.
> To reproduce:
> 1. Add a DN to dfs_hosts_allow
> 2. Refresh NN
> 3. Start DN. Now NN starts seeing DN.
> 4. Stop DN
> 5. Remove DN from dfs_hosts_allow
> 6. Refresh NN -> NN is still reporting DN as being used by HDFS.
> This is different from decom because there DN is added to exclude list in
> addition to being removed from allowed list, and in that case everything
> works correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Affects Version/s: (was: 3.0.0) 2.6.0 > Support 'hedged' write in DFSClient > --- > > Key: HDFS-8955 > URL: https://issues.apache.org/jira/browse/HDFS-8955 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: bijaya >Assignee: bijaya > > We do have hedged read which serves redundancy on read failures due to bad > sector/patch in disk. We need to have similar feature for hdfs write. This > feature may come with cost but its something to must have for use case which > needs to guarantee write success regardless of degraded disk health. > Defination of degraded disk is highly debatable but this is what I would > define. "Degraded disk is the disk which fails to read and write > intermittently" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Description: We do have hedged read which serves redundancy on read failures due to bad sector/patch in disk. We need to have similar feature for hdfs write. This feature may come with cost but its something to must have for use case which needs to guarantee write success regardless of degraded disk health. Defination of degraded disk is highly debatable but this is what I would define. "Degraded disk is the disk which fails to read and write intermittently" (was: This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509 The quorum read ability should be helpful especially to optimize read outliers we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric). The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items.) > Support 'hedged' write in DFSClient > --- > > Key: HDFS-8955 > URL: https://issues.apache.org/jira/browse/HDFS-8955 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: bijaya >Assignee: bijaya > > We do have hedged read which serves redundancy on read failures due to bad > sector/patch in disk. We need to have similar feature for hdfs write. This > feature may come with cost but its something to must have for use case which > needs to guarantee write success regardless of degraded disk health. > Defination of degraded disk is highly debatable but this is what I would > define. "Degraded disk is the disk which fails to read and write > intermittently" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Fix Version/s: (was: 2.4.0) > Support 'hedged' write in DFSClient > --- > > Key: HDFS-8955 > URL: https://issues.apache.org/jira/browse/HDFS-8955 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.6.0 >Reporter: bijaya >Assignee: bijaya > > We do have hedged read which serves redundancy on read failures due to bad > sector/patch in disk. We need to have similar feature for hdfs write. This > feature may come with cost but its something to must have for use case which > needs to guarantee write success regardless of degraded disk health. > Defination of degraded disk is highly debatable but this is what I would > define. "Degraded disk is the disk which fails to read and write > intermittently" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712445#comment-14712445
]
Hadoop QA commented on HDFS-8951:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 28s | Pre-patch trunk has 6 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 3s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 31s | The applied patch generated
63 new checkstyle issues (total was 16, now 79). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 41s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 27s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 192m 47s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 29s | Tests passed in
hadoop-hdfs-client. |
| | | 243m 54s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752353/HDFS-8951.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a4d9acc |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/console |
This message was automatically generated.
> Move shortcircuit to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
> HDFS-8951.002.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Release Note: If a write from a block is slow, start up another parallel, 'hedged' write against a different set of replica. We need to get different set of replica/data pipeline from NN. We then take the result of which ever write returns first (the outstanding write is cancelled). This 'hedged' write feature will help rein in the outliers, the odd write that takes a long time because it hit a bad patch on the disc, etc. This feature is off by default. To enable this feature, set dfs.client.hedged.write.threadpool.size to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent writes in your client. Then set dfs.client.hedged.write.threshold.millis to the number of milliseconds to wait before starting up a 'hedged' write. For example, if you set this property to 10, then if a write has not returned within 10 milliseconds, we will start up a new read against a different block replica. This feature emits new metrics: + hedgedWriteOps + hedgeWriteOpsWin -- how many times the hedged write 'beat' the original write + hedgedWriteOpsInCurThread -- how many times we went to do a hedged write but we had to run it in the current thread because dfs.client.hedged.write.threadpool.size was at a maximum. was: If a read from a block is slow, start up another parallel, 'hedged' read against a different block replica. We then take the result of which ever read returns first (the outstanding read is cancelled). This 'hedged' read feature will help rein in the outliers, the odd read that takes a long time because it hit a bad patch on the disc, etc. This feature is off by default. To enable this feature, set dfs.client.hedged.read.threadpool.size to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent reads in your client. Then set dfs.client.hedged.read.threshold.millis to the number of milliseconds to wait before starting up a 'hedged' read. For example, if you set this property to 10, then if a read has not returned within 10 milliseconds, we will start up a new read against a different block replica. This feature emits new metrics: + hedgedReadOps + hedgeReadOpsWin -- how many times the hedged read 'beat' the original read + hedgedReadOpsInCurThread -- how many times we went to do a hedged read but we had to run it in the current thread because dfs.client.hedged.read.threadpool.size was at a maximum. > Support 'hedged' write in DFSClient > --- > > Key: HDFS-8955 > URL: https://issues.apache.org/jira/browse/HDFS-8955 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: bijaya >Assignee: bijaya > Fix For: 2.4.0 > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya reassigned HDFS-8955: Assignee: bijaya (was: Liang Xie) > Support 'hedged' write in DFSClient > --- > > Key: HDFS-8955 > URL: https://issues.apache.org/jira/browse/HDFS-8955 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: bijaya >Assignee: bijaya > Fix For: 2.4.0 > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8955) Support 'hedged' write in DFSClient
bijaya created HDFS-8955: Summary: Support 'hedged' write in DFSClient Key: HDFS-8955 URL: https://issues.apache.org/jira/browse/HDFS-8955 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: bijaya Assignee: Liang Xie Fix For: 2.4.0 This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509 The quorum read ability should be helpful especially to optimize read outliers we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric). The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Description: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regards was: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regard > Not all the one block in RAM_DISK use lazy_persist policy > - > > Key: HDFS-8954 > URL: https://issues.apache.org/jira/browse/HDFS-8954 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.7.1 >Reporter: Xu Chen >Priority: Critical > > I was try to use Heterogeneous Storage feature and I config RAM_DISK volume > for each DN size of 20g , and set one directory policy of lazy_persist at > HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , > it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to > look tmpfs usage it only my execute command DN fill enough the tmpfs , other > DN tmpfs is empty ! > that a bug ? or my config issue? > Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Description: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regard was:I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , is that a bug ? or my config issue? > Not all the one block in RAM_DISK use lazy_persist policy > - > > Key: HDFS-8954 > URL: https://issues.apache.org/jira/browse/HDFS-8954 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.7.1 >Reporter: Xu Chen >Priority: Critical > > I was try to use Heterogeneous Storage feature and I config RAM_DISK volume > for each DN size of 20g , and set one directory policy of lazy_persist at > HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , > it is have 593 block , and only 141 block in RAM_DISK , I use command "df" to > look tmpfs usage it only my execute command DN fill enough the tmpfs , other > DN tmpfs is empty ! > that a bug ? or my config issue? > Regard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
Xu Chen created HDFS-8954: - Summary: Not all the one block in RAM_DISK use lazy_persist policy Key: HDFS-8954 URL: https://issues.apache.org/jira/browse/HDFS-8954 Project: Hadoop HDFS Issue Type: Bug Components: balancer & mover Affects Versions: 2.7.1 Reporter: Xu Chen Priority: Critical I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , is that a bug ? or my config issue? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8897) Loadbalancer always exits with : java.io.IOException: Another Balancer is running.. Exiting ...
[
https://issues.apache.org/jira/browse/HDFS-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712382#comment-14712382
]
Xu Chen commented on HDFS-8897:
---
[~Alexandre LINTE] I was encounter this issue too , when I use hdfs balancer
command , did you to do this patch ?
> Loadbalancer always exits with : java.io.IOException: Another Balancer is
> running.. Exiting ...
>
>
> Key: HDFS-8897
> URL: https://issues.apache.org/jira/browse/HDFS-8897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.7.1
> Environment: Centos 6.6
>Reporter: LINTE
>
> When balancer is launched, it should test if there is already a
> /system/balancer.id file in HDFS.
> When the file doesn't exist, the balancer don't want to run :
> 15/08/14 16:35:12 INFO balancer.Balancer: namenodes = [hdfs://sandbox/,
> hdfs://sandbox]
> 15/08/14 16:35:12 INFO balancer.Balancer: parameters =
> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0, max idle iteration
> = 5, number of nodes to be excluded = 0, number of nodes to be included = 0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> 15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> java.io.IOException: Another Balancer is running.. Exiting ...
> Aug 14, 2015 4:35:14 PM Balancing took 2.408 seconds
> Looking at the audit log file when trying to run the balancer, the balancer
> create the /system/balancer.id and then delete it on exiting ...
> 2015-08-14 16:37:45,844 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:45,900 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=create
> src=/system/balancer.id dst=nullperm=hdfs:hadoop:rw-r-
> proto=rpc
> 2015-08-14 16:37:45,919 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,090 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,112 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,117 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=delete
> src=/system/balancer.id dst=nullperm=null proto=rpc
> The error seems to be located in
> org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
> The function checkAndMarkRunning return null even if the /system/balancer.id
> doesn't exist before entering this function; if it exists, then it is deleted
> and the balancer exit with the same error.
>
> private OutputStream checkAndMarkRunning() throws IOException {
> try {
> if (fs.exists(idPath)) {
> // try appending to it so that it will fail fast if another balancer
> is
> // running.
> IOUtils.closeStream(fs.append(idPath));
> fs.delete(idPath, true);
> }
> final FSDataOutputStream fsout = fs.create(idPath, false);
> // mark balancer idPath to be deleted during filesystem closure
> fs.deleteOnExit(idPath);
> if (write2IdFile) {
> fsout.writeBytes(InetAddress.getLocalHost().getHostName());
> fsout.hflush();
> }
> return fsout;
> } catch(RemoteException e) {
>
> if(AlreadyBeingCreatedException.class.getName().equals(e.getClassName())){
> return null;
> } else {
> throw e;
> }
> }
> }
>
> Regards
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8763) Incremental blockreport order may replicate unnecessary block
[
https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712355#comment-14712355
]
Ajith S commented on HDFS-8763:
---
Hi all,
agree with changing {{dfs.namenode.replication.interval}}. But i somehow think
this just reduces the window of such problem to happen rather than eliminate
it. Because at some point of time it might happen that the replication monitor
thread overlaps on close and same issue reoccurs. Please correct me if i am
wrong :)
> Incremental blockreport order may replicate unnecessary block
> -
>
> Key: HDFS-8763
> URL: https://issues.apache.org/jira/browse/HDFS-8763
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: jiangyu
>Assignee: Walter Su
>Priority: Minor
>
> For our cluster, the NameNode is always very busy, so for every incremental
> block report , the contention of lock is heavy.
> The logic of incremental block report is as follow, client send block to dn1
> and dn1 mirror to dn2, dn2 mirror to dn3. After finish this block, all
> datanode will report the newly received block to namenode. In NameNode side,
> all will go to the method processIncrementalBlockReport in BlockManager
> class. But the status of the block reported from dn2,dn3 is RECEIVING_BLOCK,
> for dn1 is RECEIED_BLOCK. It is okay if dn2, dn3 report before dn1(that is
> common), but in some busy environment, it is easy to find dn1 report before
> dn2 or dn3, let’s assume dn2 report first, dn1 report second, and dn3 report
> third.
> So dn1 will addStoredBlock and find the replica of this block is not reach
> the the original number(which is 3), and the block will add to
> neededReplications construction and soon ask some node in pipeline (dn1 or
> dn2)to replica it dn4 . After sometime, dn4 and dn3 all report this block,
> then choose one node to invalidate.
> Here is one log i found in our cluster:
> 2015-07-08 01:05:34,675 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /logs/***_bigdata_spam/logs/application_1435099124107_470749/xx.xx.4.62_45454.tmp.
> BP-1386326728-xx.xx.2.131-1382089338395
> blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-a7c0f8f6-2399-4980-9479-efa08487b7b3:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-c75145a0-ed63-4180-87ee-d48ccaa647c5:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW]]}
> 2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.7.75:50010 is added to
> blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
> size 0
> 2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.4.62:50010 is added to
> blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
> size 0
> 2015-07-08 01:05:35,003 INFO BlockStateChange: BLOCK* ask xx.xx.4.62:50010 to
> replicate blk_3194502674_2121080184 to datanode(s) xx.xx.4.65:50010
> 2015-07-08 01:05:35,403 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.7.73:50010 is added to blk_3194502674_2121080184 size
> 67750
> 2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.4.65:50010 is added to blk_3194502674_2121080184 size
> 67750
> 2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
> blk_3194502674_2121080184 to xx.xx.7.75:50010
> 2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (xx.xx.7.75:50010, blk_3194502674_2121080184) is added to invalidated blocks
> set
> 2015-07-08 01:05:35,852 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask xx.xx.7.75:50010 to delete [blk_3194502674_2121080184,
> blk_3194497594_2121075104]
> Some day, the number of this situation can be 40, that is not good for
> the performance and waste network band.
> Our base version is hadoop 2.4 and i have checked hadoop 2.7.1 didn’t find
> any difference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8945) Update the description about replica placement in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712306#comment-14712306 ] Andrew Wang commented on HDFS-8945: --- Good idea to update this, thanks for working on it. Few comments, apologies but I did not dust off BlockPlacementPolicyDefault for a re-read (exercise for the author :)): * For 4+ replicas, since we've already guaranteed multi-rack with the first 3, I thought the 4th+ are just pure random. The wording makes it sound like we do something more, but not exactly what. * "was added" rather than "are added" * What wins when storage and rack awareness conflict? e.g. I specify ALL_SSD and all the SSD nodes are on the same rack. There are a number of precedence rules in BPP that are not documented like this. If you think documenting this is too confusing, can just delete this bit. > Update the description about replica placement in HDFS Architecture > documentation > - > > Key: HDFS-8945 > URL: https://issues.apache.org/jira/browse/HDFS-8945 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: HDFS-8945.001.patch > > > The description about replica placement should have > * Explanation about storage types and storage policies should be added > * placement policy for replication factor greater than 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712277#comment-14712277 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2235 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2235/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712265#comment-14712265 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #297 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/297/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712233#comment-14712233 ] Hudson commented on HDFS-8846: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2254 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2254/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Attachment: HDFS-8950.002.patch Just noticed an extraneous import in the patch. (Swing?!? WTF?) Removed and reposting patch. > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 2.8.0 > > Attachments: HDFS-8950.001.patch, HDFS-8950.002.patch > > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Fix Version/s: 2.8.0 Status: Patch Available (was: In Progress) Ready for review > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 2.8.0 > > Attachments: HDFS-8950.001.patch > > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Attachment: HDFS-8950.001.patch Once more with feeling. > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-8950.001.patch > > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8951:
Attachment: HDFS-8951.002.patch
Thank you [~wheat9]!
I refined the patch to address the two problems. This latest patch which also
rebases from the trunk branch.
There are some callsites of {{IOUtils::cleanup}} method that can hardly be
reduced to _try-with-resource_ statement. To avoid class name conflict, I
renamed the client-side util class {{IOUtilsClient}}, and added the Apache
copyright header.
Since this jira is for moving the {{shortcircuit}} directory to {{hdfs-client}}
module, we can open new jiras for coding style fixes. Meanwhile, there is no
essential new code and thus no new unit test is added.
> Move shortcircuit to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
> HDFS-8951.002.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8939) Test(S)WebHdfsFileContextMainOperations failing on branch-2
[
https://issues.apache.org/jira/browse/HDFS-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712168#comment-14712168
]
Chris Nauroth commented on HDFS-8939:
-
bq. ...am getting a bit concerned about the backwards compatibility of this
change...
Yes, I agree. Navigating through the twisty history, I can see that there was
concern expressed about this part of the change in the later comments of
HDFS-5321. There was a specific choice to restore the behavior of making
{{getDefaultPort}} respect the configuration as part of HDFS-6632, so I think
we need to preserve that.
FWIW, I don't think this is strictly correct for the contract of
{{getDefaultPort}}. Other implementations hard-code the return value to its
typical default instead of reading configuration. {{DistributedFileSystem}} is
hard-coded to return 8020. {{FTPFileSystem}} hard-codes 21. Sometimes we have
to let compatibility win though. I suppose that's why we're taking the
opportunity change it on trunk.
For these test failures on branch-2, shall we change {{WebHdfs}} and
{{SWebHdfs}} to call {{setConf}} on the wrapped instance? This would require
private static helper methods that do the instantiation + {{setConf}}, just to
satisfy Java's rules about the call to {{super}} being the first line in the
constructor. e.g.:
{code}
super(theUri, createWebHdfsFileSystem(conf), conf, SCHEME, false);
{code}
{code}
private static WebHdfsFileSystem createWebHdfsFileSystem(Configuration conf) {
WebHdfsFileSystem fs = new WebHdfsFileSystem();
fs.setConf(conf);
return fs;
}
{code}
> Test(S)WebHdfsFileContextMainOperations failing on branch-2
> ---
>
> Key: HDFS-8939
> URL: https://issues.apache.org/jira/browse/HDFS-8939
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.8.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 2.8.0
>
> Attachments: HDFS-8939-branch-2.001.patch,
> HDFS-8939-branch-2.002.patch
>
>
> After HDFS-8180, TestWebHdfsFileContextMainOperations and
> TestSWebHdfsFileContextMainOperations are failing with runtime NPEs while
> instantiating the wrapped WebHDFSFileSystems because {{getDefaultPort}} is
> trying to access a conf that was never provided. In the constructor both
> both WebHdfs and SWebhdfs the underlying (S)WebHdfsFileSystems are
> instantiated in the constructor and never have a chance to have their
> {{setConf}} methods called:
> {code} SWebHdfs(URI theUri, Configuration conf)
> throws IOException, URISyntaxException {
> super(theUri, new SWebHdfsFileSystem(), conf, SCHEME, false);
> }r{code}
> The test passes on trunk because HDFS-5321 removed the call to the
> Configuration instance as part of {{getDefaultPort}}. HDFS-5321 was applied
> to branch-2 but reverted in HDFS-6632, so there's a bit of a difference in
> how branch-2 versus trunk handles default values (branch-2 pulls them from
> configs if specified, trunk just returns the hard-coded value from the
> constants file).
> I've fixed this behave like trunk and return just the hard-coded value, which
> causes the test to pass.
> There is no WebHdfsFileSystem that takes a Config, which would be another
> way to fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712167#comment-14712167
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 43s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 54s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 3
new checkstyle issues (total was 153, now 154). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 30s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 1s | Tests failed in hadoop-hdfs. |
| | | 208m 53s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752303/HDFS-6955-03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[
https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712153#comment-14712153
]
Hadoop QA commented on HDFS-8894:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 32s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 18s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 29s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 27s | The applied patch generated 1
new checkstyle issues (total was 150, now 150). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 38s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 17s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 162m 29s | Tests failed in hadoop-hdfs. |
| | | 209m 40s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752300/HDFS-8894-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12125/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12125/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12125/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12125/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12125/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12125/console |
This message was automatically generated.
> Set SO_KEEPALIVE on DN server sockets
> -
>
> Key: HDFS-8894
> URL: https://issues.apache.org/jira/browse/HDFS-8894
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Nathan Roberts
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch
>
>
> SO_KEEPALIVE is not set on things like datastreamer sockets which can cause
> lingering ESTABLISHED sockets when there is a network glitch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712148#comment-14712148
]
Hadoop QA commented on HDFS-8771:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 41s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 18s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 19s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 18s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 160m 37s | Tests failed in hadoop-hdfs. |
| | | 207m 44s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752302/HDFS-8771-02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12124/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12124/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12124/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12124/console |
This message was automatically generated.
> If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
> send another RPC calls to Journalnodes
>
>
> Key: HDFS-8771
> URL: https://issues.apache.org/jira/browse/HDFS-8771
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takuya Fukudome
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8771-01.patch, HDFS-8771-02.patch
>
>
> In our cluster, edits has became huge(about 50GB) accidentally and our
> Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
> 30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
> Namenode couldn't send other RPC calls to Journalnodes because
> {{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
> It will cause namenode shutting down.
> I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
> like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712135#comment-14712135 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #310 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/310/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712126#comment-14712126 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #305 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/305/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712123#comment-14712123 ] Daniel Templeton commented on HDFS-8950: Pulled the patch temporarily while test_patch is running locally. Will repost with the proper naming conventions shortly. > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Attachment: (was: 28531.patch) > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712097#comment-14712097 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1038 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1038/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Attachment: 28531.patch Here's a patch for a unit test and fix. The HostFileManager now excludes any nodes not present in a non-empty includes file from the node report. All tests in the HDFS project pass. > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: 28531.patch > > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712083#comment-14712083
]
Hadoop QA commented on HDFS-8909:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 16s | Pre-patch HDFS-7285 has 4
extant Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 12 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 17s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 28s | The applied patch generated
14 new checkstyle issues (total was 1116, now 1086). |
| {color:green}+1{color} | whitespace | 0m 12s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 5s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 208m 14s | Tests failed in hadoop-hdfs. |
| | | 253m 56s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestWriteStripedFileWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752122/HDFS-8909-HDFS-7285.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 6b6a63b |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/HDFS-7285FindbugsWarningshadoop-hdfs.html
|
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/console |
This message was automatically generated.
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909-HDFS-7285.003.patch, HDFS-8909.000.patch,
> HDFS-8909.001.patch, HDFS-8909.002.patch, HDFS-8909.003.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-8950: - Target Version/s: 2.8.0 > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712075#comment-14712075
]
Hadoop QA commented on HDFS-8951:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 59s | Pre-patch trunk has 6 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 10s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 31s | The applied patch generated
62 new checkstyle issues (total was 16, now 78). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 30s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 22s | Tests passed in hadoop-hdfs.
|
| {color:green}+1{color} | hdfs tests | 0m 29s | Tests passed in
hadoop-hdfs-client. |
| | | 213m 29s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752146/HDFS-8951.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
|
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/console |
This message was automatically generated.
> Move shortcircuit to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712068#comment-14712068
]
Hadoop QA commented on HDFS-8248:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 49s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 7s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 26s | The applied patch generated 3
new checkstyle issues (total was 512, now 510). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 8s | Post-patch findbugs
hadoop-hdfs-project/hadoop-hdfs compilation is broken. |
| {color:green}+1{color} | findbugs | 2m 8s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 0m 22s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 159m 41s | Tests failed in hadoop-hdfs. |
| | | 202m 7s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.fs.TestHdfsNativeCodeLoader |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockId |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752141/HDFS-8248.007.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/console |
This message was automatically generated.
> Store INodeId instead of the INodeFile object in BlockInfoContiguous
>
>
> Key: HDFS-8248
> URL: https://issues.apache.org/jira/browse/HDFS-8248
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
> HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
> HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch
>
>
> Currently the namespace and the block manager are tightly coupled together.
> There are two couplings in terms of implementation:
> 1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
> owns the block, so that the block manager can look up the corresponding file
> when replicating blocks, recovering from pipeline failures, etc.
> 1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
> owns.
> Decoupling the namespace and the block manager allows the BM to be separated
> out from the Java heap or even as a standalone process. This jira proposes to
> remove the first coupling by storing the id of the inode instead of the
> object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712037#comment-14712037 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-trunk-Commit #8349 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8349/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-8950 started by Daniel Templeton. -- > NameNode refresh doesn't remove DataNodes > - > > Key: HDFS-8950 > URL: https://issues.apache.org/jira/browse/HDFS-8950 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN > refresh, it doesn't remove it actually and the NN UI keeps showing that node. > It may try to allocate some blocks to that DN as well during an MR job. This > issue is independent from DN decommission. > To reproduce: > 1. Add a DN to dfs_hosts_allow > 2. Refresh NN > 3. Start DN. Now NN starts seeing DN. > 4. Stop DN > 5. Remove DN from dfs_hosts_allow > 6. Refresh NN -> NN is still reporting DN as being used by HDFS. > This is different from decom because there DN is added to exclude list in > addition to being removed from allowed list, and in that case everything > works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8937) Fix the exception when set replication to Erasure Coding files
[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712025#comment-14712025 ] Zhe Zhang commented on HDFS-8937: - Thanks [~demongaorui] for reporting the issue. We will probably support striping + replication in the future (striped layout, multiple replicas for each internal block). Or even striping + EC + replication (striped layout, store both data and parity internal blocks, and multiple replica for data/parity internal blocks). So that's not a permanent no-op like setting replication on a directory. Based on that, I think we should still throw an exception now indicating that it's not yet supported. > Fix the exception when set replication to Erasure Coding files > -- > > Key: HDFS-8937 > URL: https://issues.apache.org/jira/browse/HDFS-8937 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: GAO Rui >Assignee: GAO Rui > > Setting replication to an EC file caused exception. We should simply ignore > the request, just like what we're currently doing for a setReplication > request against a directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712016#comment-14712016 ] Zhe Zhang commented on HDFS-8846: - Thanks Colin for reviewing the patch! > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8846: --- Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 2.8.0 (was: 2.7.2) Status: Resolved (was: Patch Available) committed to 2.8 > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.8.0 > > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8938) Refactor BlockManager in blockmanagement
[
https://issues.apache.org/jira/browse/HDFS-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712006#comment-14712006
]
Hadoop QA commented on HDFS-8938:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 42s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 54s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 56s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 30s | The applied patch generated 1
new checkstyle issues (total was 311, now 292). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 44s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 3s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings, and fixes 1 pre-existing warnings. |
| {color:green}+1{color} | native | 3m 49s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 141m 45s | Tests failed in hadoop-hdfs. |
| | | 189m 30s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement |
| | org.apache.hadoop.hdfs.server.namenode.TestDeleteRace |
| | org.apache.hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752279/HDFS-8938.005.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12120/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12120/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12120/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12120/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12120/console |
This message was automatically generated.
> Refactor BlockManager in blockmanagement
>
>
> Key: HDFS-8938
> URL: https://issues.apache.org/jira/browse/HDFS-8938
> Project: Hadoop HDFS
> Issue Type: Task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8938.000.patch, HDFS-8938.001.patch,
> HDFS-8938.002.patch, HDFS-8938.003.patch, HDFS-8938.004.patch,
> HDFS-8938.005.patch
>
>
> This lira tracks the effort of refactoring the {{BlockManager}} in
> {{hdfs.server.blockmanagement}} package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails
[
https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712004#comment-14712004
]
Zhe Zhang commented on HDFS-8704:
-
Thanks for the work Bo. Please find my comments below:
# The JIRA description / summary only describes the symptom. Could you briefly
describe the solution?
# IIUC you are getting rid of the {{setFailed}} logic and instead rewriting the
main logic of {{StripedDataStreamer}}. {{setFailed}} is quite fundamental in
the current fault tolerance logic, it is actually used in the new code from
HDFS-8202. [~walter.k.su] Could you comment on whether you are (or plan to)
rely on it in HDFS-8383?
# {{run}} is a loop and calling it within the while loop of another {{run}}
doesn't look right. The added code in {{StripedDataStreamer#run}} is actually a
little hard to follow. Could you provide a design either in the JIRA summary or
as a comment? What are "trivial packets"?
{code}
+while (!toTerminate && !streamerClosed &&
+dfsClient.clientRunning && !errorState.hasError()) {
+ super.run();
{code}
# The patch needs a rebase.
Smaller issues:
# For the below change, did you see negative {{numBytes}} in tests? That would
be surprising.
{code}
+ //the streamer may fail to send packets
+ if (numBytes < 0) {
+numBytes = s0.bytesSent;
+ }
for (int i = 1; i < numDataBlocks; i++) {
final StripedDataStreamer si = getStripedDataStreamer(i);
final ExtendedBlock bi = si.getBlock();
if (bi != null && bi.getGenerationStamp() >
block.getGenerationStamp()) {
block.setGenerationStamp(bi.getGenerationStamp());
}
-numBytes += atBlockGroupBoundary? bi.getNumBytes():
si.getBytesCurBlock();
+long streamerBytes = atBlockGroupBoundary ? bi.getNumBytes() :
si.getBytesCurBlock();
+if (streamerBytes < 0) {
+ streamerBytes = si.bytesSent;;
+}
+numBytes += streamerBytes;
}
{code}
> Erasure Coding: client fails to write large file when one datanode fails
>
>
> Key: HDFS-8704
> URL: https://issues.apache.org/jira/browse/HDFS-8704
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Li Bo
>Assignee: Li Bo
> Attachments: HDFS-8704-000.patch, HDFS-8704-HDFS-7285-002.patch,
> HDFS-8704-HDFS-7285-003.patch, HDFS-8704-HDFS-7285-004.patch,
> HDFS-8704-HDFS-7285-005.patch
>
>
> I test current code on a 5-node cluster using RS(3,2). When a datanode is
> corrupt, client succeeds to write a file smaller than a block group but fails
> to write a large one. {{TestDFSStripeOutputStreamWithFailure}} only tests
> files smaller than a block group, this jira will add more test situations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8520) Patch for PPC64 block size
[ https://issues.apache.org/jira/browse/HDFS-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711995#comment-14711995 ] Andrew Wang commented on HDFS-8520: --- Hi Tony, My point is that we don't really have PPC64 support right now, things like caching haven't been tested on it AFAIK, and making tests use a variable rather than a hardcoded page size doesn't change the supported status. It certainly wouldn't add support for unknown architectures that have non-4096 and non-64KB page sizes. I proposed using a hardcoded 64KB to patch over this one test in the meantime. I have no way of validating a change to use getOperatingSystemPageSize() actually works on PPC64 since I don't have a PPC64 test environment where it would return 64KB. Thus the hardcode. If there's some big push to add PPC64 support, validate it, and get it in the precommit pool, then sure we can do it more generically. > Patch for PPC64 block size > -- > > Key: HDFS-8520 > URL: https://issues.apache.org/jira/browse/HDFS-8520 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 > Environment: RHEL 7.1 /PPC64 >Reporter: Tony Reix >Assignee: Tony Reix > Attachments: HDFS-8520-2.patch, HDFS-8520.patch > > > The attached patch enables Hadoop to work on PPC64. > That deals with SystemPageSize and BloclSize , which are not 4096 on PPC64. > There are changes in 3 files: > - > hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java > - > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java > - > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java > where 4096 is replaced by getOperatingSystemPageSize() or by using PAGE_SIZE > The patch has been built on branch-2.7 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8846) Add a unit test of INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8846: --- Summary: Add a unit test of INotify functionality across a layout version upgrade (was: Create edit log files with old layout version for upgrade testing) > Add a unit test of INotify functionality across a layout version upgrade > > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8846: --- Summary: Add a unit test for INotify functionality across a layout version upgrade (was: Add a unit test of INotify functionality across a layout version upgrade) > Add a unit test for INotify functionality across a layout version upgrade > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Create edit log files with old layout version for upgrade testing
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711980#comment-14711980 ] Colin Patrick McCabe commented on HDFS-8846: +1. Thanks, [~zhz]. > Create edit log files with old layout version for upgrade testing > - > > Key: HDFS-8846 > URL: https://issues.apache.org/jira/browse/HDFS-8846 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, > HDFS-8846.02.patch, HDFS-8846.03.patch > > > Per discussion under HDFS-8480, we should create some edit log files with old > layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl
[
https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711950#comment-14711950
]
Hadoop QA commented on HDFS-8836:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 22m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 23s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 6s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 10s | The applied patch generated 2
new checkstyle issues (total was 45, now 46). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 6s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 23m 8s | Tests failed in
hadoop-common. |
| | | 73m 40s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752298/HDFS-8836-02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / eee0d45 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/console |
This message was automatically generated.
> Skip newline on empty files with getMerge -nl
> -
>
> Key: HDFS-8836
> URL: https://issues.apache.org/jira/browse/HDFS-8836
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.6.0, 2.7.1
>Reporter: Jan Filipiak
>Assignee: kanaka kumar avvaru
>Priority: Trivial
> Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch
>
>
> Hello everyone,
> I recently was in the need of using the new line option -nl with getMerge
> because the files I needed to merge simply didn't had one. I was merging all
> the files from one directory and unfortunately this directory also included
> empty files, which effectively led to multiple newlines append after some
> files. I needed to remove them manually afterwards.
> In this situation it is maybe good to have another argument that allows
> skipping empty files.
> Thing one could try to implement this feature:
> The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
> return the number of bytes copied which would be convenient as one could
> skip append the new line when 0 bytes where copied or one would check the
> file size before.
> I posted this Idea on the mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E
> but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8784) BlockInfo#numNodes should be numStorages
[ https://issues.apache.org/jira/browse/HDFS-8784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711936#comment-14711936 ] Arpit Agarwal commented on HDFS-8784: - The NameNode does not support more than one replica of a given block per DataNode, so node/storage are interchangeable and node is probably better. > BlockInfo#numNodes should be numStorages > > > Key: HDFS-8784 > URL: https://issues.apache.org/jira/browse/HDFS-8784 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Jagadesh Kiran N > Attachments: HDFS-8784-00.patch, HDFS-8784-01.patch > > > The method actually returns the number of storages holding a block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711937#comment-14711937
]
Zhe Zhang commented on HDFS-8833:
-
The failure will be addressed by HDFS-8704.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8833-HDFS-7285-merge.00.patch,
> HDFS-8833-HDFS-7285-merge.01.patch, HDFS-8833-HDFS-7285.02.patch
>
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8885) ByteRangeInputStream used in webhdfs does not override available()
[
https://issues.apache.org/jira/browse/HDFS-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711935#comment-14711935
]
Chris Nauroth commented on HDFS-8885:
-
Hello [~srevanka]. Thank you for posting the patch. Here are a few notes.
# A {{ByteRangeInputStream}} starts out in {{SEEK}} status, which indicates the
underlying HTTP stream needs to be opened. If a caller calls {{available}}
before any {{read}} calls, then the result won't be correct, because it has not
yet determined the file length. I recommend calling {{getInputStream}} first,
which lazily opens the underlying HTTP stream if necessary.
# A similar situation occurs if a caller calls {{seek}} immediately before
{{available}}. Again, calling {{getInputStream}} first would handle this case
correctly.
# It's typical for {{available}} to throw an {{IOException}} if called after
the stream has been closed. Once again, a call to {{getInputStream}} would
cover this case, because that method is already implemented to throw an
{{IOException}} if the stream is in {{CLOSED}} state.
# Can you please look into adding unit tests in {{TestByteRangeInputStream}}?
> ByteRangeInputStream used in webhdfs does not override available()
> --
>
> Key: HDFS-8885
> URL: https://issues.apache.org/jira/browse/HDFS-8885
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Shradha Revankar
>Assignee: Shradha Revankar
>Priority: Minor
> Attachments: HDFS-8885.000.patch, HDFS-8885.001.patch
>
>
> ByteRangeInputStream has to override the method {{available()}} , the super
> class implementation returns 0. Clients using the method {{available()}} to
> rely on available bytes information will end up with errors during reads if
> WebhdfsFileSystem is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711882#comment-14711882
]
Zhe Zhang commented on HDFS-8833:
-
{{TestFSNamesystem}} passes consistently on local node.
{{testWriteStripedFileWithDNFailure}} is flaky even without the change. I'm
debugging.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8833-HDFS-7285-merge.00.patch,
> HDFS-8833-HDFS-7285-merge.01.patch, HDFS-8833-HDFS-7285.02.patch
>
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8953) DataNode Metrics logging
kanaka kumar avvaru created HDFS-8953: - Summary: DataNode Metrics logging Key: HDFS-8953 URL: https://issues.apache.org/jira/browse/HDFS-8953 Project: Hadoop HDFS Issue Type: Improvement Reporter: kanaka kumar avvaru Assignee: kanaka kumar avvaru HDFS-8880 added metrics logging at NameNode. Similarly, this JIRA is to add a separate logger for metrics at DN -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711874#comment-14711874
]
Chris Nauroth commented on HDFS-8155:
-
Hi [~jghoman]. Nice work! I have a few comments.
# {{ConfRefreshTokenBasedAccessTokenProvider}},
{{ConfRefreshTokenBasedAccessTokenProvider}}: There are no timeouts specified
in the calls to the refresh URL. Timeouts can be controlled by calling
{{client.setConnectTimeout}} and {{client.setReadTimeout}}.
# {{AccessTokenProvider}}: *Optional* - consider extending {{Configured}} so
that it inherits the implementations of {{getConf}} and {{setConf}} for free.
# WebHDFS.md: Typo: "toekns" instead of "tokens"
# Please address the javac and checkstyle warnings.
[~chris.douglas], it looks like the most recent version of the patch has
addressed your feedback. Do you agree?
> Support OAuth2 in WebHDFS
> -
>
> Key: HDFS-8155
> URL: https://issues.apache.org/jira/browse/HDFS-8155
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: webhdfs
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch
>
>
> WebHDFS should be able to accept OAuth2 credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-6955: -- Attachment: HDFS-6955-03.patch updated patch for required check style errors. Test failure doesn't seem to be related to this patch > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: kanaka kumar avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8771:
--
Attachment: HDFS-8771-02.patch
> If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
> send another RPC calls to Journalnodes
>
>
> Key: HDFS-8771
> URL: https://issues.apache.org/jira/browse/HDFS-8771
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takuya Fukudome
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8771-01.patch, HDFS-8771-02.patch
>
>
> In our cluster, edits has became huge(about 50GB) accidentally and our
> Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
> 30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
> Namenode couldn't send other RPC calls to Journalnodes because
> {{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
> It will cause namenode shutting down.
> I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
> like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711839#comment-14711839
]
kanaka kumar avvaru commented on HDFS-8771:
---
updated patch for checkstyle and white space erros reported in jenkins
> If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
> send another RPC calls to Journalnodes
>
>
> Key: HDFS-8771
> URL: https://issues.apache.org/jira/browse/HDFS-8771
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takuya Fukudome
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8771-01.patch, HDFS-8771-02.patch
>
>
> In our cluster, edits has became huge(about 50GB) accidentally and our
> Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
> 30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
> Namenode couldn't send other RPC calls to Journalnodes because
> {{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
> It will cause namenode shutting down.
> I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
> like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Attachment: HDFS-8894-01.patch Some problems in earlier build. Uploading same patch again for jenkins new build. > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: kanaka kumar avvaru > Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711827#comment-14711827 ] kanaka kumar avvaru commented on HDFS-8836: --- Updated patch for jenkin reported errors > Skip newline on empty files with getMerge -nl > - > > Key: HDFS-8836 > URL: https://issues.apache.org/jira/browse/HDFS-8836 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.6.0, 2.7.1 >Reporter: Jan Filipiak >Assignee: kanaka kumar avvaru >Priority: Trivial > Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch > > > Hello everyone, > I recently was in the need of using the new line option -nl with getMerge > because the files I needed to merge simply didn't had one. I was merging all > the files from one directory and unfortunately this directory also included > empty files, which effectively led to multiple newlines append after some > files. I needed to remove them manually afterwards. > In this situation it is maybe good to have another argument that allows > skipping empty files. > Thing one could try to implement this feature: > The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't > return the number of bytes copied which would be convenient as one could > skip append the new line when 0 bytes where copied or one would check the > file size before. > I posted this Idea on the mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E > but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8836) Skip newline on empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8836: -- Attachment: HDFS-8836-02.patch > Skip newline on empty files with getMerge -nl > - > > Key: HDFS-8836 > URL: https://issues.apache.org/jira/browse/HDFS-8836 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.6.0, 2.7.1 >Reporter: Jan Filipiak >Assignee: kanaka kumar avvaru >Priority: Trivial > Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch > > > Hello everyone, > I recently was in the need of using the new line option -nl with getMerge > because the files I needed to merge simply didn't had one. I was merging all > the files from one directory and unfortunately this directory also included > empty files, which effectively led to multiple newlines append after some > files. I needed to remove them manually afterwards. > In this situation it is maybe good to have another argument that allows > skipping empty files. > Thing one could try to implement this feature: > The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't > return the number of bytes copied which would be convenient as one could > skip append the new line when 0 bytes where copied or one would check the > file size before. > I posted this Idea on the mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E > but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711775#comment-14711775
]
Haohui Mai commented on HDFS-8951:
--
Thanks for the work.
{code}
+package org.apache.hadoop.hdfs.util;
+
+import org.slf4j.Logger;
+
+import java.io.IOException;
+
+public class IOUtils {
+
+ /**
+ * Close the Closeable objects and ignore any {@link IOException} or
+ * null pointers. Must only be used for cleanup in exception handlers.
+ *
+ * @param log the log to record problems to at debug level. Can be null.
+ * @param closeables the objects to close
+ */
+ public static void cleanup(Logger log, java.io.Closeable... closeables) {
+for (java.io.Closeable c : closeables) {
+ if (c != null) {
+try {
+ c.close();
+} catch(Throwable e) {
+ if (log != null && log.isDebugEnabled()) {
+log.debug("Exception in closing " + c, e);
+ }
+}
+ }
+}
+ }
+
+}
{code}
The change is not required. It's better to change the callers to use
try-with-resource statement in Java 7.
{code}
-final DataChecksum checksum =
BlockMetadataHeader.readDataChecksum(srcMeta);
+final DataChecksum checksum = BlockMetadataHeader.readDataChecksum(srcMeta,
+DFSUtil.getIoFileBufferSize(new HdfsConfiguration()));
{code}
Creating a new configuration object will parse the XML configuration every
single time. The caller needs to use the {{Configuration}} object in the
{{FSDatasetImpl}} class. Therefore the changes need to be propagated all the
way along the call chain.
> Move shortcircuit to hdfs-client
>
>
> Key: HDFS-8951
> URL: https://issues.apache.org/jira/browse/HDFS-8951
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch
>
>
> This jira tracks the effort of moving the {{shortcircuit}} package into the
> hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Attachment: (was: HDFS-8894-01.patch) > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: kanaka kumar avvaru > Attachments: HDFS-8894-01.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-1148) Convert FSDataset to ReadWriteLock
[
https://issues.apache.org/jira/browse/HDFS-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711727#comment-14711727
]
Hadoop QA commented on HDFS-1148:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 25s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 25s | The applied patch generated 1
new checkstyle issues (total was 121, now 111). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 1s | Post-patch findbugs
hadoop-hdfs-project/hadoop-hdfs compilation is broken. |
| {color:green}+1{color} | findbugs | 2m 1s | The patch does not introduce
any new Findbugs (version ) warnings. |
| {color:green}+1{color} | native | 0m 37s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 6m 39s | Tests failed in hadoop-hdfs. |
| | | 49m 49s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestFileStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752270/HDFS-1148.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/console |
This message was automatically generated.
> Convert FSDataset to ReadWriteLock
> --
>
> Key: HDFS-1148
> URL: https://issues.apache.org/jira/browse/HDFS-1148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, performance
>Reporter: Todd Lipcon
>Assignee: Yong Zhang
> Attachments: HDFS-1148.001.patch, HDFS-1148.002.patch,
> HDFS-1148.003.patch, hdfs-1148-old.txt, hdfs-1148-trunk.txt,
> patch-HDFS-1148-rel0.20.2.txt
>
>
> In benchmarking HDFS-941 I noticed that for the random read workload, the
> FSDataset lock is highly contended. After converting it to a
> ReentrantReadWriteLock, I saw a ~25% improvement on both latency and
> ops/second.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[
https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711714#comment-14711714
]
Hadoop QA commented on HDFS-8894:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 28s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 1
new checkstyle issues (total was 150, now 150). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 17s | Post-patch findbugs
hadoop-hdfs-project/hadoop-hdfs compilation is broken. |
| {color:green}+1{color} | findbugs | 2m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 0m 24s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 0m 23s | Tests failed in hadoop-hdfs. |
| | | 43m 34s | |
\\
\\
|| Reason || Tests ||
| Failed build | hadoop-hdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752268/HDFS-8894-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/console |
This message was automatically generated.
> Set SO_KEEPALIVE on DN server sockets
> -
>
> Key: HDFS-8894
> URL: https://issues.apache.org/jira/browse/HDFS-8894
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Nathan Roberts
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch
>
>
> SO_KEEPALIVE is not set on things like datastreamer sockets which can cause
> lingering ESTABLISHED sockets when there is a network glitch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8938) Refactor BlockManager in blockmanagement
[
https://issues.apache.org/jira/browse/HDFS-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8938:
Attachment: HDFS-8938.005.patch
> Refactor BlockManager in blockmanagement
>
>
> Key: HDFS-8938
> URL: https://issues.apache.org/jira/browse/HDFS-8938
> Project: Hadoop HDFS
> Issue Type: Task
> Components: build
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-8938.000.patch, HDFS-8938.001.patch,
> HDFS-8938.002.patch, HDFS-8938.003.patch, HDFS-8938.004.patch,
> HDFS-8938.005.patch
>
>
> This lira tracks the effort of refactoring the {{BlockManager}} in
> {{hdfs.server.blockmanagement}} package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8915) TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins
[ https://issues.apache.org/jira/browse/HDFS-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDFS-8915. Resolution: Invalid > TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins > - > > Key: HDFS-8915 > URL: https://issues.apache.org/jira/browse/HDFS-8915 > Project: Hadoop HDFS > Issue Type: Bug > Components: HDFS >Affects Versions: 2.8.0 >Reporter: Anu Engineer >Assignee: Anu Engineer > > This test was added as part of HDFS-8883, There is a race condition in the > test and it has failed *once* in the Apache Jenkins run. > Here is the stack > FAILED: > org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount > Error Message: > Expected number of blocked thread not found expected:<3> but was:<1> > Stack Trace: > java.lang.AssertionError: Expected number of blocked thread not found > expected:<3> but was:<1> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at > org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount(TestFSNamesystem.java:261) > From cursory code reading , even though we call into readlock.lock() there is > no guarantee that our thread is put in the wait queue. A proposed fix could > be to check for any thread in the lock queue instead of all 3, or disable the > test. > It could also indicate an issue with the test infra-structure but any test > open to variations in result due to infra-structure issues creates noise in > tests so we are better off fixing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8915) TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins
[ https://issues.apache.org/jira/browse/HDFS-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711680#comment-14711680 ] Anu Engineer commented on HDFS-8915: This was a tracking bug to see if this failure happens in the apache tree. Since the only failure was reported when we had a VM outage, resolving this issue as won't fix. This test has been running without any issues for a week. > TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins > - > > Key: HDFS-8915 > URL: https://issues.apache.org/jira/browse/HDFS-8915 > Project: Hadoop HDFS > Issue Type: Bug > Components: HDFS >Affects Versions: 2.8.0 >Reporter: Anu Engineer >Assignee: Anu Engineer > > This test was added as part of HDFS-8883, There is a race condition in the > test and it has failed *once* in the Apache Jenkins run. > Here is the stack > FAILED: > org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount > Error Message: > Expected number of blocked thread not found expected:<3> but was:<1> > Stack Trace: > java.lang.AssertionError: Expected number of blocked thread not found > expected:<3> but was:<1> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at > org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount(TestFSNamesystem.java:261) > From cursory code reading , even though we call into readlock.lock() there is > no guarantee that our thread is put in the wait queue. A proposed fix could > be to check for any thread in the lock queue instead of all 3, or disable the > test. > It could also indicate an issue with the test infra-structure but any test > open to variations in result due to infra-structure issues creates noise in > tests so we are better off fixing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8373) Ec files can't be deleted into Trash because of that Trash isn't EC zone.
[
https://issues.apache.org/jira/browse/HDFS-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711665#comment-14711665
]
Zhe Zhang commented on HDFS-8373:
-
Good catch [~brahmareddy]. Once HDFS-8833 is finalized let's close this one as
duplicate.
> Ec files can't be deleted into Trash because of that Trash isn't EC zone.
> -
>
> Key: HDFS-8373
> URL: https://issues.apache.org/jira/browse/HDFS-8373
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: GAO Rui
>Assignee: Brahma Reddy Battula
> Labels: EC
>
> When EC files were deleted, they would be moved into {{Trash}} directory.
> But, EC files can only be placed under EC zone. So, EC files which have been
> deleted can not be moved to {{Trash}} directory.
> Problem could be solved by creating a EC zone(floder) inside {{Trash}} to
> contain deleted EC files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
[
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711664#comment-14711664
]
Shradha Revankar commented on HDFS-8943:
Thanks, I've created this https://issues.apache.org/jira/browse/HBASE-14307
> Read apis in ByteRangeInputStream does not read all the bytes specified when
> chunked transfer-encoding is used in the server
>
>
> Key: HDFS-8943
> URL: https://issues.apache.org/jira/browse/HDFS-8943
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Shradha Revankar
>Assignee: Shradha Revankar
> Attachments: HDFS-8943.000.patch
>
>
> With the default Webhdfs server implementation the read apis in
> ByteRangeInputStream work as expected reading the correct number of bytes for
> these apis :
> {{public int read(byte b[], int off, int len)}}
> {{public int read(long position, byte[] buffer, int offset, int length)}}
> But when a custom Webhdfs server implementation is plugged in which uses
> chunked Transfer-encoding, these apis read only the first chunk. Simple fix
> would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Create edit log files with old layout version for upgrade testing
[
https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711658#comment-14711658
]
Zhe Zhang commented on HDFS-8846:
-
The change is test-only and is not related to {{TestEditLog}}. It also passes
locally.
> Create edit log files with old layout version for upgrade testing
> -
>
> Key: HDFS-8846
> URL: https://issues.apache.org/jira/browse/HDFS-8846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch,
> HDFS-8846.02.patch, HDFS-8846.03.patch
>
>
> Per discussion under HDFS-8480, we should create some edit log files with old
> layout version, to test whether they can be correctly handled in upgrades.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-1148) Convert FSDataset to ReadWriteLock
[ https://issues.apache.org/jira/browse/HDFS-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yong Zhang updated HDFS-1148: - Attachment: HDFS-1148.003.patch > Convert FSDataset to ReadWriteLock > -- > > Key: HDFS-1148 > URL: https://issues.apache.org/jira/browse/HDFS-1148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, performance >Reporter: Todd Lipcon >Assignee: Yong Zhang > Attachments: HDFS-1148.001.patch, HDFS-1148.002.patch, > HDFS-1148.003.patch, hdfs-1148-old.txt, hdfs-1148-trunk.txt, > patch-HDFS-1148-rel0.20.2.txt > > > In benchmarking HDFS-941 I noticed that for the random read workload, the > FSDataset lock is highly contended. After converting it to a > ReentrantReadWriteLock, I saw a ~25% improvement on both latency and > ops/second. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Attachment: HDFS-8894-01.patch > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: kanaka kumar avvaru > Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7765) FSOutputSummer throwing ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HDFS-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-7765:
-
Assignee: Janmejay Singh (was: Surendra Singh Lilhore)
> FSOutputSummer throwing ArrayIndexOutOfBoundsException
> --
>
> Key: HDFS-7765
> URL: https://issues.apache.org/jira/browse/HDFS-7765
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
> Environment: Centos 6, Open JDK 7, Amazon EC2, Accumulo 1.6.2RC4
>Reporter: Keith Turner
>Assignee: Janmejay Singh
> Attachments:
> 0001-PATCH-HDFS-7765-FSOutputSummer-throwing-ArrayIndexOu.patch,
> HDFS-7765.patch
>
>
> While running an Accumulo test, saw exceptions like the following while
> trying to write to write ahead log in HDFS.
> The exception occurrs at
> [FSOutputSummer.java:76|https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSOutputSummer.java#L76]
> which is attempting to update a byte array.
> {noformat}
> 2015-02-06 19:46:49,769 [log.DfsLogger] WARN : Exception syncing
> java.lang.reflect.InvocationTargetException
> java.lang.ArrayIndexOutOfBoundsException: 4608
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
> at java.io.DataOutputStream.write(DataOutputStream.java:88)
> at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
> at
> org.apache.accumulo.tserver.logger.LogFileKey.write(LogFileKey.java:87)
> at org.apache.accumulo.tserver.log.DfsLogger.write(DfsLogger.java:526)
> at
> org.apache.accumulo.tserver.log.DfsLogger.logFileData(DfsLogger.java:540)
> at
> org.apache.accumulo.tserver.log.DfsLogger.logManyTablets(DfsLogger.java:573)
> at
> org.apache.accumulo.tserver.log.TabletServerLogger$6.write(TabletServerLogger.java:373)
> at
> org.apache.accumulo.tserver.log.TabletServerLogger.write(TabletServerLogger.java:274)
> at
> org.apache.accumulo.tserver.log.TabletServerLogger.logManyTablets(TabletServerLogger.java:365)
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.flush(TabletServer.java:1667)
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.closeUpdate(TabletServer.java:1754)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.accumulo.trace.instrument.thrift.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
> at
> org.apache.accumulo.server.util.RpcWrapper$1.invoke(RpcWrapper.java:47)
> at com.sun.proxy.$Proxy22.closeUpdate(Unknown Source)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.getResult(TabletClientService.java:2370)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.getResult(TabletClientService.java:2354)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:168)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
> at
> org.apache.accumulo.server.util.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:744)
> 2015-02-06 19:46:49,769 [log.DfsLogger] WARN : Exception syncing
> java.lang.reflect.InvocationTargetException
> 2015-02-06 19:46:49,772 [log.DfsLogger] ERROR:
> java.lang.ArrayIndexOutOfBoundsException: 4609
> java.lang.ArrayIndexOutOfBoundsException: 4609
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
> at java.io.DataOutputStream.write(DataOutputStream.java:88)
> at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
> at
> org.apache.accu
[jira] [Commented] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711437#comment-14711437
]
Hadoop QA commented on HDFS-8771:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 47s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 1
new checkstyle issues (total was 3, now 4). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 162m 26s | Tests failed in hadoop-hdfs. |
| | | 207m 47s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFSNamesystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752216/HDFS-8771-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/console |
This message was automatically generated.
> If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
> send another RPC calls to Journalnodes
>
>
> Key: HDFS-8771
> URL: https://issues.apache.org/jira/browse/HDFS-8771
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takuya Fukudome
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8771-01.patch
>
>
> In our cluster, edits has became huge(about 50GB) accidentally and our
> Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
> 30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
> Namenode couldn't send other RPC calls to Journalnodes because
> {{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
> It will cause namenode shutting down.
> I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
> like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8581) count cmd calculate wrong when huge files exist in one folder
[
https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711399#comment-14711399
]
Hadoop QA commented on HDFS-8581:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 43s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 24s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 9s | Tests passed in hadoop-hdfs.
|
| | | 207m 24s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752211/HDFS-8581.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/console |
This message was automatically generated.
> count cmd calculate wrong when huge files exist in one folder
> -
>
> Key: HDFS-8581
> URL: https://issues.apache.org/jira/browse/HDFS-8581
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Reporter: tongshiquan
>Assignee: J.Andreina
>Priority: Minor
> Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch
>
>
> If one directory such as "/result" exists about 20 files, then when
> execute "hdfs dfs -count /", the result will go wrong. For all directories
> whose name after "/result", file num will not be included.
> My cluster see as below, "/result_1433858936" is the directory exist huge
> files, and files in "/sparkJobHistory", "/tmp", "/user" are not included
> vm-221:/export1/BigData/current # hdfs dfs -ls /
> 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled.
> Found 9 items
> -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10
> /PRE_CREATE_DIR.SUCCESS
> drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume
> drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase
> drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt
> drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history
> drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10
> /result_1433858936
> drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory
> drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp
> drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user
> vm-221:/export1/BigData/current #
> vm-221:/export1/BigData/current # hdfs dfs -count /
> 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled.
> 1043 171536 1756375688 /
> vm-221:/export1/BigData/current #
> vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS
> 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled.
>01 0 /PRE_CREATE_DIR.SUCCESS
> vm-221:/export1/BigData/current #
> vm-221:/export1/BigData/current # hdfs dfs -count /flume
> 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled.
>10
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711326#comment-14711326
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #296 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/296/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
> Compact XAttrs to optimize memory footprint.
>
>
> Key: HDFS-8900
> URL: https://issues.apache.org/jira/browse/HDFS-8900
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
> HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
>
>
> {code}
> private final ImmutableList xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point
> of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
> and each object has memory alignment.
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6515) testPageRounder (org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
[
https://issues.apache.org/jira/browse/HDFS-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711289#comment-14711289
]
Hadoop QA commented on HDFS-6515:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 34s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 36s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 1s | Tests passed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 161m 42s | Tests failed in hadoop-hdfs. |
| | | 231m 44s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12677586/HDFS-6515-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/console |
This message was automatically generated.
> testPageRounder (org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
> -
>
> Key: HDFS-6515
> URL: https://issues.apache.org/jira/browse/HDFS-6515
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.4.0, 2.4.1
> Environment: Linux on PPC64
> Tested with Hadoop 3.0.0 SNAPSHOT, on RHEL 6.5, on Ubuntu 14.04, on Fedora
> 19, using mvn -Dtest=TestFsDatasetCache#testPageRounder -X test
>Reporter: Tony Reix
>Priority: Blocker
> Labels: BB2015-05-TBR, hadoop, test
> Attachments: HDFS-6515-1.patch, HDFS-6515-2.patch
>
>
> I have an issue with test :
>testPageRounder
> (org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
> on Linux/PowerPC.
> On Linux/Intel, test runs fine.
> On Linux/PowerPC, I have:
> testPageRounder(org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
> Time elapsed: 64.037 sec <<< ERROR!
> java.lang.Exception: test timed out after 6 milliseconds
> Looking at details, I see that some "Failed to cache " messages appear in the
> traces. Only 10 on Intel, but 186 on PPC64.
> On PPC64, it looks like some thread is waiting for something that never
> happens, generating a TimeOut.
> I'm now using IBM JVM, however I've just checked that the issue also appears
> with OpenJDK.
> I'm now using Hadoop latest, however, the issue appeared within Hadoop 2.4.0 .
> I need help for understanding what the test is doing, what traces are
> expected, in order to understand what/where is the root cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8941) DistributedFileSystem listCorruptFileBlocks API should resolve relative path
[
https://issues.apache.org/jira/browse/HDFS-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711173#comment-14711173
]
Hadoop QA commented on HDFS-8941:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 3s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 54s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 14s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 25s | The applied patch generated 1
new checkstyle issues (total was 46, now 46). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 163m 44s | Tests failed in hadoop-hdfs. |
| | | 209m 40s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestCrcCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751879/HDFS-8941-00.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12111/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12111/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12111/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12111/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12111/console |
This message was automatically generated.
> DistributedFileSystem listCorruptFileBlocks API should resolve relative path
>
>
> Key: HDFS-8941
> URL: https://issues.apache.org/jira/browse/HDFS-8941
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-8941-00.patch
>
>
> Presently {{DFS#listCorruptFileBlocks(path)}} API is not resolving the given
> path relative to the workingDir. This jira is to discuss and provide the
> implementation of the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl
[
https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711131#comment-14711131
]
Hadoop QA commented on HDFS-8836:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 50s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 58s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 0s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 8s | The applied patch generated 2
new checkstyle issues (total was 45, now 47). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 53s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 16s | Tests failed in
hadoop-common. |
| | | 68m 30s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.cli.TestCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752203/HDFS-8836-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / eee0d45 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/console |
This message was automatically generated.
> Skip newline on empty files with getMerge -nl
> -
>
> Key: HDFS-8836
> URL: https://issues.apache.org/jira/browse/HDFS-8836
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.6.0, 2.7.1
>Reporter: Jan Filipiak
>Assignee: kanaka kumar avvaru
>Priority: Trivial
> Attachments: HDFS-8836-01.patch
>
>
> Hello everyone,
> I recently was in the need of using the new line option -nl with getMerge
> because the files I needed to merge simply didn't had one. I was merging all
> the files from one directory and unfortunately this directory also included
> empty files, which effectively led to multiple newlines append after some
> files. I needed to remove them manually afterwards.
> In this situation it is maybe good to have another argument that allows
> skipping empty files.
> Thing one could try to implement this feature:
> The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
> return the number of bytes copied which would be convenient as one could
> skip append the new line when 0 bytes where copied or one would check the
> file size before.
> I posted this Idea on the mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E
> but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5696) Examples for httpfs REST API incorrect on apache.org
[ https://issues.apache.org/jira/browse/HDFS-5696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1479#comment-1479 ] Harsh J commented on HDFS-5696: --- Thanks for reporting this. Care to submit a fix changing the op values to the right one? I suspect these may be leftovers from the Hoop days. bq. Not sure what the convention should be for specifying the user.name. Use hdfs? or a name that is obviously an example? Since these are curl based examples that also likely assume no kerberos setup, why not $USER or $(whoami) instead of a hardcoded value? > Examples for httpfs REST API incorrect on apache.org > > > Key: HDFS-5696 > URL: https://issues.apache.org/jira/browse/HDFS-5696 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.2.0 > Environment: NA >Reporter: Casey Brotherton >Priority: Trivial > > The examples provided for the httpfs REST API are incorrect. > http://hadoop.apache.org/docs/r2.2.0/hadoop-hdfs-httpfs/index.html > http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-hdfs-httpfs/index.html > From the documentation: > * > HttpFS is a separate service from Hadoop NameNode. > HttpFS itself is Java web-application and it runs using a preconfigured > Tomcat bundled with HttpFS binary distribution. > HttpFS HTTP web-service API calls are HTTP REST calls that map to a HDFS file > system operation. For example, using the curl Unix command: > $ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt returns the > contents of the HDFS /user/foo/README.txt file. > $ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list returns the > contents of the HDFS /user/foo directory in JSON format. > $ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs > creates the HDFS /user/foo.bar directory. > *** > The commands have incorrect "op"erations. ( Verified through source code in > HttpFSFileSystem.java ) > In addition, although the webhdfs documentation specifies user.name as > optional, on my cluster, each action required a "user.name" > It should be included in the short examples to allow for the greatest chance > of success. > Three examples rewritten: > curl -i -L > "http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt?op=open&user.name=hdfsuser"; > curl -i > "http://httpfs-host:14000/webhdfs/v1/user/foo/?op=liststatus&user.name=hdfsuser"; > curl -i -X PUT > "http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs&user.name=hdfsuser"; > Not sure what the convention should be for specifying the user.name. Use > hdfs? or a name that is obviously an example? > It would also be beneficial if the HTTPfs page linked to the webhdfs > documentation page in the text instead of just on the menu sidebar. > http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8771:
--
Status: Patch Available (was: Open)
Attached a patch to handle the {{purgeLogsOlderThan}} calls at {{JournalNode}}
in a separate single threaded executor service so that edits are not blocked
for the purge log completion.
> If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
> send another RPC calls to Journalnodes
>
>
> Key: HDFS-8771
> URL: https://issues.apache.org/jira/browse/HDFS-8771
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takuya Fukudome
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8771-01.patch
>
>
> In our cluster, edits has became huge(about 50GB) accidentally and our
> Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
> 30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
> Namenode couldn't send other RPC calls to Journalnodes because
> {{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
> It will cause namenode shutting down.
> I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
> like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8771:
--
Attachment: HDFS-8771-01.patch
> If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
> send another RPC calls to Journalnodes
>
>
> Key: HDFS-8771
> URL: https://issues.apache.org/jira/browse/HDFS-8771
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takuya Fukudome
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-8771-01.patch
>
>
> In our cluster, edits has became huge(about 50GB) accidentally and our
> Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
> 30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
> Namenode couldn't send other RPC calls to Journalnodes because
> {{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
> It will cause namenode shutting down.
> I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
> like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1474#comment-1474
]
Hudson commented on HDFS-8900:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #304 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/304/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
> Compact XAttrs to optimize memory footprint.
>
>
> Key: HDFS-8900
> URL: https://issues.apache.org/jira/browse/HDFS-8900
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
> HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
>
>
> {code}
> private final ImmutableList xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point
> of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
> and each object has memory alignment.
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1475#comment-1475
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2234 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2234/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
> Compact XAttrs to optimize memory footprint.
>
>
> Key: HDFS-8900
> URL: https://issues.apache.org/jira/browse/HDFS-8900
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
> HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
>
>
> {code}
> private final ImmutableList xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point
> of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
> and each object has memory alignment.
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8581) count cmd calculate wrong when huge files exist in one folder
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8581: - Attachment: HDFS-8581.2.patch Patch is updated. Please review. > count cmd calculate wrong when huge files exist in one folder > - > > Key: HDFS-8581 > URL: https://issues.apache.org/jira/browse/HDFS-8581 > Project: Hadoop HDFS > Issue Type: Bug > Components: HDFS >Reporter: tongshiquan >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch > > > If one directory such as "/result" exists about 20 files, then when > execute "hdfs dfs -count /", the result will go wrong. For all directories > whose name after "/result", file num will not be included. > My cluster see as below, "/result_1433858936" is the directory exist huge > files, and files in "/sparkJobHistory", "/tmp", "/user" are not included > vm-221:/export1/BigData/current # hdfs dfs -ls / > 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. > Found 9 items > -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 > /PRE_CREATE_DIR.SUCCESS > drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume > drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase > drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt > drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history > drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 > /result_1433858936 > drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory > drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp > drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count / > 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. > 1043 171536 1756375688 / > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS > 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. >01 0 /PRE_CREATE_DIR.SUCCESS > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /flume > 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /flume > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hbase > 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. > 36 18 14807 /hbase > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hyt > 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /hyt > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /mr-history > 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. >30 0 /mr-history > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 > 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. > 1001 171517 1756360881 /result_1433858936 > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory > 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. >13 21785 /sparkJobHistory > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /tmp > 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. > 176 35958 /tmp > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /user > 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. > 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8763) Incremental blockreport order may replicate unnecessary block
[
https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711052#comment-14711052
]
Brahma Reddy Battula commented on HDFS-8763:
Change of {{dfs.namenode.replication.min}} to 3 (not 2) might solve this
scenario.
Changing the min replication can lead to new problems. Like safemode on
restart, write failures if not enough datanodes, etc.
So better to increase the replication check interval. Thats enough,I feel.
> Incremental blockreport order may replicate unnecessary block
> -
>
> Key: HDFS-8763
> URL: https://issues.apache.org/jira/browse/HDFS-8763
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: jiangyu
>Assignee: Walter Su
>Priority: Minor
>
> For our cluster, the NameNode is always very busy, so for every incremental
> block report , the contention of lock is heavy.
> The logic of incremental block report is as follow, client send block to dn1
> and dn1 mirror to dn2, dn2 mirror to dn3. After finish this block, all
> datanode will report the newly received block to namenode. In NameNode side,
> all will go to the method processIncrementalBlockReport in BlockManager
> class. But the status of the block reported from dn2,dn3 is RECEIVING_BLOCK,
> for dn1 is RECEIED_BLOCK. It is okay if dn2, dn3 report before dn1(that is
> common), but in some busy environment, it is easy to find dn1 report before
> dn2 or dn3, let’s assume dn2 report first, dn1 report second, and dn3 report
> third.
> So dn1 will addStoredBlock and find the replica of this block is not reach
> the the original number(which is 3), and the block will add to
> neededReplications construction and soon ask some node in pipeline (dn1 or
> dn2)to replica it dn4 . After sometime, dn4 and dn3 all report this block,
> then choose one node to invalidate.
> Here is one log i found in our cluster:
> 2015-07-08 01:05:34,675 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /logs/***_bigdata_spam/logs/application_1435099124107_470749/xx.xx.4.62_45454.tmp.
> BP-1386326728-xx.xx.2.131-1382089338395
> blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-a7c0f8f6-2399-4980-9479-efa08487b7b3:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-c75145a0-ed63-4180-87ee-d48ccaa647c5:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW]]}
> 2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.7.75:50010 is added to
> blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
> size 0
> 2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.4.62:50010 is added to
> blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
>
> ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
> size 0
> 2015-07-08 01:05:35,003 INFO BlockStateChange: BLOCK* ask xx.xx.4.62:50010 to
> replicate blk_3194502674_2121080184 to datanode(s) xx.xx.4.65:50010
> 2015-07-08 01:05:35,403 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.7.73:50010 is added to blk_3194502674_2121080184 size
> 67750
> 2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: xx.xx.4.65:50010 is added to blk_3194502674_2121080184 size
> 67750
> 2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
> blk_3194502674_2121080184 to xx.xx.7.75:50010
> 2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (xx.xx.7.75:50010, blk_3194502674_2121080184) is added to invalidated blocks
> set
> 2015-07-08 01:05:35,852 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> InvalidateBlocks: ask xx.xx.7.75:50010 to delete [blk_3194502674_2121080184,
> blk_3194497594_2121075104]
> Some day, the number of this situation can be 40, that is not good for
> the performance and waste network band.
> Our base version is hadoop 2.4 and i have checked hadoop 2.7.1 didn’t find
> any difference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8836) Skip newline on empty files with getMerge -nl
[
https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8836:
--
Status: Patch Available (was: Open)
Attached a patch to add a new flag {{-skip-empty-file}} and check the file
length before copy content.
> Skip newline on empty files with getMerge -nl
> -
>
> Key: HDFS-8836
> URL: https://issues.apache.org/jira/browse/HDFS-8836
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.7.1, 2.6.0
>Reporter: Jan Filipiak
>Assignee: kanaka kumar avvaru
>Priority: Trivial
> Attachments: HDFS-8836-01.patch
>
>
> Hello everyone,
> I recently was in the need of using the new line option -nl with getMerge
> because the files I needed to merge simply didn't had one. I was merging all
> the files from one directory and unfortunately this directory also included
> empty files, which effectively led to multiple newlines append after some
> files. I needed to remove them manually afterwards.
> In this situation it is maybe good to have another argument that allows
> skipping empty files.
> Thing one could try to implement this feature:
> The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
> return the number of bytes copied which would be convenient as one could
> skip append the new line when 0 bytes where copied or one would check the
> file size before.
> I posted this Idea on the mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E
> but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711045#comment-14711045
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2253 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2253/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
> Compact XAttrs to optimize memory footprint.
>
>
> Key: HDFS-8900
> URL: https://issues.apache.org/jira/browse/HDFS-8900
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.8.0
>
> Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
> HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
>
>
> {code}
> private final ImmutableList xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point
> of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
> and each object has memory alignment.
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8836) Skip newline on empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8836: -- Attachment: HDFS-8836-01.patch > Skip newline on empty files with getMerge -nl > - > > Key: HDFS-8836 > URL: https://issues.apache.org/jira/browse/HDFS-8836 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.6.0, 2.7.1 >Reporter: Jan Filipiak >Assignee: kanaka kumar avvaru >Priority: Trivial > Attachments: HDFS-8836-01.patch > > > Hello everyone, > I recently was in the need of using the new line option -nl with getMerge > because the files I needed to merge simply didn't had one. I was merging all > the files from one directory and unfortunately this directory also included > empty files, which effectively led to multiple newlines append after some > files. I needed to remove them manually afterwards. > In this situation it is maybe good to have another argument that allows > skipping empty files. > Thing one could try to implement this feature: > The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't > return the number of bytes copied which would be convenient as one could > skip append the new line when 0 bytes where copied or one would check the > file size before. > I posted this Idea on the mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E > but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Status: Open (was: Patch Available) > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: kanaka kumar avvaru > Attachments: HDFS-8894-01.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Status: Patch Available (was: Open) > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: kanaka kumar avvaru > Attachments: HDFS-8894-01.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
