[jira] [Commented] (HDFS-10275) TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly
[
https://issues.apache.org/jira/browse/HDFS-10275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234581#comment-15234581
]
Hadoop QA commented on HDFS-10275:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 54s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 38s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 115m 55s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 272m 38s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.fs.contract.hdfs.TestHDFSContractSeek |
| JDK v1.7.0_95 Failed
[jira] [Commented] (HDFS-10269) Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the datanode exit
[
https://issues.apache.org/jira/browse/HDFS-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234591#comment-15234591
]
Brahma Reddy Battula commented on HDFS-10269:
-
One Improvement we could do in this area is,
Move {{volFailuresTolerated}} and its validation code from
{{FSDataSetImpl.java}} to {{DNConf.java}}.
With this, Miss-Configuration will be detected way earlier, in Main thread
itself.
Currently it throws back exception only during the initializing the storage
which happens only after registration to any one of the NameNodes
> Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the
> datanode exit
> --
>
> Key: HDFS-10269
> URL: https://issues.apache.org/jira/browse/HDFS-10269
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-10269.001.patch
>
>
> The datanode start failed and exited when I reused configured for
> dfs.datanode.failed.volumes.tolerated as 5 from my another cluster but
> actually the new cluster only have one datadir path. And this leaded the
> Invalid volume failure config value and threw {{DiskErrorException}}, so the
> datanode shutdown. The info is below:
> {code}
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.common.Storage:
> Failed to add storage for block pool: BP-1239160341-xx.xx.xx.xx-1459929303126
> : BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block
> storage: /home/data/hdfs/data/current/BP-1239160341-xx.xx.xx.xx-1459929303126
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> java.io.IOException: All specified directories are failed to load.
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1361)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid volume failure
> config value: 5
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.(FsDatasetImpl.java:281)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1374)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,359 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,460 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Removed Block pool (Datanode Uuid unassigned)
> 2016-04-07 09:34:47,460 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exiting Datanode
> 2016-04-07 09:34:47,462 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 0
> 2016-04-07 09:34:47,463 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG:
> {code}
> IMO, this will let users feel bad because I only configured a value
> incorrectly. Instead of, we can give a warn info for this and reset this
> value to the default value. It will be a bette
[jira] [Commented] (HDFS-10172) hdfs erasurecode command should remove the redundant -usage option
[
https://issues.apache.org/jira/browse/HDFS-10172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234619#comment-15234619
]
Hadoop QA commented on HDFS-10172:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 55s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 133m 55s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 278m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
|
[jira] [Commented] (HDFS-7661) [umbrella] support hflush and hsync for erasure coded files
[ https://issues.apache.org/jira/browse/HDFS-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234667#comment-15234667 ] GAO Rui commented on HDFS-7661: --- Thanks for your comments, [~drankye],[~tlipcon],[~zhz],[~walter.k.su]. I am totally agree that we should not let this JIRA block the 3.0 release. We could make hflush/hsync of EC files as TO-BE-IMPLEMENTED, and release 3.0 . For the future phrase, along with the new decode/encode library and hardware support of EC, maybe we could use striped EC instead of replication as the default and major file format of HDFS. The function of hflush/hsync would become necessary, so [~liuml07] and I may should continue to implement hflush/hsync based on the latest design. We would break down this JIRA to several sub tasks and solved them one by one. :D > [umbrella] support hflush and hsync for erasure coded files > --- > > Key: HDFS-7661 > URL: https://issues.apache.org/jira/browse/HDFS-7661 > Project: Hadoop HDFS > Issue Type: New Feature > Components: erasure-coding >Reporter: Tsz Wo Nicholas Sze >Assignee: GAO Rui > Attachments: EC-file-flush-and-sync-steps-plan-2015-12-01.png, > HDFS-7661-unitTest-wip-trunk.patch, HDFS-7661-wip.01.patch, > HDFS-EC-file-flush-sync-design-v20160323.pdf, > HDFS-EC-file-flush-sync-design-version1.1.pdf, Undo-Log-Design-20160406.jpg > > > We also need to support hflush/hsync and visible length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6489) DFS Used space is not correct computed on frequent append operations
[
https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234692#comment-15234692
]
Weiwei Yang commented on HDFS-6489:
---
These tests can run successfully on my local environment, attach the output
{quote}
Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.709 sec -
in org.apache.hadoop.hdfs.TestEncryptionZones
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.741 sec - in
org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.997 sec - in
org.apache.hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.338 sec - in
org.apache.hadoop.http.TestHttpServerLifecycle
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.343 sec -
in org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
{quote}
Can someone help to review this patch? Thanks a lot.
> DFS Used space is not correct computed on frequent append operations
>
>
> Key: HDFS-6489
> URL: https://issues.apache.org/jira/browse/HDFS-6489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0, 2.7.1, 2.7.2
>Reporter: stanley shi
>Assignee: Weiwei Yang
> Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch,
> HDFS-6489.003.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space
> on each block write operation. This is correct in most scenario (create new
> file), but sometimes it will behave in-correct(append small data to a large
> block).
> For example, I have a file with only one block(say, 60M). Then I try to
> append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the
> block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large
> number of files (1000+), assume the block size is 32M (half of the default
> value), then the dfs used will be increased 1000*32M = 32G on each append to
> the files; but actually I only write 10K bytes; this will cause the datanode
> to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
> BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received
> exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException:
> Insufficient space for appending to FinalizedReplica, blk_1073742834_45306,
> FINALIZED{quote}
> But the actual disk usage:
> {quote}
> [root@hdsh143 ~]# df -h
> FilesystemSize Used Avail Use% Mounted on
> /dev/sda3 16G 2.9G 13G 20% /
> tmpfs 1.9G 72K 1.9G 1% /dev/shm
> /dev/sda1 97M 32M 61M 35% /boot
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9833) Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data
[
https://issues.apache.org/jira/browse/HDFS-9833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-9833:
Labels: hdfs-ec-3.0-must-do (was: )
> Erasure coding: recomputing block checksum on the fly by reconstructing the
> missed/corrupt block data
> -
>
> Key: HDFS-9833
> URL: https://issues.apache.org/jira/browse/HDFS-9833
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Labels: hdfs-ec-3.0-must-do
>
> As discussed in HDFS-8430 and HDFS-9694, to compute striped file checksum
> even some of striped blocks are missed, we need to consider recomputing block
> checksum on the fly for the missed/corrupt blocks. To recompute the block
> checksum, the block data needs to be reconstructed by erasure decoding, and
> the main needed codes for the block reconstruction could be borrowed from
> HDFS-9719, the refactoring of the existing {{ErasureCodingWorker}}. In EC
> worker, reconstructed blocks need to be written out to target datanodes, but
> here in this case, the remote writing isn't necessary, as the reconstructed
> block data is only used to recompute the checksum.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10220) Namenode failover due to too long loking in LeaseManager.Monitor
[ https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Fraison updated HDFS-10220: --- Status: Patch Available (was: Reopened) > Namenode failover due to too long loking in LeaseManager.Monitor > > > Key: HDFS-10220 > URL: https://issues.apache.org/jira/browse/HDFS-10220 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Nicolas Fraison >Assignee: Nicolas Fraison >Priority: Minor > Attachments: HADOOP-10220.001.patch, HADOOP-10220.002.patch, > HADOOP-10220.003.patch, threaddump_zkfc.txt > > > I have faced a namenode failover due to unresponsive namenode detected by the > zkfc with lot's of WARN messages (5 millions) like this one: > _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All > existing blocks are COMPLETE, lease removed, file closed._ > On the threaddump taken by the zkfc there are lots of thread blocked due to a > lock. > Looking at the code, there are a lock taken by the LeaseManager.Monitor when > some lease must be released. Due to the really big number of lease to be > released the namenode has taken too many times to release them blocking all > other tasks and making the zkfc thinking that the namenode was not > available/stuck. > The idea of this patch is to limit the number of leased released each time we > check for lease so the lock won't be taken for a too long time period. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10270) TestJMXGet:testNameNode() fails
[
https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234823#comment-15234823
]
Gergely Novák commented on HDFS-10270:
--
We observed that the fail is caused by this assert:
{code}
DFSTestUtil.waitForMetric(jmx, "NumOpenConnections", numDatanodes);
{code}
This checks if the number of open connections equals to the number of data
nodes. But the number of open connections has absolutely no dependency from the
data nodes: it's either 0, 1 (DataNodeProtocol) or 2 (DataNodeProtocol and
ClientProtocol). The test passes in those rare cases when the ClientProtocol
hasn't timed out when the {{TestNameNode}} runs (this can only happen if the
tests are run separately). If we increment the number of data nodes (to 3, or
so) the test will always fail. Contrarily if we increase the client timeout
({{ipc.client.connection.maxidletime}}) the test will always pass.
Our suggestion is to remove this assert.
> TestJMXGet:testNameNode() fails
> ---
>
> Key: HDFS-10270
> URL: https://issues.apache.org/jira/browse/HDFS-10270
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Andras Bokor
>Priority: Minor
> Attachments: TestJMXGet.log, TestJMXGetFails.log
>
>
> It fails with java.util.concurrent.TimeoutException. Actually the problem
> here is that we expect 2 as NumOpenConnections metric but it is only 1. So
> the test waits 60 sec then fails.
> Please find maven output so the stack trace attached ([^TestJMXGetFails.log]).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HDFS-10270: - Affects Version/s: 2.8.0 3.0.0 Status: Patch Available (was: Open) > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Priority: Minor > Attachments: HDFS-10270.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HDFS-10270: - Attachment: HDFS-10270.patch > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Priority: Minor > Attachments: HDFS-10270.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HDFS-10270: - Attachment: (was: HDFS-10270.patch) > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Priority: Minor > Attachments: HDFS-10270.001.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HDFS-10270: - Attachment: HDFS-10270.001.patch > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Priority: Minor > Attachments: HDFS-10270.001.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák reassigned HDFS-10270: Assignee: Gergely Novák > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Assignee: Gergely Novák >Priority: Minor > Attachments: HDFS-10270.001.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HDFS-10270: - Affects Version/s: (was: 2.8.0) (was: 3.0.0) Status: Open (was: Patch Available) > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Andras Bokor >Assignee: Gergely Novák >Priority: Minor > Attachments: HDFS-10270.001.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HDFS-10270: - Affects Version/s: 2.8.0 3.0.0 > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Assignee: Gergely Novák >Priority: Minor > Attachments: HDFS-10270.001.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated HDFS-10270: - Status: Patch Available (was: Open) > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Assignee: Gergely Novák >Priority: Minor > Attachments: HDFS-10270.001.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10220) Namenode failover due to too long loking in LeaseManager.Monitor
[
https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234903#comment-15234903
]
Hadoop QA commented on HDFS-10220:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 9 new +
603 unchanged - 2 fixed = 612 total (was 605) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 57s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 29s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 113m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestFileAppend |
| JDK v1.8.0_77 Timed out junit tests |
org.apache.hadoop.hdfs.Tes
[jira] [Commented] (HDFS-334) dfsadmin -metasave should also log corrupt replicas info
[ https://issues.apache.org/jira/browse/HDFS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234959#comment-15234959 ] Andras Bokor commented on HDFS-334: --- [~lohit] What is your exact expectation with corrupt replicas map? What information do we need? Is it still a valid request? > dfsadmin -metasave should also log corrupt replicas info > > > Key: HDFS-334 > URL: https://issues.apache.org/jira/browse/HDFS-334 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Lohit Vijayarenu >Priority: Minor > Labels: newbie > > _hadoop dfsadmin -metasave _ should also dump information about > corrupt replicas map. This could help in telling if pending replication was > due to corrupt replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5059) Unnecessary permission denied error when creating/deleting snapshots with a non-existent directory
[
https://issues.apache.org/jira/browse/HDFS-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234981#comment-15234981
]
Andras Bokor commented on HDFS-5059:
[~jxiang] is right. HDFS-5111 has solved this. I am not getting extra
'Permission denied' now.
[~schu] Can we set this as resolved?
> Unnecessary permission denied error when creating/deleting snapshots with a
> non-existent directory
> --
>
> Key: HDFS-5059
> URL: https://issues.apache.org/jira/browse/HDFS-5059
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0
>Reporter: Stephen Chu
>Priority: Trivial
> Labels: newbie
>
> As a non-superuser, when you create and delete a snapshot but accidentally
> specify a non-existent directory to snapshot, you will see an
> extra/unnecessary permission denied error right after the "No such file or
> directory" error.
> {code}
> [schu@hdfs-snapshots-vanilla ~]$ hdfs dfs -deleteSnapshot /user/schuf/ snap1
> deleteSnapshot: `/user/schuf/': No such file or directory
> deleteSnapshot: Permission denied
> [schu@hdfs-snapshots-vanilla ~]$ hdfs dfs -createSnapshot /user/schuf/ snap1
> createSnapshot: `/user/schuf/': No such file or directory
> createSnapshot: Permission denied
> {code}
> As the HDFS superuser, instead of the "Permission denied" error you'll get an
> extra "Directory does not exist" error.
> {code}
> [root@hdfs-snapshots-vanilla ~]# hdfs dfs -deleteSnapshot /user/schuf/ snap1
> deleteSnapshot: `/user/schuf/': No such file or directory
> deleteSnapshot: Directory does not exist: /user/schuf
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10270) TestJMXGet:testNameNode() fails
[
https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235117#comment-15235117
]
Hadoop QA commented on HDFS-10270:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 25s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 231m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.datanode.TestDataNodeMXBean |
| | hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.fs.contract.hdfs.TestHDFSContractSeek |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namen
[jira] [Commented] (HDFS-10270) TestJMXGet:testNameNode() fails
[ https://issues.apache.org/jira/browse/HDFS-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235349#comment-15235349 ] Kihwal Lee commented on HDFS-10270: --- It looks like a race was introduced after the jmx caching was "fixed". Certain metrics values in JMX is now using the cached value, which expires in 10 seconds by default. Since the ipc client idle timeout is also 10 seconds, The ipc connection may get closed before the jmx refresh. So the number of actual connections reaches to 2, but can drop to 1 just before a new jmx data is fetched. We could fix this by either lowering the jmx cache expiration or increasing ipc client idle timeout. Since it is a JMX test, it might be better to leave the jmx setting as default and change the ipc timeout to, say, 15 seconds. > TestJMXGet:testNameNode() fails > --- > > Key: HDFS-10270 > URL: https://issues.apache.org/jira/browse/HDFS-10270 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: Andras Bokor >Assignee: Gergely Novák >Priority: Minor > Attachments: HDFS-10270.001.patch, TestJMXGet.log, TestJMXGetFails.log > > > It fails with java.util.concurrent.TimeoutException. Actually the problem > here is that we expect 2 as NumOpenConnections metric but it is only 1. So > the test waits 60 sec then fails. > Please find maven output so the stack trace attached ([^TestJMXGetFails.log]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-10276) Different results for exist call for file.ext/name
Kevin Cox created HDFS-10276:
Summary: Different results for exist call for file.ext/name
Key: HDFS-10276
URL: https://issues.apache.org/jira/browse/HDFS-10276
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kevin Cox
Given you have a file {{/file}} an existence check for the path
{{/file/whatever}} will give different responses for different implementations
of FileSystem.
LocalFileSystem will return false while DistributedFileSystem will throw
{{org.apache.hadoop.security.AccessControlException: Permission denied: ...,
access=EXECUTE, ...}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
Steve Loughran created HDFS-10277:
-
Summary: PositionedReadable test testReadFullyZeroByteFile failing
in HDFS
Key: HDFS-10277
URL: https://issues.apache.org/jira/browse/HDFS-10277
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Environment: Jenkins
Reporter: Steve Loughran
Assignee: Steve Loughran
Jenkins is failing after HADOOP-12994,
{{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235465#comment-15235465
]
Steve Loughran commented on HDFS-10277:
---
{code}
java.lang.AssertionError: expected:<0> but was:<-1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> PositionedReadable test testReadFullyZeroByteFile failing in HDFS
> -
>
> Key: HDFS-10277
> URL: https://issues.apache.org/jira/browse/HDFS-10277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Jenkins is failing after HADOOP-12994,
> {{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8872) Reporting of missing blocks is different in fsck and namenode ui/metasave
[
https://issues.apache.org/jira/browse/HDFS-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235491#comment-15235491
]
Wei-Chiu Chuang commented on HDFS-8872:
---
Looking at the source code, I think there is a mismatch of understanding
between webui and fsck.
In webui, the missing blocks actually means the number of blocks that have no
replicas at all. (that is, the number of corrupt blocks); whereas in fsck, it
shows the difference between the expected replication and the available
replications.
More specifically, the webui gets the number from
{{FSNamesystem#getMissingBlocksCount}}, and this in turn calls
{{LowRedundancyBlocks#getCorruptBlockSize}}.
I think that to fix it, {{FSNamesystem#getMissingBlocksCount}} should not
return the number from {{LowRedundancyBlocks#getCorruptBlockSize}}. Instead, it
should derive the number in the same way as in
{{NamenodeFsck#collectBlocksSummary}}.
> Reporting of missing blocks is different in fsck and namenode ui/metasave
> -
>
> Key: HDFS-8872
> URL: https://issues.apache.org/jira/browse/HDFS-8872
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> Namenode ui and metasave will not report a block as missing if the only
> replica is on decommissioning/decomissioned node while fsck will show it as
> MISSING.
> Since decommissioned node can be formatted/removed anytime, we can actually
> lose the block.
> Its better to alert on namenode ui if the only copy is on
> decomissioned/decommissioning node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10220) Namenode failover due to too long loking in LeaseManager.Monitor
[
https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235511#comment-15235511
]
Vinayakumar B commented on HDFS-10220:
--
Thanks @Nicolas Fraison for the patch.
Looks almost good. Some comments below.
1. "DFS_NAMENODE_MAX_FILES_CHECKED_PER_ITERATION_KEY" could be
"DFS_NAMENODE_LEASE_MAX_FILES_CHECKED_PER_ITERATION_KEY", same for DEFAULT as
well.
2. {code}
/** Maximum number of files whose lease will be released in one
iteration
* of LeaseManager.checkLeases()
*/
private final int maxFilesLeasesCheckedForRelease;
{code}
Comment could be corrected, saying 'number of files checked to release lease in
one iteration'
3. {code}
if (filesLeasesChecked >=
fsnamesystem.getMaxFilesLeasesCheckedForRelease()) {
removeFilesInLease(leaseToCheck, removing);
LOG.warn("Breaking out of checkLeases() after " +
filesLeasesChecked
+ " file leases checked.");
break;
}
{code}
Here, {{removeFilesInLease(leaseToCheck, removing);}} may not be required. as
just breaking out from for-loop will do the removal outside while-loop. So
extracting to separate method also may not be necessary.
4. {code}
- Long[] leaseINodeIds = files.toArray(new Long[files.size()]);
+ final int numLeasePaths = files.size();
+ Long[] leaseINodeIds = files.toArray(new Long[numLeasePaths]);
{code}
This also may not be required.
5. checkstyle comments can be addressed.
+1, once these are addressed.
> Namenode failover due to too long loking in LeaseManager.Monitor
>
>
> Key: HDFS-10220
> URL: https://issues.apache.org/jira/browse/HDFS-10220
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Nicolas Fraison
>Assignee: Nicolas Fraison
>Priority: Minor
> Attachments: HADOOP-10220.001.patch, HADOOP-10220.002.patch,
> HADOOP-10220.003.patch, threaddump_zkfc.txt
>
>
> I have faced a namenode failover due to unresponsive namenode detected by the
> zkfc with lot's of WARN messages (5 millions) like this one:
> _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All
> existing blocks are COMPLETE, lease removed, file closed._
> On the threaddump taken by the zkfc there are lots of thread blocked due to a
> lock.
> Looking at the code, there are a lock taken by the LeaseManager.Monitor when
> some lease must be released. Due to the really big number of lease to be
> released the namenode has taken too many times to release them blocking all
> other tasks and making the zkfc thinking that the namenode was not
> available/stuck.
> The idea of this patch is to limit the number of leased released each time we
> check for lease so the lock won't be taken for a too long time period.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8872) Reporting of missing blocks is different in fsck and namenode ui/metasave
[ https://issues.apache.org/jira/browse/HDFS-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235510#comment-15235510 ] Wei-Chiu Chuang commented on HDFS-8872: --- Or, instead of calling it as "missing blocks" on the web UI, it should be called "corrupt blocks". I'd like to hear comments from web UI experts, [~wheat9] could you comment? Thanks. > Reporting of missing blocks is different in fsck and namenode ui/metasave > - > > Key: HDFS-8872 > URL: https://issues.apache.org/jira/browse/HDFS-8872 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > > Namenode ui and metasave will not report a block as missing if the only > replica is on decommissioning/decomissioned node while fsck will show it as > MISSING. > Since decommissioned node can be formatted/removed anytime, we can actually > lose the block. > Its better to alert on namenode ui if the only copy is on > decomissioned/decommissioning node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7499) Add NFSv4 + Kerberos / client authentication support
[ https://issues.apache.org/jira/browse/HDFS-7499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-7499: - Assignee: (was: John Zhuge) > Add NFSv4 + Kerberos / client authentication support > > > Key: HDFS-7499 > URL: https://issues.apache.org/jira/browse/HDFS-7499 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.4.0 > Environment: HDP2.1 >Reporter: Hari Sekhon > > We have a requirement for secure file share access to HDFS on a kerberized > cluster. > This is spun off from HDFS-7488 where adding Kerberos to the front end client > was considered, I believe this would require NFSv4 support? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7499) Add NFSv4 + Kerberos / client authentication support
[ https://issues.apache.org/jira/browse/HDFS-7499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235557#comment-15235557 ] John Zhuge commented on HDFS-7499: -- Do not have bandwidth to work on it for now. > Add NFSv4 + Kerberos / client authentication support > > > Key: HDFS-7499 > URL: https://issues.apache.org/jira/browse/HDFS-7499 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 2.4.0 > Environment: HDP2.1 >Reporter: Hari Sekhon > > We have a requirement for secure file share access to HDFS on a kerberized > cluster. > This is spun off from HDFS-7488 where adding Kerberos to the front end client > was considered, I believe this would require NFSv4 support? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6819) make HDFS fault injection framework working with maven
[
https://issues.apache.org/jira/browse/HDFS-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235595#comment-15235595
]
Colin Patrick McCabe commented on HDFS-6819:
My vote would be for removing the documentation, as well as the
{{./hadoop-hdfs-project/hadoop-hdfs/src/test/aop}} directory. I honestly don't
think we need another heavyweight framework like aspect4j. It's another thing
for everyone to learn, and provides little that a library like Mockito can't
provide. This code hasn't been compiled in years, and we've done pretty well
at injecting faults without it. If you post a patch to remove this, I will +1.
> make HDFS fault injection framework working with maven
> --
>
> Key: HDFS-6819
> URL: https://issues.apache.org/jira/browse/HDFS-6819
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: George Wong
>Assignee: George Wong
>
> In current trunk code repo, the FI framework does not work. Because maven
> build process does not execute the AspectJ injection.
> Since FI is very useful for testing and bug reproduce, it is better to make
> FI framework working in the trunk code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
[
https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235651#comment-15235651
]
Colin Patrick McCabe commented on HDFS-3702:
So the proposal is to add a new API to {{DistributedFileSystem}} that would
support NO_LOCAL_WRITE. Since we would want to continue to support all the
existing options to create, it would look something like this:
{code}
public FSDataOutputStream create(Path f,
FsPermission permission,
EnumSet flags,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
ChecksumOpt checksumOpt,
boolean noLocalWrite) throws IOException {
{code}
That's 9 different parameters, which seems like a definite code smell.
Is it really worth adding this ugly API to avoid introducing a new
{{CreateFlag}}? Adding a new {{CreateFlag}} seems harmless to me. If
{{CreateFlag.NO_LOCAL_WRITE}} doesn't work out, we will stick a
{{\@deprecated}} next to it and start ignoring it. It's still better than
adding a 9-argument function which requires a typecast to use. We already have
a lot of {{CreateFlags}} that are HDFS-specific such as {{LAZY_PERSIST}},
{{SYNC_BLOCK}}, and {{APPEND_NEWBLOCK}}.
Also keep in mind that once we add this ugly 9-argument API to
{{DistributedFileSystem}}, we won't be able to remove it. In order for HBase
to use it, it has to be public and part of an official Apache release of
Hadoop. By our own API policies, it becomes permanent at that point. Why is
HDFS so unkind to our downstream projects?
> Add an option for NOT writing the blocks locally if there is a datanode on
> the same box as the client
> -
>
> Key: HDFS-3702
> URL: https://issues.apache.org/jira/browse/HDFS-3702
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.5.1
>Reporter: Nicolas Liochon
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch,
> HDFS-3702.002.patch, HDFS-3702.003.patch, HDFS-3702.004.patch,
> HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch,
> HDFS-3702.008.patch, HDFS-3702.009.patch, HDFS-3702.010.patch,
> HDFS-3702.011.patch, HDFS-3702_Design.pdf
>
>
> This is useful for Write-Ahead-Logs: these files are writen for recovery
> only, and are not read when there are no failures.
> Taking HBase as an example, these files will be read only if the process that
> wrote them (the 'HBase regionserver') dies. This will likely come from a
> hardware failure, hence the corresponding datanode will be dead as well. So
> we're writing 3 replicas, but in reality only 2 of them are really useful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10217) show "blockScheduled count" in datanodes table.
[
https://issues.apache.org/jira/browse/HDFS-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235716#comment-15235716
]
Ravi Prakash commented on HDFS-10217:
-
What is the motivation behind this Brahma? Besides I see this returns
{{(currApproxBlocksScheduled.sum() + prevApproxBlocksScheduled.sum()}} . If its
to check how busy a datanode is at present, a lot factors come into play. e.g.
the block sizes of the blocks, the heartbeat frequency, other readers, writers
etc.
> show "blockScheduled count" in datanodes table.
> ---
>
> Key: HDFS-10217
> URL: https://issues.apache.org/jira/browse/HDFS-10217
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-10217.patch
>
>
> It will more useful for debugging purpose where user can see how many blocks
> got schduled for DN
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10269) Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the datanode exit
[
https://issues.apache.org/jira/browse/HDFS-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235753#comment-15235753
]
Andrew Wang commented on HDFS-10269:
Brahma's idea seems reasonable, but I think we should file a new JIRA for that.
Brahma, do you mind doing the honors?
I'll close this one as WONTFIX, thanks all for the discussion.
> Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the
> datanode exit
> --
>
> Key: HDFS-10269
> URL: https://issues.apache.org/jira/browse/HDFS-10269
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-10269.001.patch
>
>
> The datanode start failed and exited when I reused configured for
> dfs.datanode.failed.volumes.tolerated as 5 from my another cluster but
> actually the new cluster only have one datadir path. And this leaded the
> Invalid volume failure config value and threw {{DiskErrorException}}, so the
> datanode shutdown. The info is below:
> {code}
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.common.Storage:
> Failed to add storage for block pool: BP-1239160341-xx.xx.xx.xx-1459929303126
> : BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block
> storage: /home/data/hdfs/data/current/BP-1239160341-xx.xx.xx.xx-1459929303126
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> java.io.IOException: All specified directories are failed to load.
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1361)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid volume failure
> config value: 5
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.(FsDatasetImpl.java:281)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1374)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,359 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,460 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Removed Block pool (Datanode Uuid unassigned)
> 2016-04-07 09:34:47,460 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exiting Datanode
> 2016-04-07 09:34:47,462 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 0
> 2016-04-07 09:34:47,463 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG:
> {code}
> IMO, this will let users feel bad because I only configured a value
> incorrectly. Instead of, we can give a warn info for this and reset this
> value to the default value. It will be a better way for this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10269) Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the datanode exit
[
https://issues.apache.org/jira/browse/HDFS-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-10269:
---
Resolution: Not A Bug
Status: Resolved (was: Patch Available)
Closing as per above, thanks all.
> Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the
> datanode exit
> --
>
> Key: HDFS-10269
> URL: https://issues.apache.org/jira/browse/HDFS-10269
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-10269.001.patch
>
>
> The datanode start failed and exited when I reused configured for
> dfs.datanode.failed.volumes.tolerated as 5 from my another cluster but
> actually the new cluster only have one datadir path. And this leaded the
> Invalid volume failure config value and threw {{DiskErrorException}}, so the
> datanode shutdown. The info is below:
> {code}
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.common.Storage:
> Failed to add storage for block pool: BP-1239160341-xx.xx.xx.xx-1459929303126
> : BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block
> storage: /home/data/hdfs/data/current/BP-1239160341-xx.xx.xx.xx-1459929303126
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> java.io.IOException: All specified directories are failed to load.
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1361)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid volume failure
> config value: 5
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.(FsDatasetImpl.java:281)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1374)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,359 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,460 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Removed Block pool (Datanode Uuid unassigned)
> 2016-04-07 09:34:47,460 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exiting Datanode
> 2016-04-07 09:34:47,462 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 0
> 2016-04-07 09:34:47,463 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG:
> {code}
> IMO, this will let users feel bad because I only configured a value
> incorrectly. Instead of, we can give a warn info for this and reset this
> value to the default value. It will be a better way for this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235758#comment-15235758 ] Vinod Kumar Vavilapalli commented on HDFS-10271: Can somebody from HDFS please help review / commit this? It's one of the last few blocking tickets for 2.7.3! > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes reserved may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-3743) QJM: improve formatting behavior for JNs
[ https://issues.apache.org/jira/browse/HDFS-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235906#comment-15235906 ] Jian Fang commented on HDFS-3743: - I implemented the "-newEditsOnly" option and tested it on a cluster with 3 instances that ran server side daemons such as name nodes, resource managers, journal nodes, and others. I replaced two out of the three instances successfully with name node running fine by issuing the command "hdfs namenode -initializeSharedEdits -newEditsOnly" before started a name node. However, I ran into an issue when I replaced the last existing instance, i.e., replaced the original m1,m2, and m3 with m4, m5, and m6 one at a time. The name node suddenly stuck in loading fsimage. The error message was as follows. " 2016-04-04 06:51:34,612 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode (main): Loading 1 INodes. 2016-04-04 06:51:34,670 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf (main): Loaded FSImage in 0 seconds. 2016-04-04 06:51:34,670 INFO org.apache.hadoop.hdfs.server.namenode.FSImage (main): Loaded image for txid 0 from /mnt/var/lib/hadoop/dfs-name/current/fsimage_000 2016-04-04 06:51:34,678 INFO org.apache.hadoop.hdfs.server.namenode.FSImage (main): Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@3af6b48f expecting start txid #1 2016-04-04 06:51:34,678 INFO org.apache.hadoop.hdfs.server.namenode.FSImage (main): Start loading edits file http://ip-10-171-94-13.ec2.internal:8480/getJournal?jid=my-cluster&segmentTxId=19&storageInfo=-63%3A1708928638%3A0%3ACID-218c0d5a-8772-4a75-8ede-44f84d24f280 2016-04-04 06:51:34,683 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream (main): Fast-forwarding stream 'http://ip-10-171-94-13.ec2.internal:8480/getJournal?jid=my-cluster&segmentTxId=19&storageInfo=-63%3A1708928638%3A0%3ACID-218c0d5a-8772-4a75-8ede-44f84d24f280' to transaction ID 1 2016-04-04 06:51:34,683 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream (main): Fast-forwarding stream 'http://ip-10-171-94-13.ec2.internal:8480/getJournal?jid=my-cluster&segmentTxId=19&storageInfo=-63%3A1708928638%3A0%3ACID-218c0d5a-8772-4a75-8ede-44f84d24f280' to transaction ID 1 2016-04-04 06:51:34,845 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem (main): Encountered exception loading fsimage java.io.IOException: There appears to be a gap in the edit log. We expected txid 1, but got txid 19. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:814) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:798) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1491) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1557) " Seems the fsimage was messed up during the replacement. Could someone shed some light on what could possibly be wrong? In my "-newEditsOnly" logic, I skipped the following code snippet in the initializeSharedEdits() method because the other name node could be still running and be an active one when the above command was issued. I assumed that the new journal node could rely on the recovery protocol in QJM to sync up with other peers for the latest transactions. -- NNStorage newSharedStorage = sharedEditsImage.getStorage(); // Call Storage.format instead of FSImage.format here, since we don't // actually want to save a checkpoint - just prime the dirs with // the existing namespace info newSharedStorage.format(nsInfo); sharedEditsImage.getEditLog().formatNonFileJournals(nsInfo); // Need to make sure the edit log segments are in good shape to initialize // the shared edits dir. fsns.getFSImage().getEditLog().close(); fsns.getFSImage().getEditLog().initJournalsForWrite(); fsns.getFSImage().getEditLog
[jira] [Created] (HDFS-10278) Ozone: Add paging support to list Volumes
Anu Engineer created HDFS-10278: --- Summary: Ozone: Add paging support to list Volumes Key: HDFS-10278 URL: https://issues.apache.org/jira/browse/HDFS-10278 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: HDFS-7240 Add paging support to list volumes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10278) Ozone: Add paging support to list Volumes
[ https://issues.apache.org/jira/browse/HDFS-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10278: Status: Patch Available (was: Open) > Ozone: Add paging support to list Volumes > - > > Key: HDFS-10278 > URL: https://issues.apache.org/jira/browse/HDFS-10278 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-10278-HDFS-7240.001.patch > > > Add paging support to list volumes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10278) Ozone: Add paging support to list Volumes
[ https://issues.apache.org/jira/browse/HDFS-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10278: Attachment: HDFS-10278-HDFS-7240.001.patch Checkstyle will have a warning about 9 args. I chose to do it that way so that JAX-RS annotations are visible in REST handler signature instead of being embedded in the code. With this approach it is easy to see what the arguments to list volume calls are. > Ozone: Add paging support to list Volumes > - > > Key: HDFS-10278 > URL: https://issues.apache.org/jira/browse/HDFS-10278 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-10278-HDFS-7240.001.patch > > > Add paging support to list volumes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236019#comment-15236019
]
Steve Loughran commented on HDFS-10277:
---
What's failing here is a file of size zero, with a {{read(new buf[0], 0, 0))}}
That is: it's returning -1, "no data", rather than 0 "you didn't ask for
anything".
Fixing the code *and* clarifying the behaviour, based on that defined in
`java.io.inputstream`: the check for zero remaining bytes comes ahead of the
check for the end of file
> PositionedReadable test testReadFullyZeroByteFile failing in HDFS
> -
>
> Key: HDFS-10277
> URL: https://issues.apache.org/jira/browse/HDFS-10277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Jenkins is failing after HADOOP-12994,
> {{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10247) libhdfs++: Datanode protocol version mismatch
[
https://issues.apache.org/jira/browse/HDFS-10247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236021#comment-15236021
]
James Clampffer commented on HDFS-10247:
Results of tons of continued testing:
-Valgrind shows no memory issues; it also slows things down enough that the bug
doesn't show up.
-Strace gets brutally slow when running 1500 concurrent requests so that wasn't
of any use.
-Dumping out buffers that are handed off to asio show the correct number of
write invocations and valid data for each. This data is guaranteed to still be
on the heap. Both of those make me think it's safe to say that this isn't a
libhdfs++ issue. There aren't enough write calls to overflow the iovec asio
maintains for each descriptor, in fact there will only be one pending write
call at any given time per FD.
*Running the test locally and running tcpdump on the loopback interface is
where interesting stuff becomes visible. Some connections appear to actually be
missing the version # and opcode; the rest of the contents look correct. It
looks like something is happening in asio, the kernel, or the tcp stack.
Asio's epoll reactor is fairly simple and nothing stands out as buggy (and it's
been heavily used). Nothing shows up in dmesg.
Based on these results I think it's worth cleaning up the patch I already
posted that fixes this stuff (somehow), adding some retry logic, and keeping an
eye on the situation. It only shows up in unrealistic situations - 1500
threads all reading 30 bytes each and closing/disposing of the DN connection
between each read.
> libhdfs++: Datanode protocol version mismatch
> -
>
> Key: HDFS-10247
> URL: https://issues.apache.org/jira/browse/HDFS-10247
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-10247.HDFS-8707.000.patch,
> HDFS-10247.HDFS-8707.001.patch
>
>
> Occasionally "Version Mismatch (Expected: 28, Received: 22794 )" shows up in
> the logs. This doesn't happen much at all with less than 500 concurrent
> reads and starts happening often enough to be an issue at 1000 concurrent
> reads.
> I've seen 3 distinct numbers: 23050 (most common), 22538, and 22794. If you
> break these shorts into bytes you get
> {code}
> 23050 -> [90,10]
> 22794 -> [89,10]
> 22538 -> [88,10]
> {code}
> Interestingly enough if we dump buffers holding protobuf messages just before
> they hit the wire we see things like the following with the first two bytes
> as 90,10
> {code}
> buffer
> ={90,10,82,10,64,10,52,10,37,66,80,45,49,51,56,49,48,51,51,57,57,49,45,49,50,55,46,48,46,48,46,49,45,49,52,53,57,53,50,53,54,49,53,55,50,53,16,-127,-128,-128,-128,4,24,-23,7,32,-128,-128,64,18,8,10,0,18,0,26,0,34,0,18,14,108,105,98,104,100,102,115,43,43,95,75,67,43,49,16,0,24,23,32,1}
> {code}
> The first 3 bytes the DN is expecting for an unsecured read block request =
> {code}
> {0,28,81} //[0, 28]->a short for protocol, 81 is read block opcode
> {code}
> This seems like either connections are getting swapped between readers or
> the header isn't being sent for some reason but the protobuf message is.
> I've ruled out memory stomps on the header data (see HDFS-10241) by sticking
> the 3 byte header in it's own static buffer that all requests use.
> Some notes:
> -The mismatched number will stay the same for the duration of a stress test.
> -The mismatch is distributed fairly evenly throughout the logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-10277:
--
Status: Patch Available (was: Open)
> PositionedReadable test testReadFullyZeroByteFile failing in HDFS
> -
>
> Key: HDFS-10277
> URL: https://issues.apache.org/jira/browse/HDFS-10277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HDFS-10277-001.patch
>
>
> Jenkins is failing after HADOOP-12994,
> {{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-10277:
--
Attachment: HDFS-10277-001.patch
> PositionedReadable test testReadFullyZeroByteFile failing in HDFS
> -
>
> Key: HDFS-10277
> URL: https://issues.apache.org/jira/browse/HDFS-10277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HDFS-10277-001.patch
>
>
> Jenkins is failing after HADOOP-12994,
> {{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9735) DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer
[ https://issues.apache.org/jira/browse/HDFS-9735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236135#comment-15236135 ] Lei (Eddy) Xu commented on HDFS-9735: - Thanks a lot for the explanations, [~vinayrpet]. It makes sense. +1. Thanks the good work, [~anu]. > DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer > -- > > Key: HDFS-9735 > URL: https://issues.apache.org/jira/browse/HDFS-9735 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-9735-HDFS-1312.001.patch, > HDFS-9735-HDFS-1312.002.patch > > > Refactor moveBlockAcrossStorage so that code can be shared by both mover and > diskbalancer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9525) hadoop utilities need to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236162#comment-15236162 ] Ravi Prakash commented on HDFS-9525: Thanks for the spirited discussion everyone! :-) [~steve_l] : https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L842 seems like its logging @ info in case the file is missing. Is this not what you are thinking of? Cool tool Kdiag that! I've filed HADOOP-13018 for your 2nd point. I'm going to merge the branch-2 patch if there are no additional comments. > hadoop utilities need to support provided delegation tokens > --- > > Key: HDFS-9525 > URL: https://issues.apache.org/jira/browse/HDFS-9525 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.005.patch, > HDFS-7984.006.patch, HDFS-7984.007.patch, HDFS-7984.patch, > HDFS-9525.008.patch, HDFS-9525.009.patch, HDFS-9525.009.patch, > HDFS-9525.branch-2.008.patch, HDFS-9525.branch-2.009.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9735) DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer
[ https://issues.apache.org/jira/browse/HDFS-9735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-9735: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) [~eddyxu] Thank you for the code reviews. I have committed this to the feature branch. > DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer > -- > > Key: HDFS-9735 > URL: https://issues.apache.org/jira/browse/HDFS-9735 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-9735-HDFS-1312.001.patch, > HDFS-9735-HDFS-1312.002.patch > > > Refactor moveBlockAcrossStorage so that code can be shared by both mover and > diskbalancer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10175) add per-operation stats to FileSystem.Statistics
[ https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HDFS-10175: - Attachment: HDFS-10175.004.patch > add per-operation stats to FileSystem.Statistics > > > Key: HDFS-10175 > URL: https://issues.apache.org/jira/browse/HDFS-10175 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Ram Venkatesh >Assignee: Mingliang Liu > Attachments: HDFS-10175.000.patch, HDFS-10175.001.patch, > HDFS-10175.002.patch, HDFS-10175.003.patch, HDFS-10175.004.patch, > TestStatisticsOverhead.java > > > Currently FileSystem.Statistics exposes the following statistics: > BytesRead > BytesWritten > ReadOps > LargeReadOps > WriteOps > These are in-turn exposed as job counters by MapReduce and other frameworks. > There is logic within DfsClient to map operations to these counters that can > be confusing, for instance, mkdirs counts as a writeOp. > Proposed enhancement: > Add a statistic for each DfsClient operation including create, append, > createSymlink, delete, exists, mkdirs, rename and expose them as new > properties on the Statistics object. The operation-specific counters can be > used for analyzing the load imposed by a particular job on HDFS. > For example, we can use them to identify jobs that end up creating a large > number of files. > Once this information is available in the Statistics object, the app > frameworks like MapReduce can expose them as additional counters to be > aggregated and recorded as part of job summary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236187#comment-15236187
]
Hadoop QA commented on HDFS-10277:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 13s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 24s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:blac
[jira] [Commented] (HDFS-10278) Ozone: Add paging support to list Volumes
[
https://issues.apache.org/jira/browse/HDFS-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236188#comment-15236188
]
Hadoop QA commented on HDFS-10278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 35s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
10s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 31s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 9s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77 with JDK v1.8.0_77
generated 1 new + 34 unchanged - 1 fixed = 35 total (was 35) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 51s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95 with JDK v1.7.0_95
generated 1 new + 36 unchanged - 1 fixed = 37 total (was 37) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
2 unchanged - 0 fixed = 4 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 25s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 11s
{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77 with JDK
v1.8.0_77 generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m 56s
{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95 with JDK
v1.7.0_95 generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
[jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
[ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236197#comment-15236197 ] Tsz Wo Nicholas Sze commented on HDFS-3702: --- I suggest HBase should do the same way of how it today is using favoredNodes. > ... Would we remove the create override when the NO_LOCAL_WRITE FS hint gets > added? ... We will deprecate it first before removing it. I believe the method can stay for some time and has no urgency of being removed. > Add an option for NOT writing the blocks locally if there is a datanode on > the same box as the client > - > > Key: HDFS-3702 > URL: https://issues.apache.org/jira/browse/HDFS-3702 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.5.1 >Reporter: Nicolas Liochon >Assignee: Lei (Eddy) Xu >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, > HDFS-3702.002.patch, HDFS-3702.003.patch, HDFS-3702.004.patch, > HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, > HDFS-3702.008.patch, HDFS-3702.009.patch, HDFS-3702.010.patch, > HDFS-3702.011.patch, HDFS-3702_Design.pdf > > > This is useful for Write-Ahead-Logs: these files are writen for recovery > only, and are not read when there are no failures. > Taking HBase as an example, these files will be read only if the process that > wrote them (the 'HBase regionserver') dies. This will likely come from a > hardware failure, hence the corresponding datanode will be dead as well. So > we're writing 3 replicas, but in reality only 2 of them are really useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10278) Ozone: Add paging support to list Volumes
[ https://issues.apache.org/jira/browse/HDFS-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236264#comment-15236264 ] Anu Engineer commented on HDFS-10278: - The findbugs issue we can be ignore. it is true that all instances for listVolume will be userArgs, but when we support paging for listBuckets, then the template param can be either userArgs or volumeArgs. So having that precondition is useful for future readers to recognize invariants of this code. In other words, this findbug issue will go away automatically in the next code patch. > Ozone: Add paging support to list Volumes > - > > Key: HDFS-10278 > URL: https://issues.apache.org/jira/browse/HDFS-10278 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-10278-HDFS-7240.001.patch > > > Add paging support to list volumes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10175) add per-operation stats to FileSystem.Statistics
[
https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236382#comment-15236382
]
Mingliang Liu commented on HDFS-10175:
--
Thank you very much [~mingma] for the comments.
Yes NN also has audit log which tracks DFS operations. The missing DFSClient
name makes it less useful for the user. Your point about the motivation and map
vs. inline log implementation of [HDFS-9579] made a lot of sense to me. I see
no performance benefit from map approach. I was wondering whether using a
composite (e.g. enum map, array) data structure to manage the
distance->bytesRead mapping makes the code simpler.
0) {{StatisticsData}} should be a bit shorter by delegating the operations to
the composite data structure.
1) The {{incrementBytesReadByDistance(int distance, long newBytes)}} and
{{getBytesReadByDistance(int distance)}} which switch-case all hard-code
variables, may be simplified as we can set/get the bytesRead by distance
directly from map/array.
2) Move {{long getBytesReadByDistance(int distance)}} from {{Statistics}} to
{{StatisticsData}}. If the user is getting bytes read for all distances, she
can call getData() and then iterate the map/array, in which case the getData()
will be called only once. For cases of 1K client threads, this may save the
effort of aggregation.
[~cmccabe] may have different comments about this?
For newly supported APIs, adding an entry in the map and one line of increment
in the new method will make the magic done. From the point of file system APIs,
its public methods are not evolving rapidly. Another dimension will be needed
for cross-DC analysis, while based on the current use case, I don't think this
dimension is heavily needed. One point is that, all the same kind of file
system is sharing the statistic data among threads, regardless of the
remote/local HDFS clusters.
[~jnp] also suggested to make this feature optional/configurable. I found it
hard largely because different file system object has individual
configurations. As they share the statistic data by FS class type (scheme), it
will be confusing if one FS disables this feature and another FS enables this
features. Is there any easy approach to handling this case?
p.s. the v4 patch is to address the {{HAS_NEXT}}/{{LIST_STATUS}}.
> add per-operation stats to FileSystem.Statistics
>
>
> Key: HDFS-10175
> URL: https://issues.apache.org/jira/browse/HDFS-10175
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Ram Venkatesh
>Assignee: Mingliang Liu
> Attachments: HDFS-10175.000.patch, HDFS-10175.001.patch,
> HDFS-10175.002.patch, HDFS-10175.003.patch, HDFS-10175.004.patch,
> TestStatisticsOverhead.java
>
>
> Currently FileSystem.Statistics exposes the following statistics:
> BytesRead
> BytesWritten
> ReadOps
> LargeReadOps
> WriteOps
> These are in-turn exposed as job counters by MapReduce and other frameworks.
> There is logic within DfsClient to map operations to these counters that can
> be confusing, for instance, mkdirs counts as a writeOp.
> Proposed enhancement:
> Add a statistic for each DfsClient operation including create, append,
> createSymlink, delete, exists, mkdirs, rename and expose them as new
> properties on the Statistics object. The operation-specific counters can be
> used for analyzing the load imposed by a particular job on HDFS.
> For example, we can use them to identify jobs that end up creating a large
> number of files.
> Once this information is available in the Statistics object, the app
> frameworks like MapReduce can expose them as additional counters to be
> aggregated and recorded as part of job summary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9820) Improve distcp to support efficient restore to an earlier snapshot
[
https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236386#comment-15236386
]
Yongjun Zhang commented on HDFS-9820:
-
HI [~jingzhao],
I'm uploading rev 003 to use {{-diff "" }} as command line to indicate that
we are reverting changes made at target cluster.
Would you please help taking a look?
Thanks a lot!
> Improve distcp to support efficient restore to an earlier snapshot
> --
>
> Key: HDFS-9820
> URL: https://issues.apache.org/jira/browse/HDFS-9820
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: distcp
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch
>
>
> HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are
> some complexity and challenges.
> HDFS-7535 improved distcp performance by avoiding copying files that changed
> name since last backup.
> On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data
> from source to target cluster, by only copying changed files since last
> backup. The way it works is use snapshot diff to find out all files changed,
> and copy the changed files only.
> See
> https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/
> This jira is to propose a variation of HDFS-8828, to find out the files
> changed in target cluster since last snapshot sx, and copy these from the
> source target's same snapshot sx, to restore target cluster to sx.
> If a file/dir is
> - renamed, rename it back
> - created in target cluster, delete it
> - modified, put it to the copy list
> - run distcp with the copy list, copy from the source cluster's corresponding
> snapshot
> This could be a new command line switch -rdiff in distcp.
> HDFS-4167 would still be nice to have. It just seems to me that HDFS-9820
> would hopefully be easier to implement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9820) Improve distcp to support efficient restore to an earlier snapshot
[ https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-9820: Attachment: HDFS-9820.003.patch > Improve distcp to support efficient restore to an earlier snapshot > -- > > Key: HDFS-9820 > URL: https://issues.apache.org/jira/browse/HDFS-9820 > Project: Hadoop HDFS > Issue Type: New Feature > Components: distcp >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch, > HDFS-9820.003.patch > > > HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are > some complexity and challenges. > HDFS-7535 improved distcp performance by avoiding copying files that changed > name since last backup. > On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data > from source to target cluster, by only copying changed files since last > backup. The way it works is use snapshot diff to find out all files changed, > and copy the changed files only. > See > https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/ > This jira is to propose a variation of HDFS-8828, to find out the files > changed in target cluster since last snapshot sx, and copy these from the > source target's same snapshot sx, to restore target cluster to sx. > If a file/dir is > - renamed, rename it back > - created in target cluster, delete it > - modified, put it to the copy list > - run distcp with the copy list, copy from the source cluster's corresponding > snapshot > This could be a new command line switch -rdiff in distcp. > HDFS-4167 would still be nice to have. It just seems to me that HDFS-9820 > would hopefully be easier to implement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10269) Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the datanode exit
[
https://issues.apache.org/jira/browse/HDFS-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236405#comment-15236405
]
Lin Yiqun commented on HDFS-10269:
--
The Brahma's idea also looks good to me, I am glad to do further optimization
on this, can assign this work for me?
> Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the
> datanode exit
> --
>
> Key: HDFS-10269
> URL: https://issues.apache.org/jira/browse/HDFS-10269
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-10269.001.patch
>
>
> The datanode start failed and exited when I reused configured for
> dfs.datanode.failed.volumes.tolerated as 5 from my another cluster but
> actually the new cluster only have one datadir path. And this leaded the
> Invalid volume failure config value and threw {{DiskErrorException}}, so the
> datanode shutdown. The info is below:
> {code}
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.common.Storage:
> Failed to add storage for block pool: BP-1239160341-xx.xx.xx.xx-1459929303126
> : BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block
> storage: /home/data/hdfs/data/current/BP-1239160341-xx.xx.xx.xx-1459929303126
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> java.io.IOException: All specified directories are failed to load.
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1361)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid volume failure
> config value: 5
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.(FsDatasetImpl.java:281)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1374)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,359 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,460 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Removed Block pool (Datanode Uuid unassigned)
> 2016-04-07 09:34:47,460 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exiting Datanode
> 2016-04-07 09:34:47,462 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 0
> 2016-04-07 09:34:47,463 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG:
> {code}
> IMO, this will let users feel bad because I only configured a value
> incorrectly. Instead of, we can give a warn info for this and reset this
> value to the default value. It will be a better way for this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10175) add per-operation stats to FileSystem.Statistics
[
https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236410#comment-15236410
]
Colin Patrick McCabe commented on HDFS-10175:
-
Thanks, [~mingma]. I agree that if we are going with a map-based approach, we
can simplify the implementation of HDFS-9579. This may also save a small
amount of space... for example, if FileSystem threads don't have any bytes read
at distance 3 or 4, they will no longer need to store space for
{{bytesReadDistanceOfThreeOrFour}}.
It also seems like a good idea to make the detailed statistics optional. Then
clients could opt-in to paying the overheads, rather than getting the overheads
imposed whether they want them or not.
[~liuml07] wrote:
bq. 2) Move long getBytesReadByDistance(int distance) from Statistics to
StatisticsData. If the user is getting bytes read for all distances, she can
call getData() and then iterate the map/array, in which case the getData() will
be called only once. For cases of 1K client threads, this may save the effort
of aggregation. Colin Patrick McCabe may have different comments about this?
Hmm. I'm not sure I see much of a benefit to this. Users should not be
iterating over internal data structures. It exposes implementation details
that we don't want to expose. I also don't see how it's more efficient, since
you have to iterate over it in either case.
bq. For newly supported APIs, adding an entry in the map and one line of
increment in the new method will make the magic done. From the point of file
system APIs, its public methods are not evolving rapidly.
I agree that adding new statistics is relatively easy with the current patch.
My comment was more that currently, many of these statistics are HDFS-specific.
Since MapReduce needs to work with alternate filesystems, it will need to
handle the case where these detailed statistics are missing or slightly
different.
bq. Another dimension will be needed for cross-DC analysis, while based on the
current use case, I don't think this dimension is heavily needed. One point is
that, all the same kind of file system is sharing the statistic data among
threads, regardless of the remote/local HDFS clusters.
It is not the case that "all the same kind of file system is sharing the
statistic data among threads." Each {{Filesystem}} instance has a separate
Statistics object associated with it:
{code}
/** Recording statistics per a FileSystem class */
private static final Map, Statistics>
statisticsTable =
new IdentityHashMap, Statistics>();
{code}
So as long as you use separate FS instances for the remote FS and local FS, it
should work for this use-case. This is also part of the point that I was
making earlier that memory overheads add up quickly, since each new FS instance
multiplies the overhead.
> add per-operation stats to FileSystem.Statistics
>
>
> Key: HDFS-10175
> URL: https://issues.apache.org/jira/browse/HDFS-10175
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Ram Venkatesh
>Assignee: Mingliang Liu
> Attachments: HDFS-10175.000.patch, HDFS-10175.001.patch,
> HDFS-10175.002.patch, HDFS-10175.003.patch, HDFS-10175.004.patch,
> TestStatisticsOverhead.java
>
>
> Currently FileSystem.Statistics exposes the following statistics:
> BytesRead
> BytesWritten
> ReadOps
> LargeReadOps
> WriteOps
> These are in-turn exposed as job counters by MapReduce and other frameworks.
> There is logic within DfsClient to map operations to these counters that can
> be confusing, for instance, mkdirs counts as a writeOp.
> Proposed enhancement:
> Add a statistic for each DfsClient operation including create, append,
> createSymlink, delete, exists, mkdirs, rename and expose them as new
> properties on the Statistics object. The operation-specific counters can be
> used for analyzing the load imposed by a particular job on HDFS.
> For example, we can use them to identify jobs that end up creating a large
> number of files.
> Once this information is available in the Statistics object, the app
> frameworks like MapReduce can expose them as additional counters to be
> aggregated and recorded as part of job summary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-10175) add per-operation stats to FileSystem.Statistics
[
https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236410#comment-15236410
]
Colin Patrick McCabe edited comment on HDFS-10175 at 4/12/16 1:40 AM:
--
Thanks, [~mingma]. I agree that if we are going with a map-based approach, we
can simplify the implementation of HDFS-9579. This may also save a small
amount of space... for example, if FileSystem threads don't have any bytes read
at distance 3 or 4, they will no longer need to store space for
{{bytesReadDistanceOfThreeOrFour}}.
It also seems like a good idea to make the detailed statistics optional. Then
clients could opt-in to paying the overheads, rather than getting the overheads
imposed whether they want them or not.
[~liuml07] wrote:
bq. 2) Move long getBytesReadByDistance(int distance) from Statistics to
StatisticsData. If the user is getting bytes read for all distances, she can
call getData() and then iterate the map/array, in which case the getData() will
be called only once. For cases of 1K client threads, this may save the effort
of aggregation. Colin Patrick McCabe may have different comments about this?
Hmm. I'm not sure I see much of a benefit to this. Users should not be
iterating over internal data structures. It exposes implementation details
that we don't want to expose. I also don't see how it's more efficient, since
you have to iterate over it in either case.
bq. For newly supported APIs, adding an entry in the map and one line of
increment in the new method will make the magic done. From the point of file
system APIs, its public methods are not evolving rapidly.
I agree that adding new statistics is relatively easy with the current patch.
My comment was more that currently, many of these statistics are HDFS-specific.
Since MapReduce needs to work with alternate filesystems, it will need to
handle the case where these detailed statistics are missing or slightly
different.
bq. Another dimension will be needed for cross-DC analysis, while based on the
current use case, I don't think this dimension is heavily needed. One point is
that, all the same kind of file system is sharing the statistic data among
threads, regardless of the remote/local HDFS clusters.
(revised earlier comment about per-class stats) Yes, it does appear that stats
are per-class.
was (Author: cmccabe):
Thanks, [~mingma]. I agree that if we are going with a map-based approach, we
can simplify the implementation of HDFS-9579. This may also save a small
amount of space... for example, if FileSystem threads don't have any bytes read
at distance 3 or 4, they will no longer need to store space for
{{bytesReadDistanceOfThreeOrFour}}.
It also seems like a good idea to make the detailed statistics optional. Then
clients could opt-in to paying the overheads, rather than getting the overheads
imposed whether they want them or not.
[~liuml07] wrote:
bq. 2) Move long getBytesReadByDistance(int distance) from Statistics to
StatisticsData. If the user is getting bytes read for all distances, she can
call getData() and then iterate the map/array, in which case the getData() will
be called only once. For cases of 1K client threads, this may save the effort
of aggregation. Colin Patrick McCabe may have different comments about this?
Hmm. I'm not sure I see much of a benefit to this. Users should not be
iterating over internal data structures. It exposes implementation details
that we don't want to expose. I also don't see how it's more efficient, since
you have to iterate over it in either case.
bq. For newly supported APIs, adding an entry in the map and one line of
increment in the new method will make the magic done. From the point of file
system APIs, its public methods are not evolving rapidly.
I agree that adding new statistics is relatively easy with the current patch.
My comment was more that currently, many of these statistics are HDFS-specific.
Since MapReduce needs to work with alternate filesystems, it will need to
handle the case where these detailed statistics are missing or slightly
different.
bq. Another dimension will be needed for cross-DC analysis, while based on the
current use case, I don't think this dimension is heavily needed. One point is
that, all the same kind of file system is sharing the statistic data among
threads, regardless of the remote/local HDFS clusters.
It is not the case that "all the same kind of file system is sharing the
statistic data among threads." Each {{Filesystem}} instance has a separate
Statistics object associated with it:
{code}
/** Recording statistics per a FileSystem class */
private static final Map, Statistics>
statisticsTable =
new IdentityHashMap, Statistics>();
{code}
So as long as you use separate FS instances for the remote FS and local FS, it
should work for this use
[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236418#comment-15236418 ] Ravi Prakash commented on HDFS-10271: - Hi Vinod! Counting of reservedSpace on Datanodes is all messed up right now :( I don't know if you should block on getting all these fixes in, because they may be a while. These issues already exist in 2.7.* (and I haven't heard very many complaints). There exists a workaround (restart the Datanode). I'm going to push out Target Version to 2.7.4 . Please revert if you disagree > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes reserved may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-10271: Target Version/s: 2.6.5, 2.7.4 (was: 2.7.3, 2.6.5) > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes reserved may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10269) Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the datanode exit
[
https://issues.apache.org/jira/browse/HDFS-10269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236420#comment-15236420
]
Andrew Wang commented on HDFS-10269:
Sure, let's file a new JIRA for the further optimization, ping me and I can
review.
> Invalid value configured for dfs.datanode.failed.volumes.tolerated cause the
> datanode exit
> --
>
> Key: HDFS-10269
> URL: https://issues.apache.org/jira/browse/HDFS-10269
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-10269.001.patch
>
>
> The datanode start failed and exited when I reused configured for
> dfs.datanode.failed.volumes.tolerated as 5 from my another cluster but
> actually the new cluster only have one datadir path. And this leaded the
> Invalid volume failure config value and threw {{DiskErrorException}}, so the
> datanode shutdown. The info is below:
> {code}
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.common.Storage:
> Failed to add storage for block pool: BP-1239160341-xx.xx.xx.xx-1459929303126
> : BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block
> storage: /home/data/hdfs/data/current/BP-1239160341-xx.xx.xx.xx-1459929303126
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> java.io.IOException: All specified directories are failed to load.
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1361)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> Block pool (Datanode Uuid unassigned) service to
> /xx.xx.xx.xx:9000. Exiting.
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid volume failure
> config value: 5
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.(FsDatasetImpl.java:281)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1374)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:801)
> at java.lang.Thread.run(Thread.java:745)
> 2016-04-07 09:34:45,358 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,359 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ending block pool service for: Block pool (Datanode Uuid
> unassigned) service to /xx.xx.xx.xx:9000
> 2016-04-07 09:34:45,460 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Removed Block pool (Datanode Uuid unassigned)
> 2016-04-07 09:34:47,460 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exiting Datanode
> 2016-04-07 09:34:47,462 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 0
> 2016-04-07 09:34:47,463 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> SHUTDOWN_MSG:
> {code}
> IMO, this will let users feel bad because I only configured a value
> incorrectly. Instead of, we can give a warn info for this and reset this
> value to the default value. It will be a better way for this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[
https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236421#comment-15236421
]
Lin Yiqun commented on HDFS-9847:
-
Hi, everyone, can we go ahead of this JIRA and do a final review and commit. It
seems there are no large disagreement from the latest comments, thanks.
> HDFS configuration without time unit name should accept friendly time units
> ---
>
> Key: HDFS-9847
> URL: https://issues.apache.org/jira/browse/HDFS-9847
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-9847-branch-2.001.patch,
> HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch,
> HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch,
> HDFS-9847-nothrow.004.patch, HDFS-9847.001.patch, HDFS-9847.002.patch,
> HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch,
> HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch
>
>
> In HDFS-9821, it talks about the issue of leting existing keys use friendly
> units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names
> contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make
> some other configurations which without time unit name to accept friendly
> time units. The time unit {{seconds}} is frequently used in hdfs. We can
> updating this configurations first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9820) Improve distcp to support efficient restore to an earlier snapshot
[
https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236436#comment-15236436
]
Hadoop QA commented on HDFS-9820:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-tools/hadoop-distcp: patch generated 8 new + 90
unchanged - 11 fixed = 98 total (was 101) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s
{color} | {color:red} hadoop-tools/hadoop-distcp generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 59s {color}
| {color:red} hadoop-distcp in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 19s {color}
| {color:red} hadoop-distcp in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 4s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-distcp |
| | Comparison of String parameter using == or != in
org.apache.hadoop.tools.DistCpOptions.setUseDiff(boolean, String, String) At
DistCpOptions.java:== or != in
org.apache.hadoop.tools.DistCpOptions.setUseDiff(boolean, String, String) At
DistCpOptions.java:[line 308] |
| JDK v1.8.0_77 Failed junit tests | hadoop.tools.TestOptionsParser |
| JDK v1.7.0_95 Failed junit tests | hadoop.tools.TestOptionsParser |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:y
[jira] [Commented] (HDFS-8356) Document missing properties in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236464#comment-15236464
]
Hadoop QA commented on HDFS-8356:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 54s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 53s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 50s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 1s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black}
[jira] [Commented] (HDFS-10175) add per-operation stats to FileSystem.Statistics
[
https://issues.apache.org/jira/browse/HDFS-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236486#comment-15236486
]
Hadoop QA commented on HDFS-10175:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 0s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 13s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 6s
{color} | {color:red} root: patch generated 1 new + 211 unchanged - 1 fixed =
212 total (was 212) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 58s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 45s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 6s {color}
| {color:red} hadoop
[jira] [Created] (HDFS-10279) Improve the validation for tolerated volumes
Lin Yiqun created HDFS-10279: Summary: Improve the validation for tolerated volumes Key: HDFS-10279 URL: https://issues.apache.org/jira/browse/HDFS-10279 Project: Hadoop HDFS Issue Type: Improvement Reporter: Lin Yiqun Assignee: Lin Yiqun Now the misconfiguration for dfs.datanode.failed.volumes.tolerated are detected too late and not easily be found. We can move the validation logic for tolerated volumes to a eariler time that before datanode regists to namenode. And this will let us detect the misconfiguration soon and easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10279) Improve the validation for tolerated volumes
[ https://issues.apache.org/jira/browse/HDFS-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Yiqun updated HDFS-10279: - Status: Patch Available (was: Open) > Improve the validation for tolerated volumes > > > Key: HDFS-10279 > URL: https://issues.apache.org/jira/browse/HDFS-10279 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-10279.001.patch > > > Now the misconfiguration for dfs.datanode.failed.volumes.tolerated are > detected too late and not easily be found. We can move the validation logic > for tolerated volumes to a eariler time that before datanode regists to > namenode. And this will let us detect the misconfiguration soon and easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10279) Improve the validation for tolerated volumes
[ https://issues.apache.org/jira/browse/HDFS-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Yiqun updated HDFS-10279: - Attachment: HDFS-10279.001.patch > Improve the validation for tolerated volumes > > > Key: HDFS-10279 > URL: https://issues.apache.org/jira/browse/HDFS-10279 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-10279.001.patch > > > Now the misconfiguration for dfs.datanode.failed.volumes.tolerated are > detected too late and not easily be found. We can move the validation logic > for tolerated volumes to a eariler time that before datanode regists to > namenode. And this will let us detect the misconfiguration soon and easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236627#comment-15236627 ] Vinayakumar B commented on HDFS-10271: -- This seems to be a pretty straight-forward bug. Simple Append op releases more reserved space than it actually should. Thanks [~brahmareddy] for nice find. +1 for the patch. I will commit it shortly. > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes reserved may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10279) Improve the validation for tolerated volumes
[ https://issues.apache.org/jira/browse/HDFS-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236630#comment-15236630 ] Lin Yiqun commented on HDFS-10279: -- Attach a initial patch. Thanks [~brahmareddy] for great idea. [~andrew.wang], can see this JIRA and review my patch. > Improve the validation for tolerated volumes > > > Key: HDFS-10279 > URL: https://issues.apache.org/jira/browse/HDFS-10279 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-10279.001.patch > > > Now the misconfiguration for dfs.datanode.failed.volumes.tolerated are > detected too late and not easily be found. We can move the validation logic > for tolerated volumes to a eariler time that before datanode regists to > namenode. And this will let us detect the misconfiguration soon and easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236632#comment-15236632
]
Akira AJISAKA commented on HDFS-10277:
--
+1, I verified the patch fixes the test failure.
> PositionedReadable test testReadFullyZeroByteFile failing in HDFS
> -
>
> Key: HDFS-10277
> URL: https://issues.apache.org/jira/browse/HDFS-10277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HDFS-10277-001.patch
>
>
> Jenkins is failing after HADOOP-12994,
> {{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8356) Document missing properties in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236636#comment-15236636 ] Ray Chiang commented on HDFS-8356: -- RE: Failing unit tests All tests pass in my tree. > Document missing properties in hdfs-default.xml > --- > > Key: HDFS-8356 > URL: https://issues.apache.org/jira/browse/HDFS-8356 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.0 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability, test > Attachments: HDFS-8356.001.patch, HDFS-8356.002.patch, > HDFS-8356.003.patch, HDFS-8356.004.patch, HDFS-8356.005.patch, > HDFS-8356.006.patch, HDFS-8356.007.patch, HDFS-8356.008.patch, > HDFS-8356.009.patch, HDFS-8356.010.patch, HDFS-8356.011.patch, > HDFS-8356.012.patch, HDFS-8356.013.patch, HDFS-8356.014.branch-2.patch, > HDFS-8356.014.patch, HDFS-8356.015.branch-2.patch, HDFS-8356.015.patch > > > The following properties are currently not defined in hdfs-default.xml. These > properties should either be > A) documented in hdfs-default.xml OR > B) listed as an exception (with comments, e.g. for internal use) in the > TestHdfsConfigFields unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-10277:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-2, and branch-2.8. Thanks [~ste...@apache.org]
for the contribution!
> PositionedReadable test testReadFullyZeroByteFile failing in HDFS
> -
>
> Key: HDFS-10277
> URL: https://issues.apache.org/jira/browse/HDFS-10277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.8.0
>
> Attachments: HDFS-10277-001.patch
>
>
> Jenkins is failing after HADOOP-12994,
> {{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-10271: - Description: 1. File already have some bytes available in block. (ex: 1024B) 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) 3. write one byte and flush, 4. close() After close(), releasing *BlockSize-1* bytes from reservedspace instead of *BlockSize-1025* bytes. Extra bytes released from reservedSpace may create problem for other writers. was: 1. File already have some bytes available in block. (ex: 1024B) 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) 3. write one byte and flush, 4. close() After close(), releasing *BlockSize-1* bytes from reservedspace instead of *BlockSize-1025* bytes. Extra bytes reserved may create problem for other writers. > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes released from reservedSpace may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-10271: - Target Version/s: 2.6.4, 2.7.3 (was: 2.6.5, 2.7.4) > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes released from reservedSpace may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8356) Document missing properties in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236668#comment-15236668 ] Akira AJISAKA commented on HDFS-8356: - +1, committing these patches. > Document missing properties in hdfs-default.xml > --- > > Key: HDFS-8356 > URL: https://issues.apache.org/jira/browse/HDFS-8356 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.0 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability, test > Attachments: HDFS-8356.001.patch, HDFS-8356.002.patch, > HDFS-8356.003.patch, HDFS-8356.004.patch, HDFS-8356.005.patch, > HDFS-8356.006.patch, HDFS-8356.007.patch, HDFS-8356.008.patch, > HDFS-8356.009.patch, HDFS-8356.010.patch, HDFS-8356.011.patch, > HDFS-8356.012.patch, HDFS-8356.013.patch, HDFS-8356.014.branch-2.patch, > HDFS-8356.014.patch, HDFS-8356.015.branch-2.patch, HDFS-8356.015.patch > > > The following properties are currently not defined in hdfs-default.xml. These > properties should either be > A) documented in hdfs-default.xml OR > B) listed as an exception (with comments, e.g. for internal use) in the > TestHdfsConfigFields unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-10271: - Target Version/s: 2.7.3, 2.6.5 (was: 2.7.3, 2.6.4) > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes released from reservedSpace may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-10271: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.6.5 2.7.3 Status: Resolved (was: Patch Available) Committed to trunk, branch-2, branch-2.8, branch-2.7 and branch-2.6. Thanks [~brahmareddy] for the contribution. Thanks [~raviprak] for the participation. > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Fix For: 2.7.3, 2.6.5 > > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes released from reservedSpace may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236682#comment-15236682 ] Hudson commented on HDFS-10271: --- FAILURE: Integrated in Hadoop-trunk-Commit #9596 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9596/]) HDFS-10271. Extra bytes are getting released from reservedSpace for (vinayakumarb: rev a9a607f8fc0d996af3fb37f7efa7591d6655900d) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Fix For: 2.7.3, 2.6.5 > > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes released from reservedSpace may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10277) PositionedReadable test testReadFullyZeroByteFile failing in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236683#comment-15236683
]
Hudson commented on HDFS-10277:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9596 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9596/])
HDFS-10277. PositionedReadable test testReadFullyZeroByteFile failing in
(aajisaka: rev a409508b3f4c46b419c41b9cdff83429d9d025ce)
*
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
> PositionedReadable test testReadFullyZeroByteFile failing in HDFS
> -
>
> Key: HDFS-10277
> URL: https://issues.apache.org/jira/browse/HDFS-10277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.8.0
>
> Attachments: HDFS-10277-001.patch
>
>
> Jenkins is failing after HADOOP-12994,
> {{che.hadoop.fs.contract.AbstractContractSeekTest.testReadFullyZeroByteFile(AbstractContractSeekTest.java:373)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8356) Document missing properties in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236681#comment-15236681 ] Hudson commented on HDFS-8356: -- FAILURE: Integrated in Hadoop-trunk-Commit #9596 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9596/]) HDFS-8356. Document missing properties in hdfs-default.xml. Contributed (aajisaka: rev 209303be3a4d038b420ae3c11230d419bba07575) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-common-project/hadoop-common/src/site/markdown/DeprecatedProperties.md > Document missing properties in hdfs-default.xml > --- > > Key: HDFS-8356 > URL: https://issues.apache.org/jira/browse/HDFS-8356 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.0 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability, test > Attachments: HDFS-8356.001.patch, HDFS-8356.002.patch, > HDFS-8356.003.patch, HDFS-8356.004.patch, HDFS-8356.005.patch, > HDFS-8356.006.patch, HDFS-8356.007.patch, HDFS-8356.008.patch, > HDFS-8356.009.patch, HDFS-8356.010.patch, HDFS-8356.011.patch, > HDFS-8356.012.patch, HDFS-8356.013.patch, HDFS-8356.014.branch-2.patch, > HDFS-8356.014.patch, HDFS-8356.015.branch-2.patch, HDFS-8356.015.patch > > > The following properties are currently not defined in hdfs-default.xml. These > properties should either be > A) documented in hdfs-default.xml OR > B) listed as an exception (with comments, e.g. for internal use) in the > TestHdfsConfigFields unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8356) Document missing properties in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8356: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Committed the patches to trunk and branch-2. Thanks [~rchiang] for the great work! > Document missing properties in hdfs-default.xml > --- > > Key: HDFS-8356 > URL: https://issues.apache.org/jira/browse/HDFS-8356 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.0 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability, test > Fix For: 2.9.0 > > Attachments: HDFS-8356.001.patch, HDFS-8356.002.patch, > HDFS-8356.003.patch, HDFS-8356.004.patch, HDFS-8356.005.patch, > HDFS-8356.006.patch, HDFS-8356.007.patch, HDFS-8356.008.patch, > HDFS-8356.009.patch, HDFS-8356.010.patch, HDFS-8356.011.patch, > HDFS-8356.012.patch, HDFS-8356.013.patch, HDFS-8356.014.branch-2.patch, > HDFS-8356.014.patch, HDFS-8356.015.branch-2.patch, HDFS-8356.015.patch > > > The following properties are currently not defined in hdfs-default.xml. These > properties should either be > A) documented in hdfs-default.xml OR > B) listed as an exception (with comments, e.g. for internal use) in the > TestHdfsConfigFields unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10271) Extra bytes are getting released from reservedSpace for append
[ https://issues.apache.org/jira/browse/HDFS-10271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236691#comment-15236691 ] Brahma Reddy Battula commented on HDFS-10271: - [~vinayrpet] thanks a lot for review and commit.. > Extra bytes are getting released from reservedSpace for append > -- > > Key: HDFS-10271 > URL: https://issues.apache.org/jira/browse/HDFS-10271 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Critical > Fix For: 2.7.3, 2.6.5 > > Attachments: HDFS-10271-01.patch, HDFS-10271-branch-2.7-01.patch > > > 1. File already have some bytes available in block. (ex: 1024B) > 2. Re-open the file for append, (Here reserving for (BlockSize-1024) bytes) > 3. write one byte and flush, > 4. close() > After close(), releasing *BlockSize-1* bytes from reservedspace instead of > *BlockSize-1025* bytes. > Extra bytes released from reservedSpace may create problem for other writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
