[jira] [Commented] (HDFS-12966) Ozone: owner name should be set properly when the container allocation happens
[
https://issues.apache.org/jira/browse/HDFS-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310936#comment-16310936
]
genericqa commented on HDFS-12966:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
12 unchanged - 0 fixed = 14 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
39s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}205m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.TestStorageContainerManager |
\\
\\
|| Subsystem || Rep
[jira] [Comment Edited] (HDFS-12347) TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
[
https://issues.apache.org/jira/browse/HDFS-12347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310885#comment-16310885
]
Brahma Reddy Battula edited comment on HDFS-12347 at 1/4/18 7:13 AM:
-
Since it's staright forward change from testcase, Pushed to
{{branch-2.7}},{{branch-2.8}} and {{branch-2.9}}. Thanks [~vinayrpet] .
was (Author: brahmareddy):
Pushed to {{branch-2.7}},{{branch-2.8}} and {{branch-2.9}}. Thanks [~vinayrpet]
.
> TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
> ---
>
> Key: HDFS-12347
> URL: https://issues.apache.org/jira/browse/HDFS-12347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-beta1, 2.7.5, 3.0.1
>Reporter: Xiao Chen
>Assignee: Bharat Viswanadham
>Priority: Critical
> Fix For: 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: HDFS-12347.00.patch, trunk.failed.xml
>
>
> Seems to be failing consistently on trunk from yesterday-ish.
> A sample failure is
> https://builds.apache.org/job/PreCommit-HDFS-Build/20824/testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancerRPCDelay/testBalancerRPCDelay/
> Running locally failed with:
> {noformat}
> type="java.lang.AssertionError">
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12347) TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
[
https://issues.apache.org/jira/browse/HDFS-12347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310885#comment-16310885
]
Brahma Reddy Battula commented on HDFS-12347:
-
Pushed to {{branch-2.7}},{{branch-2.8}} and {{branch-2.9}}. Thanks [~vinayrpet]
.
> TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
> ---
>
> Key: HDFS-12347
> URL: https://issues.apache.org/jira/browse/HDFS-12347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-beta1, 2.7.5, 3.0.1
>Reporter: Xiao Chen
>Assignee: Bharat Viswanadham
>Priority: Critical
> Fix For: 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: HDFS-12347.00.patch, trunk.failed.xml
>
>
> Seems to be failing consistently on trunk from yesterday-ish.
> A sample failure is
> https://builds.apache.org/job/PreCommit-HDFS-Build/20824/testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancerRPCDelay/testBalancerRPCDelay/
> Running locally failed with:
> {noformat}
> type="java.lang.AssertionError">
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12347) TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
[
https://issues.apache.org/jira/browse/HDFS-12347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12347:
Fix Version/s: 2.7.6
2.8.4
2.9.1
> TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
> ---
>
> Key: HDFS-12347
> URL: https://issues.apache.org/jira/browse/HDFS-12347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-beta1, 2.7.5, 3.0.1
>Reporter: Xiao Chen
>Assignee: Bharat Viswanadham
>Priority: Critical
> Fix For: 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: HDFS-12347.00.patch, trunk.failed.xml
>
>
> Seems to be failing consistently on trunk from yesterday-ish.
> A sample failure is
> https://builds.apache.org/job/PreCommit-HDFS-Build/20824/testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancerRPCDelay/testBalancerRPCDelay/
> Running locally failed with:
> {noformat}
> type="java.lang.AssertionError">
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12860) StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310875#comment-16310875
]
Xiao Chen commented on HDFS-12860:
--
Thanks [~eddyxu] for creating the jira and providing the fix. Patch 01 LGTM.
[~Sammi] would you like to clarify about the comment on testing?
> StripedBlockUtil#getRangesInternalBlocks throws exception for the block group
> size larger than 2GB
> --
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch, HDFS-12860.01.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[
https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310845#comment-16310845
]
Xiao Chen commented on HDFS-12974:
--
Thanks [~zhenyi]. I played with this a little more, and believe your fix is
correct. Good work on tracing this down!
I initially thought this has to do at the client side in CryptoAdmin, but it
turns out the message actually was lost when building server rpc response.
Thanks [~shahrs87] for chiming in. I can see the {{AuthorizationException}}
correctly propagated. Don't think ValueQueue does any tricks to it - this is
when creating the zones, so should fail when {{getMetadata}}. KMSCP just throws
the AE which is an IOE.
I still think it worths a end-to-end test here though, as Rushabh also echo'ed.
You can look at {{TestEncryptionZones}} for some existing example on how this
could be done. {{EncryptionFaultInjector}} is of direct interest. We can inject
an {{AuthorizationException}} in {{createEncryptionZone}} on NN, and verify the
exception at the client side. {{GenericTestUtils#assertExceptionContains}} is
the helper function we usually use.
> Exception information can not be returned when I create transparent
> encryption zone.
>
>
> Key: HDFS-12974
> URL: https://issues.apache.org/jira/browse/HDFS-12974
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 3.0.0
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch
>
>
> When I add the following configuration to the kms-acl.xml file, I create
> encrypted space and I can not get any exception information.
>
> key.acl.key2.GENERATE_EEK
> mr
>
> root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone
> 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> RemoteException:
> root@fangzhenyi01:~#
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12347) TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
[
https://issues.apache.org/jira/browse/HDFS-12347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310815#comment-16310815
]
Vinayakumar B commented on HDFS-12347:
--
bq. I feel, this should be push to branch-2.7,branch-2.8 and branch-2.9 as well
since HDFS-11384 pushed..?
It makes sense to push to branch-2.7/2.8/2.9 as well.
Please go ahead [~brahmareddy].
> TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
> ---
>
> Key: HDFS-12347
> URL: https://issues.apache.org/jira/browse/HDFS-12347
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-beta1, 2.7.5, 3.0.1
>Reporter: Xiao Chen
>Assignee: Bharat Viswanadham
>Priority: Critical
> Fix For: 2.10.0, 3.0.1
>
> Attachments: HDFS-12347.00.patch, trunk.failed.xml
>
>
> Seems to be failing consistently on trunk from yesterday-ish.
> A sample failure is
> https://builds.apache.org/job/PreCommit-HDFS-Build/20824/testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancerRPCDelay/testBalancerRPCDelay/
> Running locally failed with:
> {noformat}
> type="java.lang.AssertionError">
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Status: Patch Available (was: In Progress) > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0, 3.0.0-beta1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, > HDFS-12935.004.patch, HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Attachment: HDFS-12935.004.patch > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-beta1, 3.0.0 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, > HDFS-12935.004.patch, HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Status: In Progress (was: Patch Available) > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0, 3.0.0-beta1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, > HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310725#comment-16310725
]
Xiao Chen commented on HDFS-12897:
--
Thanks [~GeLiXin] for the continued investigation into this.
bq. What's the supposed, right behavior?
{{TestErasureCodingPolicyWithSnapshot#testSnapshotsOnErasureCodingDirsParentDir}}
was added by HDFS-8777. Ping [~rakeshr] [~zhz] [~jingzhao], was this
intentional or was this a miss? I didn't see any discussions about the behavior
in that jira.
IMO the existing test is wrong:
{code}
...
fs.setErasureCodingPolicy(ecDir, ecPolicy.getName());
final Path snap1 = fs.createSnapshot(ecDirParent, "snap1");
assertEquals("Got unexpected erasure coding policy", ecPolicy,
fs.getErasureCodingPolicy(snap1ECDir));
...
fs.delete(ecDir, true);
fs.mkdir(ecDir, FsPermission.getDirDefault());
final Path snap2 = fs.createSnapshot(ecDirParent, "snap2");
assertNull("Expected null erasure coding policy",
fs.getErasureCodingPolicy(snap2ECDir));
...
fs.setErasureCodingPolicy(ecDir, ecPolicy.getName());
final Path snap3 = fs.createSnapshot(ecDirParent, "snap3");
assertEquals("Got unexpected erasure coding policy", ecPolicy, ezSnap3);
...
assertEquals("Got unexpected erasure coding policy", ecPolicy,
fs.getErasureCodingPolicy(snap1ECDir));
assertEquals("Got unexpected erasure coding policy", ecPolicy,
fs.getErasureCodingPolicy(snap2ECDir)); // <-- shouldn't this be null?
{code}
> Path not found when we get the ec policy for a .snapshot dir
>
>
> Key: HDFS-12897
> URL: https://issues.apache.org/jira/browse/HDFS-12897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs, snapshots
>Affects Versions: 3.0.0-alpha1, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: LiXin Ge
> Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch,
> HDFS-12897.003.patch
>
>
> Scenario:-
> ---
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
> replicationFallbacks=[]}
> *Which is the correct behavior ?*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12966) Ozone: owner name should be set properly when the container allocation happens
[ https://issues.apache.org/jira/browse/HDFS-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12966: --- Attachment: HDFS-12966-HDFS-7240.003.patch Thanks [~anu], for the review comments. patch v3 is rebased. > Ozone: owner name should be set properly when the container allocation happens > -- > > Key: HDFS-12966 > URL: https://issues.apache.org/jira/browse/HDFS-12966 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12966-HDFS-7240.001.patch, > HDFS-12966-HDFS-7240.002.patch, HDFS-12966-HDFS-7240.003.patch > > > Currently , while the container allocation happens, the owner name is > hardcoded as "OZONE". > It should be set to KSM instance id/ CBlock Manager instance Id from where > the container creation call happens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Open (was: Patch Available) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Virajith Jalaparti >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch, > HDFS-9240.003.patch, HDFS-9240.004.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Patch Available (was: Open) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Virajith Jalaparti >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch, > HDFS-9240.003.patch, HDFS-9240.004.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310661#comment-16310661 ] Hudson commented on HDFS-12948: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13447 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13447/]) HDFS-12948. DiskBalancer report command top option should only take (yqlin: rev 2a48b3594c502c4dcf201f2b60386383c0d9ae91) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Affects Versions: 3.0.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.1.0 > > Attachments: HDFS-12948.001.patch, HDFS-12948.002.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12948: - Affects Version/s: 3.0.0 Component/s: diskbalancer > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Affects Versions: 3.0.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.1.0 > > Attachments: HDFS-12948.001.patch, HDFS-12948.002.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12948: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Committed this to trunk. Thanks [~shashikant] for the contribution. > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.1.0 > > Attachments: HDFS-12948.001.patch, HDFS-12948.002.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310650#comment-16310650 ] Yiqun Lin commented on HDFS-12948: -- LGTM, +1. Will commit this shortly. > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Attachments: HDFS-12948.001.patch, HDFS-12948.002.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Status: Open (was: Patch Available) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch, HDFS-12942.004.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Status: Patch Available (was: Open) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch, HDFS-12942.004.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310591#comment-16310591
]
LiXin Ge commented on HDFS-12897:
-
Hi [~xiaochen], Thanks for your new suggestions, I'm done on my local branch,
and It's no problem for me to create a new issue to work on the
{{StoragePolicy}} error you mentioned.
But I have a bit of confusion on the new issue. Since snapshots are supposed to
be immutable and read only, the appearance seems reasonable now:
bq. getPolicy on s1 should say RS(6,3), and on s2 should say RS(3,2).
The 1-liner change seems exactly correct to get the specified snapshot, but
after this 1-liner change take effect, the appearance changes from older
snapshots still have the old EC Policy to older snapshots have NULL EC Policy
now, and some exist unit test like
{{testSnapshotsOnErasureCodingDirsParentDir}} fails.
What's the supposed, right behavior?
> Path not found when we get the ec policy for a .snapshot dir
>
>
> Key: HDFS-12897
> URL: https://issues.apache.org/jira/browse/HDFS-12897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs, snapshots
>Affects Versions: 3.0.0-alpha1, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: LiXin Ge
> Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch,
> HDFS-12897.003.patch
>
>
> Scenario:-
> ---
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
> replicationFallbacks=[]}
> *Which is the correct behavior ?*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[
https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310537#comment-16310537
]
genericqa commented on HDFS-9240:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 7s{color}
| {color:red} root generated 3 new + 1232 unchanged - 0 fixed = 1235 total (was
1232) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 13s{color} | {color:orange} root: The patch generated 9 new + 536 unchanged
- 39 fixed = 545 total (was 575) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
20s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
32s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
17s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 53s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 13s{color}
| {color:red} hadoop-gridmix in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-openstack in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 35s{color}
| {color:red} hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
42s{color} | {color:red} The patch generated 1 ASF License warnings. {color}
[jira] [Commented] (HDFS-12964) Read a opened file timely and effectively when it's being written by other
[ https://issues.apache.org/jira/browse/HDFS-12964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310520#comment-16310520 ] Konstantin Shvachko commented on HDFS-12964: I thought this is already supported via FsShel Tail command? > Read a opened file timely and effectively when it's being written by other > -- > > Key: HDFS-12964 > URL: https://issues.apache.org/jira/browse/HDFS-12964 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Yang Yun >Priority: Minor > Attachments: HADOOP-12964.001.patch > > > One thread opens a HDFS file and keeps writing. Another thread opens same > file and keeps reading it at the same time, also want to get the newest > content of file. that happens in many environments, for example, in some > message transmission applications. And it also requires the new content can > be read timely and effectively, for there maybe many tasks are working in > same time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12985) NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted
[
https://issues.apache.org/jira/browse/HDFS-12985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-12985:
--
Description:
NameNode crashes repeatedly with NPE at the startup when trying to find the
total number of under construction blocks. This crash happens after an open
file, which was also part of a snapshot gets deleted along with the snapshot.
{noformat}
Failed to start namenode.
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager.getNumUnderConstructionBlocks(LeaseManager.java:146)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCompleteBlocksTotal(FSNamesystem.java:6537)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1232)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:706)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
{noformat}
was:
NameNode crashes repeatedly with NPE at the startup when trying to find the
total number of under construction blocks. This crash happens after an open
file, which was also part of a snapshot gets deleted along with the snapshot.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager.getNumUnderConstructionBlocks(LeaseManager.java:144)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCompleteBlocksTotal(FSNamesystem.java:4456)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1158)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:825)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:968)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:947)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1674)
at
org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2110)
at
org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2075)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testSnapshotsForOpenFilesAndDeletion3(TestOpenFilesWithSnapshot.java:747)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
> NameNode crashes during restart after an OpenForWrite file present in the
> Snapshot got deleted
> --
>
> Key: HDFS-12985
> URL: https://issues.apache.org/jira/browse/HDFS-12985
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>
> NameNode crashes repeatedly with NPE at the startup when trying to find the
> total number of under construction blocks. This crash happens after an open
> file, which was also part of a snapshot gets deleted along with the snapshot.
> {noformat}
> Failed to start namenode.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager.getNumUnderConstructionBlocks(LeaseManager.java:146)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCompleteBlocksTotal(FSNamesystem.java:6537)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1232)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692)
>
[jira] [Commented] (HDFS-12966) Ozone: owner name should be set properly when the container allocation happens
[
https://issues.apache.org/jira/browse/HDFS-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310494#comment-16310494
]
Anu Engineer commented on HDFS-12966:
-
[~shashikant] Can you please rebase this patch? I am getting a patch
application failure.
{noformat}
patching file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeySpaceManager.java
Hunk #1 FAILED at 172.
{noformat}
Thanks
> Ozone: owner name should be set properly when the container allocation happens
> --
>
> Key: HDFS-12966
> URL: https://issues.apache.org/jira/browse/HDFS-12966
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Fix For: HDFS-7240
>
> Attachments: HDFS-12966-HDFS-7240.001.patch,
> HDFS-12966-HDFS-7240.002.patch
>
>
> Currently , while the container allocation happens, the owner name is
> hardcoded as "OZONE".
> It should be set to KSM instance id/ CBlock Manager instance Id from where
> the container creation call happens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12870) Ozone: Service Discovery: REST endpoint in KSM for getServiceList
[ https://issues.apache.org/jira/browse/HDFS-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12870: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~nandakumar131] Thanks for the contribution. I have committed this to the feature branch. > Ozone: Service Discovery: REST endpoint in KSM for getServiceList > - > > Key: HDFS-12870 > URL: https://issues.apache.org/jira/browse/HDFS-12870 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar > Fix For: HDFS-7240 > > Attachments: HDFS-12870-HDFS-7240.000.patch, > HDFS-12870-HDFS-7240.001.patch, HDFS-12870-HDFS-7240.002.patch > > > A new REST call to be added in KSM which will return the list of Services > that are there in Ozone cluster, this will be used by OzoneClient for > establishing the connection. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12985) NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted
Manoj Govindassamy created HDFS-12985:
-
Summary: NameNode crashes during restart after an OpenForWrite
file present in the Snapshot got deleted
Key: HDFS-12985
URL: https://issues.apache.org/jira/browse/HDFS-12985
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 2.8.0
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
NameNode crashes repeatedly with NPE at the startup when trying to find the
total number of under construction blocks. This crash happens after an open
file, which was also part of a snapshot gets deleted along with the snapshot.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager.getNumUnderConstructionBlocks(LeaseManager.java:144)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCompleteBlocksTotal(FSNamesystem.java:4456)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1158)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:825)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:751)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:968)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:947)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1674)
at
org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2110)
at
org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2075)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testSnapshotsForOpenFilesAndDeletion3(TestOpenFilesWithSnapshot.java:747)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310469#comment-16310469
]
Manoj Govindassamy commented on HDFS-11848:
---
Thanks for posting a patch revision [~linyiqun]. Sorry for the delay, will
review this week.
> Enhance dfsadmin listOpenFiles command to list files under a given path
> ---
>
> Key: HDFS-11848
> URL: https://issues.apache.org/jira/browse/HDFS-11848
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Yiqun Lin
> Attachments: HDFS-11848.001.patch, HDFS-11848.002.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> One more thing that would be nice here is to filter the output on a passed
> path or DataNode. Usecases: An admin might already know a stale file by path
> (perhaps from fsck's -openforwrite), and wants to figure out who the lease
> holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files
> filtered by path.
> {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the
> open file list for any given ancestor directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12931) Handle InvalidEncryptionKeyException during DistributedFileSystem#getFileChecksum
[
https://issues.apache.org/jira/browse/HDFS-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310462#comment-16310462
]
Hudson commented on HDFS-12931:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13445 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13445/])
HDFS-12931. Handle InvalidEncryptionKeyException during (xyao: rev
3ba985997d1dc37e5ba017dd0ab1d36083b5f77b)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptedTransfer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java
> Handle InvalidEncryptionKeyException during
> DistributedFileSystem#getFileChecksum
> -
>
> Key: HDFS-12931
> URL: https://issues.apache.org/jira/browse/HDFS-12931
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HDFS-12931.001.patch, HDFS-12931.002.patch,
> HDFS-12931.003.patch
>
>
> Hive external table create fails because of HDFS fails to fetch block MD5
> checksum.
> This happens because of the InvalidEncryptionKeyException.
> {code}
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=-1675775329) doesn't exist. Current key: 1496422662
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:478)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.newSocketSend(SaslDataTransferClient.java:142)
> at org.apache.hadoop.hdfs.DFSClient.connectToDN(DFSClient.java:2450)
> at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:2310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1569)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1565)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1577)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12931) Handle InvalidEncryptionKeyException during DistributedFileSystem#getFileChecksum
[
https://issues.apache.org/jira/browse/HDFS-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-12931:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Thanks [~msingh] for the contribution and all for the reviews. I've committed
the fix to the trunk.
> Handle InvalidEncryptionKeyException during
> DistributedFileSystem#getFileChecksum
> -
>
> Key: HDFS-12931
> URL: https://issues.apache.org/jira/browse/HDFS-12931
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HDFS-12931.001.patch, HDFS-12931.002.patch,
> HDFS-12931.003.patch
>
>
> Hive external table create fails because of HDFS fails to fetch block MD5
> checksum.
> This happens because of the InvalidEncryptionKeyException.
> {code}
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=-1675775329) doesn't exist. Current key: 1496422662
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:478)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.newSocketSend(SaslDataTransferClient.java:142)
> at org.apache.hadoop.hdfs.DFSClient.connectToDN(DFSClient.java:2450)
> at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:2310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1569)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1565)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1577)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12931) Handle InvalidEncryptionKeyException during DistributedFileSystem#getFileChecksum
[
https://issues.apache.org/jira/browse/HDFS-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-12931:
--
Summary: Handle InvalidEncryptionKeyException during
DistributedFileSystem#getFileChecksum (was: Uncaught
InvalidEncryptionKeyException during Hive external table create fails because
of failure to fetch block MD5 checksum)
> Handle InvalidEncryptionKeyException during
> DistributedFileSystem#getFileChecksum
> -
>
> Key: HDFS-12931
> URL: https://issues.apache.org/jira/browse/HDFS-12931
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12931.001.patch, HDFS-12931.002.patch,
> HDFS-12931.003.patch
>
>
> Hive external table create fails because of HDFS fails to fetch block MD5
> checksum.
> This happens because of the InvalidEncryptionKeyException.
> {code}
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=-1675775329) doesn't exist. Current key: 1496422662
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:478)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.newSocketSend(SaslDataTransferClient.java:142)
> at org.apache.hadoop.hdfs.DFSClient.connectToDN(DFSClient.java:2450)
> at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:2310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1569)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1565)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1577)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12931) Uncaught InvalidEncryptionKeyException during Hive external table create fails because of failure to fetch block MD5 checksum
[
https://issues.apache.org/jira/browse/HDFS-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-12931:
--
Summary: Uncaught InvalidEncryptionKeyException during Hive external table
create fails because of failure to fetch block MD5 checksum (was: Hive
external table create fails because of failure to fetch block MD5 checksum)
> Uncaught InvalidEncryptionKeyException during Hive external table create
> fails because of failure to fetch block MD5 checksum
> -
>
> Key: HDFS-12931
> URL: https://issues.apache.org/jira/browse/HDFS-12931
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12931.001.patch, HDFS-12931.002.patch,
> HDFS-12931.003.patch
>
>
> Hive external table create fails because of HDFS fails to fetch block MD5
> checksum.
> This happens because of the InvalidEncryptionKeyException.
> {code}
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=-1675775329) doesn't exist. Current key: 1496422662
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:478)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.newSocketSend(SaslDataTransferClient.java:142)
> at org.apache.hadoop.hdfs.DFSClient.connectToDN(DFSClient.java:2450)
> at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:2310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1569)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1565)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1577)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12931) Hive external table create fails because of failure to fetch block MD5 checksum
[
https://issues.apache.org/jira/browse/HDFS-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310420#comment-16310420
]
Xiaoyu Yao commented on HDFS-12931:
---
Thanks [~msingh] for the fix. Patch v3 LGTM, +1. I will commit it shortly.
> Hive external table create fails because of failure to fetch block MD5
> checksum
> ---
>
> Key: HDFS-12931
> URL: https://issues.apache.org/jira/browse/HDFS-12931
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12931.001.patch, HDFS-12931.002.patch,
> HDFS-12931.003.patch
>
>
> Hive external table create fails because of HDFS fails to fetch block MD5
> checksum.
> This happens because of the InvalidEncryptionKeyException.
> {code}
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=-1675775329) doesn't exist. Current key: 1496422662
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:478)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.newSocketSend(SaslDataTransferClient.java:142)
> at org.apache.hadoop.hdfs.DFSClient.connectToDN(DFSClient.java:2450)
> at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:2310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1569)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1565)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1577)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12976) Introduce StandbyReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310398#comment-16310398
]
Konstantin Shvachko commented on HDFS-12976:
Thanks for the link [~elgoiri]. Yes something similar to what you did for RBF,
but we will need more sophisticated logic than just random. The trade-offs I
see:
* Switch over from active to standby could be expensive since the client should
wait while SBN catches up with its lastSeenId, so {{StandbyReadProxyProvider}}
should be sticky once it starts reading from SBN.
* But we also want load balancing so that the overall load is distributed
evenly between nodes.
> Introduce StandbyReadProxyProvider
> --
>
> Key: HDFS-12976
> URL: https://issues.apache.org/jira/browse/HDFS-12976
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>
> {{StandbyReadProxyProvider}} should implement {{FailoverProxyProvider}}
> interface and be able to submit read requests to ANN and SBN(s).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota
[
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310337#comment-16310337
]
Íñigo Goiri commented on HDFS-12934:
Thanks [~linyiqun] for [^HDFS-12934.004.patch].
Regarding {{Router#getChildrenPaths()}}, my main concern is that it's a
function in {{Router}} with no reference to quota while basically does that.
{{getQuotaUsage(String path)}} was just a proposal but anything with "quota"
there should be fine.
The change to {{TreeMap}} seems to simplify the code a bunch.
I'll be testing the code this week and give a fully tested review tomorrow.
> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch,
> HDFS-12934.003.patch, HDFS-12934.004.patch, RBF support global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder.
> Currently the quota will be applied for each subcluster under the specified
> folder via RPC call.
> It will be very useful for users that federation can support setting global
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can
> exceed maximun quota value.
> # We need to construct one cache map for storing the sum
> quota usage of these subclusters under federation folder. Every time we want
> to do WRITE operation under specified folder, we will get its quota usage
> from cache and verify its quota. If quota exceeded, throw exception,
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The
> set/unset command will be like:
> {noformat}
> hdfs dfsrouteradmin -setQuota -ns -ss
> hdfs dfsrouteradmin -clrQuota
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12895) RBF: Add ACL support for mount table
[
https://issues.apache.org/jira/browse/HDFS-12895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310328#comment-16310328
]
Íñigo Goiri commented on HDFS-12895:
bq. If the users don't have rights to access real filesystem path, it will be
rejected. Do we still need to make this check in mount table level?
That's true. This would be more like preventing forwarding requests at a
federated level. It would still be mount table level (federation) and not file
system itself.
Not 100% sure is completely valuable though. I'll give it a thought to see if
we have any good use case for this. It makes more sense in terms of semantics
than use.
> RBF: Add ACL support for mount table
>
>
> Key: HDFS-12895
> URL: https://issues.apache.org/jira/browse/HDFS-12895
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha3
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: RBF, incompatible
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HDFS-12895-branch-2.001.patch, HDFS-12895.001.patch,
> HDFS-12895.002.patch, HDFS-12895.003.patch, HDFS-12895.004.patch,
> HDFS-12895.005.patch, HDFS-12895.006.patch, HDFS-12895.007.patch
>
>
> Adding ACL support for the Mount Table management. Following is the initial
> design of ACL control for the mount table management.
> Each mount table has its owner, group name and permission.
> The mount table permissions (FsPermission), here we use
> {{org.apache.hadoop.fs.permission.FsPermission}} to do the access check:
> # READ permission: you can read the mount table info.
> # WRITE permission: you can add remove or update this mount table info.
> # EXECUTE permission: This won't be used.
> The add command of mount table will be extended like this
> {noformat}
> $HADOOP_HOME/bin/hdfs dfsrouteradmin [-add
> [-owner ] [-group ] [-mode ]]
> {noformat}
> * is UNIX-style permissions for the mount table. Permissions are
> specified in octal, e.g. 0755. By default, this is set to 0755*.
> If we want update the ACL info of specfied mount table, just execute add
> command again. This command not only adding for new mount talle but also
> updating mount table once it finds given mount table is existed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12976) Introduce StandbyReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-12976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310316#comment-16310316
]
Íñigo Goiri commented on HDFS-12976:
Any consideration on using the hedged approach?
In HDFS-6648, I added the ability to make the access to the NNs in a random
order too.
In addition, the Router-based federation, remembers the state of the Namenodes
and issues the requests to the active one.
Once this is done, I should add an option to mimic the behavior you are
describing and send the READ requests to the standby namenodes.
> Introduce StandbyReadProxyProvider
> --
>
> Key: HDFS-12976
> URL: https://issues.apache.org/jira/browse/HDFS-12976
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>
> {{StandbyReadProxyProvider}} should implement {{FailoverProxyProvider}}
> interface and be able to submit read requests to ANN and SBN(s).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12982) [SPS]: Reduce the locking and cleanup the Namesystem access
[
https://issues.apache.org/jira/browse/HDFS-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310248#comment-16310248
]
genericqa commented on HDFS-12982:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
36s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10285 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 25 new + 398 unchanged - 6 fixed = 423 total (was 404) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
59s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Nullcheck of fileStatus at line 282 of value previously dereferenced in
org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier.run() At
StoragePolicySatisfier.java:282 of value previously dereferenced in
org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier.run() At
StoragePolicySatisfier.java:[line 282] |
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
| | hadoop.hdfs.server.namenode.sps.TestBlockStorageMovementAttemptedItems |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12982 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904439/HDFS-12982-HDFS-10285-00.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f49068632628 3.
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Patch Available (was: Open) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Virajith Jalaparti >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch, > HDFS-9240.003.patch, HDFS-9240.004.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Attachment: HDFS-9240.004.patch Posting a rebased patch (v004). > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Virajith Jalaparti >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch, > HDFS-9240.003.patch, HDFS-9240.004.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Open (was: Patch Available) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Virajith Jalaparti >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch, > HDFS-9240.003.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[
https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310207#comment-16310207
]
Rushabh S Shah edited comment on HDFS-12974 at 1/3/18 7:59 PM:
---
[~zhenyi]: Thanks for reporting the bug.
Couple of things.
1. Are you sure that {{AuthorizationException}} is getting propagated all the
way back to CryptoAdmin ?
And after the fix applied, are you able to see the exception message ?
The code path that it is following involves rpc call from {{DFSClient}} to
{{namenode}} and then Rest call from {{namenode}} to {{kms}}.
On top of that, {{KMSClientProvider}} uses Guava caching to load the keys very
first time and that method {{ValueQueue#initializeQueuesForKeys}} just throws
{{ExecutionException}}.
2. Also I think the test case is not sufficient for the bug.
If the bug is in {{CreateEncryptionZoneCommand}} then it will be nice to create
one which throws {{AuthorizationException}} and see whether it prints right
exception.
was (Author: shahrs87):
[~zhenyi]: Thanks for reporting the bug.
Couple of things.
1. Are you sure that {{AuthorizationException}} is getting propagated back all
the way back to CryptoAdmin ?
And after the fix applied, you are able to see the exception message ?
The code path that it is following involves rpc call from {{DFSClient}} to
{{namenode}} and then Rest call from {{namenode}} to {{kms}}.
On top of that, {{KMSClientProvider}} uses Guava caching to load the keys very
first time and that method {{ValueQueue#initializeQueuesForKeys}} just throws
{{ExecutionException}}.
2. Also I think the test case is not sufficient for the bug.
If the bug is in {{CreateEncryptionZoneCommand}} then it will be nice to create
one which throws {{AuthorizationException}} and see whether it prints right
exception.
> Exception information can not be returned when I create transparent
> encryption zone.
>
>
> Key: HDFS-12974
> URL: https://issues.apache.org/jira/browse/HDFS-12974
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 3.0.0
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch
>
>
> When I add the following configuration to the kms-acl.xml file, I create
> encrypted space and I can not get any exception information.
>
> key.acl.key2.GENERATE_EEK
> mr
>
> root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone
> 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> RemoteException:
> root@fangzhenyi01:~#
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[
https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310207#comment-16310207
]
Rushabh S Shah commented on HDFS-12974:
---
[~zhenyi]: Thanks for reporting the bug.
Couple of things.
1. Are you sure that {{AuthorizationException}} is getting propagated back all
the way back to CryptoAdmin ?
And after the fix applied, you are able to see the exception message ?
The code path that it is following involves rpc call from {{DFSClient}} to
{{namenode}} and then Rest call from {{namenode}} to {{kms}}.
On top of that, {{KMSClientProvider}} uses Guava caching to load the keys very
first time and that method {{ValueQueue#initializeQueuesForKeys}} just throws
{{ExecutionException}}.
2. Also I think the test case is not sufficient for the bug.
If the bug is in {{CreateEncryptionZoneCommand}} then it will be nice to create
one which throws {{AuthorizationException}} and see whether it prints right
exception.
> Exception information can not be returned when I create transparent
> encryption zone.
>
>
> Key: HDFS-12974
> URL: https://issues.apache.org/jira/browse/HDFS-12974
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 3.0.0
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch
>
>
> When I add the following configuration to the kms-acl.xml file, I create
> encrypted space and I can not get any exception information.
>
> key.acl.key2.GENERATE_EEK
> mr
>
> root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone
> 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> RemoteException:
> root@fangzhenyi01:~#
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310137#comment-16310137
]
Rakesh R commented on HDFS-12911:
-
bq. readLockNS/readUnlockNS are sensitive APIs to expose to an external service.
Thanks [~chris.douglas] for the comment. I have uploaded initial patch to
HDFS-12982 jira, I've used {{Context}} interface proposed earlier in this jira
and included simple methods into it, which can be used by the SPS class.
Majorly, query part is getting {{HdfsLocatedFileStatus}} from NN and does the
computation, assignment logic based on the file status object. Please review
the approach/patch when you get a chance. Thanks!
> [SPS]: Fix review comments from discussions in HDFS-10285
> -
>
> Key: HDFS-12911
> URL: https://issues.apache.org/jira/browse/HDFS-12911
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
> Attachments: HDFS-12911-HDFS-10285-01.patch,
> HDFS-12911-HDFS-10285-02.patch, HDFS-12911.00.patch
>
>
> This is the JIRA for tracking the possible improvements or issues discussed
> in main JIRA
> So far comments to handle
> Daryn:
> # Lock should not kept while executing placement policy.
> # While starting up the NN, SPS Xattrs checks happen even if feature
> disabled. This could potentially impact the startup speed.
> UMA:
> # I am adding one more possible improvement to reduce Xattr objects
> significantly.
> SPS Xattr is constant object. So, we create one Xattr deduplication object
> once statically and use the same object reference when required to add SPS
> Xattr to Inode. So, here additional bytes required for storing SPS Xattr
> would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr
> overhead should come down significantly IMO. Lets explore the feasibility on
> this option.
> Xattr list Future will not be specially created for SPS, that list would have
> been created by SetStoragePolicy already on the same directory. So, no extra
> Feature creation because of SPS alone.
> # Currently SPS putting long id objects in Q for tracking SPS called Inodes.
> So, it is additional created and size of it would be (obj ref + value) = (8 +
> 8) bytes [ ignoring alignment for time being]
> So, the possible improvement here is, instead of creating new Long obj, we
> can keep existing inode object for tracking. Advantage is, Inode object
> already maintained in NN, so no new object creation is needed. So, we just
> need to maintain one obj ref. Above two points should significantly reduce
> the memory requirements of SPS. So, for SPS call: 8bytes for called inode
> tracking + 8 bytes for Xattr ref.
> # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will
> reduce unnecessary Node creations inside LinkedList.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[
https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310044#comment-16310044
]
genericqa commented on HDFS-12948:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 19s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904416/HDFS-12948.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1c4995097f69 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9bf813 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22546/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22546/testReport/ |
| Max. process+thread count | 3637 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job
[jira] [Assigned] (HDFS-12984) BlockPoolSlice can leak in a mini dfs cluster
[ https://issues.apache.org/jira/browse/HDFS-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDFS-12984: - Assignee: Ajay Kumar > BlockPoolSlice can leak in a mini dfs cluster > - > > Key: HDFS-12984 > URL: https://issues.apache.org/jira/browse/HDFS-12984 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5 >Reporter: Robert Joseph Evans >Assignee: Ajay Kumar > > When running some unit tests for storm we found that we would occasionally > get out of memory errors on the HDFS integration tests. > When I got a heap dump I found that the ShutdownHookManager was full of > BlockPoolSlice$1 instances. Which hold a reference to the BlockPoolSlice > which then in turn holds a reference to the DataNode etc > It looks like when shutdown is called on the BlockPoolSlice there is no way > to remove the shut down hook in because no reference to it is saved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12982) [SPS]: Reduce the locking and cleanup the Namesystem access
[
https://issues.apache.org/jira/browse/HDFS-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-12982:
Status: Patch Available (was: Open)
> [SPS]: Reduce the locking and cleanup the Namesystem access
> ---
>
> Key: HDFS-12982
> URL: https://issues.apache.org/jira/browse/HDFS-12982
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-12982-HDFS-10285-00.patch
>
>
> This task is to optimize the NS lock usage in SPS and cleanup the Namesystem
> access via {{Context}} interface.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12982) [SPS]: Reduce the locking and cleanup the Namesystem access
[
https://issues.apache.org/jira/browse/HDFS-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-12982:
Attachment: HDFS-12982-HDFS-10285-00.patch
I've tried an initial attempt to refactor the code by separating out the
querying the HdfsFileStatus info and does computation outside name system lock.
Welcome feedback on the approach.
Note: I have adopted the {{Context}} interface discussed in HDFS-12911 to keep
all the namesystem/blockManager NN calls behind the context interface.
> [SPS]: Reduce the locking and cleanup the Namesystem access
> ---
>
> Key: HDFS-12982
> URL: https://issues.apache.org/jira/browse/HDFS-12982
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-12982-HDFS-10285-00.patch
>
>
> This task is to optimize the NS lock usage in SPS and cleanup the Namesystem
> access via {{Context}} interface.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6804) race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"
[
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309934#comment-16309934
]
genericqa commented on HDFS-6804:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
18s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:26 |
| Failed junit tests | hadoop.hdfs.TestWriteConfigurationToDFS |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.TestTrashWithSecureEncryptionZones |
| | hadoop.hdfs.TestBalancerBandwidth |
| | hadoop.hdfs.TestDFSClientRetries |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.hdfs.TestDatanodeRegistration |
| | org.apache.hadoop.hdfs.TestBlocksScheduledCounter |
| | org.apache.hadoop.hdfs.TestDFSClientFailover |
| | org.apache.hadoop.hdfs.TestListFilesInDFS |
| | org.apache.hadoop.hdfs.web.TestWebHdfsTokens |
| | org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade |
| | org.apache.hadoop.hdfs.TestFileAppendRestart |
| | org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter |
| | org.apache.hadoop.hdfs.TestSeekBug |
| | org.apache.hadoop.hdfs.TestDFSMkdirs |
| | org.apache.hadoop.hdfs.TestDatanodeReport |
| | org.apache.hadoop.hdfs.web.TestWebHDFS |
| | org.apache.hadoop.hdfs.web.TestWebHDFSXAttr |
| | org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | org.apache.hadoop.hdfs.TestMiniDFSCluster |
| | org.apache.hadoop.hdfs.TestDistributedFileSystem |
| | org.apache.hadoop.hdfs.web.TestWebHDFSForHA |
| | org.apache.hadoop.hdfs.TestTrashWithEncryptionZones |
| | org.apache.hadoop.hdfs.TestSetTimes |
| | org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication |
| | org.apache.hadoop.hdfs.TestDFSShell |
| | org.apache.hadoop.hdfs.web.TestWebHDFSAcl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:y
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309922#comment-16309922 ] Daryn Sharp commented on HDFS-10285: Will be catching up this week, sorry I missed the meeting due to vacation. What was the outcome regarding using a placement policy? That would require absolutely minimal change to the NN. I need to look at the latest patch. bq. When we start SPS outside, we +should have security+ and HA: +probably we will handle this post merge+ Yes... But: # s/should have security/MUST have security/ # s/probably we will handle this post merge/MUST handle this before merge/ Seriously, skimping/punting on security for any feature is a non-negotiable red line. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12966) Ozone: owner name should be set properly when the container allocation happens
[
https://issues.apache.org/jira/browse/HDFS-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309915#comment-16309915
]
genericqa commented on HDFS-12966:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
22s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
39s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 19s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
12 unchanged - 0 fixed = 15 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12966 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904412/HDFS-12966-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 189a258d2d99 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / ace9aec |
| maven | version:
[jira] [Commented] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309914#comment-16309914 ] Daryn Sharp commented on HDFS-12911: Will review soon. > [SPS]: Fix review comments from discussions in HDFS-10285 > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911.00.patch > > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. > # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will > reduce unnecessary Node creations inside LinkedList. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12984) BlockPoolSlice can leak in a mini dfs cluster
Robert Joseph Evans created HDFS-12984: -- Summary: BlockPoolSlice can leak in a mini dfs cluster Key: HDFS-12984 URL: https://issues.apache.org/jira/browse/HDFS-12984 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.5 Reporter: Robert Joseph Evans When running some unit tests for storm we found that we would occasionally get out of memory errors on the HDFS integration tests. When I got a heap dump I found that the ShutdownHookManager was full of BlockPoolSlice$1 instances. Which hold a reference to the BlockPoolSlice which then in turn holds a reference to the DataNode etc It looks like when shutdown is called on the BlockPoolSlice there is no way to remove the shut down hook in because no reference to it is saved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12947) Limit the number of Snapshots allowed to be created for a Snapshottable Directory
[ https://issues.apache.org/jira/browse/HDFS-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309854#comment-16309854 ] Tsz Wo Nicholas Sze commented on HDFS-12947: Shash, thanks for working on this. We already have per snapshottable directory snapshot quota. It is a good idea to add a system-wide snapshot limit. - Why checking (maxSnapshotLimit > snapshotQuota)? It seems okay if maxSnapshotLimit > snapshotQuota. I think we should not change the snapshot quota logic. - Let's remove DirectorySnapshottableFeature.SNAPSHOT_LIMIT. > Limit the number of Snapshots allowed to be created for a Snapshottable > Directory > - > > Key: HDFS-12947 > URL: https://issues.apache.org/jira/browse/HDFS-12947 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12947.001.patch, HDFS-12947.002.patch > > > Currently, A snapshottable directory is able to accommodate 65,536 snapshots. > In case a directory has large no of snapshots , deletion of any of the > earlier snapshots take a lot of time which might lead to namenode crash > (HDFS-11225). > This jira is introduced to limit the no of the snapshots under a > snapshottable directory to a reasonable value(say 10) which can be overriden. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12948: --- Attachment: HDFS-12948.002.patch Thanks [~linyiqun], for the review comments. Patch v2 addresses the same. Please have a look. > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Attachments: HDFS-12948.001.patch, HDFS-12948.002.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12966) Ozone: owner name should be set properly when the container allocation happens
[ https://issues.apache.org/jira/browse/HDFS-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12966: --- Attachment: HDFS-12966-HDFS-7240.002.patch Thanks [~anu], for the review comments. patch v2 addresses the review comments. TestSCMCli tests are not related and tracked via HDFS-12968. KeySpaceManager tests are fixed and other tests are not related . Few other tests have timed out in the tests run though they work on my local box. Please have a look. > Ozone: owner name should be set properly when the container allocation happens > -- > > Key: HDFS-12966 > URL: https://issues.apache.org/jira/browse/HDFS-12966 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12966-HDFS-7240.001.patch, > HDFS-12966-HDFS-7240.002.patch > > > Currently , while the container allocation happens, the owner name is > hardcoded as "OZONE". > It should be set to KSM instance id/ CBlock Manager instance Id from where > the container creation call happens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6804) race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"
[
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-6804:
---
Attachment: HDFS-6804-branch-2.8-003.patch
jenkins failures are unrelated,even I raised HADOOP-15153 to track this.
Re-uploading the patch with javadoc error fix to re-trigger jenkins.
> race condition between transferring block and appending block causes
> "Unexpected checksum mismatch exception"
> --
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-6804-branch-2.8-002.patch,
> HDFS-6804-branch-2.8-003.patch, HDFS-6804-branch-2.8.patch,
> Testcase_append_transfer_block.patch
>
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException:
> Unexpected checksum mismatch while writing
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from
> /192.168.2.101:39495
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Da
> taTransfer: Transmitted
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad
> block to NameNode and NameNode marks the replica on the source datanode as
> corrupt. But actually, the replica on the source datanode is valid. Because
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309733#comment-16309733
]
genericqa commented on HDFS-11848:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 620
unchanged - 1 fixed = 620 total (was 621) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-11848 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904378/HDFS-11848.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| u
[jira] [Commented] (HDFS-12636) Ozone: OzoneFileSystem: rpc backend should be supported using unified OzoneClient client
[
https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309635#comment-16309635
]
genericqa commented on HDFS-12636:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 11s{color} | {color:orange} root: The patch generated 8 new + 2 unchanged -
0 fixed = 10 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
14s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 2 new
+ 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
59s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-ozone in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}236m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Inconsistent synchronization of
org.apache.hadoop.scm.storage.ChunkInputStream.bufferIndex; locked 75% of time
Unsynchronized access at ChunkInputStream.java:75% of time Unsynchronized
access at ChunkInputStream.java:[line 240] |
| | Inconsistent synchronization of
org.apache.hadoop.scm.storage.ChunkInputStream.buffers; locked 66% of time
Unsynchronized access at ChunkInputStream.java:66% of time Unsynchronized
access at ChunkInputStream.java:[line 232] |
| Failed junit
[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309608#comment-16309608
]
Brahma Reddy Battula commented on HDFS-12935:
-
Thanks for updating the patch. Apart from the following,rest changes are looks
good to me.
1) As I mentioned earlier,dn't include the {{safemode}} changes in this jira,
there is a jira for that HDFS-8277.
2) *setBalancerBandwidth* :
* This needs to be executed only in active namenode so you can remove the
following. Refer {{allowsnapshot}}
{code}
if (isHaEnabled) {
String nsId = dfsUri.getHost();
List> proxies =
HAUtil.getProxiesForAllNameNodesInNameservice(dfsConf,
nsId, ClientProtocol.class);
for (ProxyAndInfo proxy : proxies) {
proxy.getProxy().setBalancerBandwidth(bandwidth);
System.out.println("Balancer bandwidth is set to " + bandwidth +
" for " + proxy.getAddress());
}
{code}
* and change like following in
org.apache.hadoop.hdfs.server.namenode.FSNamesystem#setBalancerBandwidth
checkOperation(OperationCategory.READ);
Basically the commands which are for {{read}} no need to execute on both the
nodes.
> Get ambiguous result for DFSAdmin command in HA mode when only one namenode
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch,
> HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When
> considering the following two occasions:
> (1)nn1 up and nn2 down
> (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin
> commands will have ambiguous results. The commands can be send successfully
> to the up namenode and are always functionally useful only when nn1 is up
> regardless of exception (IOException when connecting to the down namenode
> nn2). If only nn2 is up, the commands have no use at all and only exception
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to
> set balancer bandwidth value for datanodes as an example. It works and all
> the datanodes can get the setting values only when nn1 is up. If only nn2 is
> up, the command throws exception directly and no datanode get the bandwidth
> setting. Approximately ten DFSAdmin commands use the similar logical process
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei02:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to
> jiangjianfei01:9820 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]#
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11848:
-
Attachment: HDFS-11848.002.patch
> Enhance dfsadmin listOpenFiles command to list files under a given path
> ---
>
> Key: HDFS-11848
> URL: https://issues.apache.org/jira/browse/HDFS-11848
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Yiqun Lin
> Attachments: HDFS-11848.001.patch, HDFS-11848.002.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> One more thing that would be nice here is to filter the output on a passed
> path or DataNode. Usecases: An admin might already know a stale file by path
> (perhaps from fsck's -openforwrite), and wants to figure out who the lease
> holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files
> filtered by path.
> {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the
> open file list for any given ancestor directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11848:
-
Attachment: (was: HDFS-11848.002.patch)
> Enhance dfsadmin listOpenFiles command to list files under a given path
> ---
>
> Key: HDFS-11848
> URL: https://issues.apache.org/jira/browse/HDFS-11848
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Yiqun Lin
> Attachments: HDFS-11848.001.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> One more thing that would be nice here is to filter the output on a passed
> path or DataNode. Usecases: An admin might already know a stale file by path
> (perhaps from fsck's -openforwrite), and wants to figure out who the lease
> holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files
> filtered by path.
> {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the
> open file list for any given ancestor directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309503#comment-16309503
]
genericqa commented on HDFS-11848:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
59s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
17s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 17s{color} |
{color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 17s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 615
unchanged - 1 fixed = 615 total (was 616) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m
3s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
40s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
35s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-11848 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904367/HDFS-11848.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 98c18a3539d8 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c0c7
[jira] [Updated] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11848:
-
Attachment: HDFS-11848.002.patch
Re-attach the patch to fix compile error.
> Enhance dfsadmin listOpenFiles command to list files under a given path
> ---
>
> Key: HDFS-11848
> URL: https://issues.apache.org/jira/browse/HDFS-11848
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Yiqun Lin
> Attachments: HDFS-11848.001.patch, HDFS-11848.002.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> One more thing that would be nice here is to filter the output on a passed
> path or DataNode. Usecases: An admin might already know a stale file by path
> (perhaps from fsck's -openforwrite), and wants to figure out who the lease
> holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files
> filtered by path.
> {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the
> open file list for any given ancestor directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11848:
-
Attachment: (was: HDFS-11848.002.patch)
> Enhance dfsadmin listOpenFiles command to list files under a given path
> ---
>
> Key: HDFS-11848
> URL: https://issues.apache.org/jira/browse/HDFS-11848
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Yiqun Lin
> Attachments: HDFS-11848.001.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> One more thing that would be nice here is to filter the output on a passed
> path or DataNode. Usecases: An admin might already know a stale file by path
> (perhaps from fsck's -openforwrite), and wants to figure out who the lease
> holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files
> filtered by path.
> {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the
> open file list for any given ancestor directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12940) Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309398#comment-16309398
]
Nanda kumar commented on HDFS-12940:
[~vagarychen], it fails occasionally. Once or twice in every twenty runs. Even
in the previous build of this jira testWriteSize and testListKeys has failed
with {{Commit key failed, error:KEY_NOT_FOUND}}
[TestKeySpaceManager#testWriteSize failure |
https://builds.apache.org/job/PreCommit-HDFS-Build/22483/testReport/org.apache.hadoop.ozone.ksm/TestKeySpaceManager/testWriteSize/]
[TestKeySpaceManager#testListKeys failure |
https://builds.apache.org/job/PreCommit-HDFS-Build/22483/testReport/org.apache.hadoop.ozone.ksm/TestKeySpaceManager/testListKeys/]
I will move {{testExpiredOpenKey}} to a separate test class as you suggested.
> Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
> -
>
> Key: HDFS-12940
> URL: https://issues.apache.org/jira/browse/HDFS-12940
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Attachments: HDFS-12940-HDFS-7240.000.patch,
> HDFS-12940-HDFS-7240.001.patch
>
>
> {{TestKeySpaceManager#testExpiredOpenKey}} is flaky.
> In {{testExpiredOpenKey}} we are opening four keys for writing and wait for
> them to expire (without committing). Verification/Assertion is done by
> querying {{MiniOzoneCluster}} and matching the count. Since the {{cluster}}
> instance of {{MiniOzoneCluster}} is shared between test-cases in
> {{TestKeySpaceManager}}, we should not rely on the count. The verification
> should only happen by matching the keyNames and not with the count.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12983) Ozone: dozone: provide docker-compose file for cblock clusters
[ https://issues.apache.org/jira/browse/HDFS-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12983: - Parent Issue: HDFS-8 (was: HDFS-7240) > Ozone: dozone: provide docker-compose file for cblock clusters > -- > > Key: HDFS-12983 > URL: https://issues.apache.org/jira/browse/HDFS-12983 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: ozone >Reporter: Elek, Marton >Assignee: Elek, Marton > > Since HDFS-12469 we have a docker compose file at dev-support/compose/ozone > which makes it easy to start local ozone clusers with multiple datanodes. > In this patch I propose similar config file for the cblock/iscsi servers > (jscsi + cblock + scm + namenode + datanode) to make it easier to check the > latest state. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12931) Hive external table create fails because of failure to fetch block MD5 checksum
[
https://issues.apache.org/jira/browse/HDFS-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309346#comment-16309346
]
genericqa commented on HDFS-12931:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
35s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}124m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.web.TestWebHDFSForHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12931 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904331/HDFS-12931.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0ba45f1b541a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revisio
[jira] [Updated] (HDFS-12636) Ozone: OzoneFileSystem: rpc backend should be supported using unified OzoneClient client
[ https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12636: --- Summary: Ozone: OzoneFileSystem: rpc backend should be supported using unified OzoneClient client (was: Ozone: OzoneFileSystem: both rest/rpc backend should be supported using unified OzoneClient client) > Ozone: OzoneFileSystem: rpc backend should be supported using unified > OzoneClient client > > > Key: HDFS-12636 > URL: https://issues.apache.org/jira/browse/HDFS-12636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain > Fix For: HDFS-7240 > > Attachments: HDFS-12636-HDFS-7240.001.patch > > > OzoneClient library provides a method to invoke both RPC as well as REST > based methods to ozone. This api will help in the improving both the > performance as well as the interface management in OzoneFileSystem. > This jira will be used to convert the REST based calls to use this new > unified client. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12636) Ozone: OzoneFileSystem: both rest/rpc backend should be supported using unified OzoneClient client
[ https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12636: --- Status: Patch Available (was: Open) > Ozone: OzoneFileSystem: both rest/rpc backend should be supported using > unified OzoneClient client > -- > > Key: HDFS-12636 > URL: https://issues.apache.org/jira/browse/HDFS-12636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain > Fix For: HDFS-7240 > > Attachments: HDFS-12636-HDFS-7240.001.patch > > > OzoneClient library provides a method to invoke both RPC as well as REST > based methods to ozone. This api will help in the improving both the > performance as well as the interface management in OzoneFileSystem. > This jira will be used to convert the REST based calls to use this new > unified client. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12870) Ozone: Service Discovery: REST endpoint in KSM for getServiceList
[
https://issues.apache.org/jira/browse/HDFS-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309325#comment-16309325
]
genericqa commented on HDFS-12870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12870 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904330/HDFS-12870-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 04f4f1ec624f 3.13.0-135-generic #18
[jira] [Created] (HDFS-12983) Ozone: dozone: provide docker-compose file for cblock clusters
Elek, Marton created HDFS-12983: --- Summary: Ozone: dozone: provide docker-compose file for cblock clusters Key: HDFS-12983 URL: https://issues.apache.org/jira/browse/HDFS-12983 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: ozone Reporter: Elek, Marton Assignee: Elek, Marton Since HDFS-12469 we have a docker compose file at dev-support/compose/ozone which makes it easy to start local ozone clusers with multiple datanodes. In this patch I propose similar config file for the cblock/iscsi servers (jscsi + cblock + scm + namenode + datanode) to make it easier to check the latest state. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309294#comment-16309294
]
genericqa commented on HDFS-11848:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
30s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
52s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 52s{color} |
{color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project: The patch generated 5 new +
616 unchanged - 1 fixed = 621 total (was 617) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
11s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-11848 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904341/HDFS-11848.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 7af33caf04a2 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk /
[jira] [Commented] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[
https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309282#comment-16309282
]
genericqa commented on HDFS-12974:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 31s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 33s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12974 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904338/HDFS-12974.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2ff2d1ea52ac 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c0c7cce |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22539/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22539/testReport/ |
| Max. process+thread count | 1423 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22539/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache
[jira] [Commented] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[
https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309274#comment-16309274
]
genericqa commented on HDFS-12974:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 118 unchanged - 0 fixed = 120 total (was 118) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 9s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12974 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904336/HDFS-12974.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6cec20e23a37 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c0c7cce |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22538/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22538/testReport/ |
| Max. process+thread count | 1439 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22538/console |
| Powered by | Apache Yetus 0.7.
